Subscribe to this thread
Home - General / All posts - SID images not correctly projected in M9
drtees
203 post(s)
#26-Jun-19 19:01

We recently downloaded 14 SID files from a local jurisdiction. M8 can be set up to open SID files, but I prefer using GeoViewer to open and export SID files. Exporting these 14 images as one combined image creates a tiff file that is too large for M8 to handle. My solution was to use M09 to load up the SID files (or the previously created tiff) and export it as either ecw or png. M8 will have no issues linking to either image format.

The problem is that M9 does not seem to be using any of the metadata from the SID file or the world file created by Geoviewer for the tiff. It seems to default to metric instead of standard units we in the US are condemned to use. Repairing the projection does not bring the image into displaying where it should in the world.

I am attaching a Dropbox link containing one of the SID files. This image should be roughly centered at lat. 47.7922 and long. -122.2008. Local scale should be for both X and Y. Local offset should be 0 for both X and Y. False northing should be 0 and false easting should be 1640416.6666732284.

I am using Manifold 9.0.169.4 on a Dell Precision with 16 GB RAM and an nVidia Quadro K620 GPU.

https://www.dropbox.com/sh/k20y7q1l6tf4ci2/AABLUD_MgsJnF0k_4RHwcfSaa?dl=0

tjhb
10,094 post(s)
#26-Jun-19 23:10

I am attaching a Dropbox link containing one of the SID files. This image should be roughly centered at lat. 47.7922 and long. -122.2008. Local scale should be for both X and Y. Local offset should be 0 for both X and Y. False northing should be 0 and false easting should be 1640416.6666732284.

There seem to be some mistakes in your report here. Suggested corrections:

Local scale should be 0.25 US survey feet for both X and Y. (Added missing value, from .sdw worldfile.)

Local offset should be 0 for both X and Y 1305000 for X and 282000 for Y. (Corrected from .sdw with translation.)

False northing should be 0 (correct) and false easting should be 1640416.6666732284 1640416.667 (corrected from EPSG definition).

The problem is that M9 does not seem to be using any of the metadata from the SID file or the world file...

Manifold 9 is picking up metadata--but it may be incorrect (more on that below).

First though, you might be misunderstanding the import/link workflow for images. I have said a couple of times that I think the UI sets some traps here. Have a look at this thread (for a different projection) for basic workflow and some warnings.

For your image, try this.

  • Import 27891.sid from a folder where the .sid, .sdw and .sid.aux.xml are present.
  • Open the image 27891_Image.
  • Go to Contents > Component.
  • Click the set of calipers at the bottom, hover over Repair Initial Coordinate System, and crucially don't choose a projection from your favourites (a natural instinct, but incorrect for image data). Instead choose Edit Coordinate System at the top.
  • Now you see the EPSG tab selected, EPSG:2926 highlighted in the list, a string of coordinate system parameters in JSON below that (as you say, the figures here are in metres), and at the bottom, a list of Metrics.

It is the metrics that are crucial.

  • Click on the XY (Metrics) button at bottom right, then choose Edit Metrics.

Manifold initially has the XY metrics as follows (in US Survey Feet):

Local scale X 0.8202083333333332

Local scale Y 0.8202083333333332

Local offset X 4281487.499999999

Local offset Y 925194.9999999999

So yes, Manifold has picked up metadata for the image. But the values don't seem to be right (AFAIK). We can come back to why that might be, but for now, try correcting the XY metrics to:

Local scale X 0.25

Local scale Y 0.25

Local offset X 1305000

Local offset Y 282000

After pressing OK of the Metrics dialog you'll see the new metrics summarised in the panel at the bottom.

How does that line up?

tjhb
10,094 post(s)
#27-Jun-19 09:10

A really basic observation here that has only just occurred to me.

The use of the term metrics in the coordinate system dialogs and infrastructure has nothing whatsoever to do with internal units being in the metric or SI system (or some other).

Metrics in the sense used in the dialogs just means "measures" (or distances), basically Latin. The measures used in the Metrics dialog can be metric in the SI sense, or Imperial (established under the former British Empire), or French, or US, or anything else. That always follows the units of the coordinate system in use.

There is absolutely no relation between the two uses of the word. Just in case anyone thought there was.

tjhb
10,094 post(s)
#26-Jun-19 23:42

A bit more on the possible causes of the initial error.

The above result is what I got from importing 27891.sid from a folder where 27891.sdw and 27891.sid.aux.xml were also present.

How about if we try a new import from a folder where only 27891.sid is present?

Then we get initial metrics like this:

Local scale X 0.25

Local scale Y 0.25

Local offset X 1305000.125

Local offset Y 281999.875

So without the worldfile (or .sid.aux.xml), Manifold is getting almost the same metrics as are specified in the worldfile(!).

What does that mean? First, that Manifold apparently prefers metrics from the .sgw worldfile to metrics from the .sid file itself. Secondly, that it seems to misinterpret metrics from the worldfile (though possibly with "help" from the .sid file--it might partly use both sources).

Third, it looks as if Manifold's reading of metrics from the .sid file, when it uses that alone, introduces a half-pixel shift to the east and to the south. That assumes that the worldfile offsets are themselves correct (and that I have translated them correctly). That may not be the case, although I have checked against the offsets given by Global Mapper 18, which agree with the offsets suggested in my previous post. Again, Global Mapper could be wrong too.

I think there are a couple of things here that need to be checked here as possible bugs in Manifold's calculation of metrics from .sid metadata.

tjhb
10,094 post(s)
#27-Jun-19 00:08

A bit more:

I think that when it reads the offsets from the worldfile, Manifold 9 is reading the worldfile values as if they were in metres, not in US Survey Feet (the native units of the projection); then mistakenly "converting" to US Survey Feet for native scales and offsets.

Taking the scales and offsets from the worldfile and the values Manifold derives:

0.25 (local scales in US ft) / 0.3048006096... (metres to US ft) ≈ 0.8202083333333332 (value used for local scales).

1305000 (X offset in US ft) / 0.3048006096... (metres to US ft) ≈ 4281487.499999999 (value used for X offset).

282000 (Y offset in US ft) / 0.3048006096... (metres to US ft) ≈ 925194.9999999999 (value used for Y offset).

(That still leaves the half-pixel offsets when the worldfile is not used.)

adamw


10,447 post(s)
#27-Jun-19 14:42

Thanks to drtees for providing example data, and thanks to you, Tim, for the detailed analysis.

We have investigated the issue, the results are as follows:

The three files in the first post specify the coordinate system in *four* different ways. This is pretty amazing and this makes the files a great test case.

  1. There is coordinate system data embedded in SID. We start by reading that and we read it pretty much correct except we shift the image by half-pixel in both X and Y directions. We'll remove the shift.
  2. There are scales and offsets in the world file. We overlay it on top of what we read from SID, but we fail to convert numbers from US Survey feet to meters. We'll convert data.
  3. There is an embedded WKT (PRJ) string in the XML. We do not currently read it in production builds, but we already read it in some internal builds, overriding whatever coordinate system info we already had. We'll add this feature to production builds when it is ready.
  4. There are TIFF-style tags with coordinate system info in the XML, too. We do not currently read this in production builds either, but we will. Again, the coordinate system info will override whatever info we already had.

Until we implement fixes for 1 and 2 above, the easiest way to read the file is to remove the world file, then override local offset X / Y in the coordinate system metrics, shifting them by half-pixel (for this particular image the effect is the same as rounding offsets to the closest integer value, I guess it's going to be the same for other images).

As a general note, we get why the coordinate system info is specified in so many different ways - because some software packages can only read that info specified as 4 above, some only as 3, etc. However, this has its own perils in that if the specifications disagree it can be rather hard to determine what went wrong. In such cases we recommend removing all auxiliary files and try linking data without them, then start slowly adding auxiliary files one by one - this may help determine which files to use vs which to leave out (because they contain erroneous data or because our code misinterprets the data they contain).

Our current priorities when reading coordinate system info are as follows:

  • read coordinate system info embedded in the data file (eg, TIFF),
  • read PRJ and use it to override existing coordinate system info (when the data file already contains coordinate system info, we assume that the info in PRJ is better, and that PRJ exists specifically to correct potentially wrong info in the data file - because correcting that wrong info directly in the data file is difficult or impossible),
  • read the world file and use it to adjust existing coordinate system info (world files only contain scales / offsets so they can only be used to adjust existing info),
  • read XML produced by Manifold 8 and use it to override existing coordinate system info,
  • read MAPMETA produced by Manifold 9 and use it to override existing coordinate system info.

When we add support for PAM XML, we will likely read it before the world file.

drtees
203 post(s)
#27-Jun-19 17:24

I isloated the SID files and imported them into M9. It appears to have worked well! With all 14 SID files loaded into a map, I was able to merge the images in 149 seconds. I love this feature of M9. Doing the same in M8 (while maintaining maximum image resolution and projection information) would have taken me a couple of hours. This also assumes that there is an easier way to merge images without loss of resolution that I am not aware of in M8. Next step is to export the merged image as ecw so that we can link it in M8.

tjhb
10,094 post(s)
#28-Jun-19 10:48

I love this feature of M9. Doing the same in M8... would have taken me a couple of hours.

I love this feature too. It means I no longer need any other GIS imaging software. (Thanks for 11 great necessary years, Global Mapper.)

I isloated the SID files and imported them into M9. It appears to have worked well!

That is fine so long as you bear in mind that your images will be offset by half a pixel in X and Y.

(I should not need to remind you of that because Adam has confirmed it, but I am reminding you anyway.)

drtees
203 post(s)
#28-Jun-19 21:13

Considering the high resolution of the images, a half-pixel offset may not be noticable (I am assuming that the half-pixel offset refers to the image at maximum resolution). The images will be used mainly to provide visual consideration of current conditions, likely at a scale of 1:1200 or greater. The finer detailed figures will be provided by CAD and will likely not include aerial imagery.

adamw


10,447 post(s)
#01-Jul-19 13:34

If all images are SID, then the half-pixel offset is consistent between them, so you can simply correct it in the merged image.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.