Subscribe to this thread
Home - General / All posts - offset problem with images in M9 (9.0.167.0)
vincent

1,972 post(s)
#11-Jun-18 19:23

Hi,

I imported a drawing and a surface in M8. Then I saved the project and open it in M9. I exported the drawing and the surface (now an image). The projection has been defined in M8 and is the same for both. Everything is fine in M9, all is aligned.

Then I exported the components to a .tif and a .shp

Back in M8, the .tif is shifted around 15m to the North. The shapefile is fine.

Same in ArcGIS, the .tif is shifted around 15m to the North. The shapefile is fine.

The projection is 32187 (EPSG)

I attached a subset. If someone can confirm the problem, that would be appreciated.

The circles in the drawing are visible in the surface. That's the way to see the offset.

Attachments:
20180611.zip

tjhb
10,094 post(s)
#12-Jun-18 01:05

Excellent, though the workflow can be simplified a bit.

(1)

The projection is 32187 (EPSG)

The projection is not EPSG:32187 but standard MTM zone 7 (NAD83). We don't need to be concerned with EPSG.

When the Manifold 8 project is opened in both Manifold 8 and Manifold 9, all of the projection parameters initially match.

(Manifold 9 leaves off one decimal digit from both local offset values, compared to the values shown in 8. Perhaps this is due to different C++ compilers. In any case, the difference would not normally be noticeable, and it is not the issue here.)

So far so good.


(2)

In Manifold 8, when the Map is opened, the drawing and the surface are exactly aligned.

But in Manifold 9, when the Map is opened, there is an initial offset in the Y axis between drawing and image. The image is shown about 3m south of the aligned position in Manifold 8.

So that's one problem.


(3)

I now export the image from Manifold 9 to TIFF, and reimport the result straight back into Manifold 9. All projection parameters are again correct, unchanged from before.

But when this import is added to the same Map, it is further offset to the south--now about 31m compared with the drawing.

That seems very mysterious.


(4)

I reimport the TIFF exported from Manifold 9 into Manifold 8.

The projection shows as plain Transverse Mercator, and with datum WGS84 rather than NAD83. The change to plain TM should not cause misalignment, but the change of datum could be a minor problem.

(This is largely a red herring. It is a limitation in the way Manifold 8 reads projection information in a GeoTIFF header. This limitation is familiar to anyone working with Transverse Mercator imagery from other parts of the world--New Zealand Transverse Mercator in my case. The usual fix is to check the projection after import to Manifold 8, and adjust as necessary.)

Now reassign the correct projection to the imported image in Manifold 8 (using the original image as the source), and add it to the Map. Again it shows with an offset southwards of about 31m.


Results are the same using Manifold 9.0.167.0 and in 9.0.167.3.

Dimitri


7,413 post(s)
#12-Jun-18 06:33

The image is shown about 3m south of the aligned position in Manifold 8.

A wild guess without looking into it in detail (traveling right now):Is the datum the same as reported in the JSON string of the coordinate system actually used?

If not, here is another wild guess: could this or (3) be related to how the data is centered, that is whether in the center of the pixel or the corner? Consider this discussion from the Example: Link NLCD using Scan Raw Binary File topic:

The Local scalevalues came from the Abscissa_Resolutionand Ordinate_Resolutionvalues, both 30, in the delaware_fgdc.txtfile. The Local offsetvalues came from the Upper Left CornerX value and the Lower Right CornerY value in the delaware.txtfile, subtracting 15from both. We do that because the data set is centered on the middle of a 30 x 30 size pixel, but Manifold aligns rasters to the lower left edge. Therefore, we must subtract half a pixel's worth of meters, 15, from the local offsets to match alignment in Manifold with the alignment used by the raw data.

vincent

1,972 post(s)
#12-Jun-18 13:26

Pixels are 1 meter Dimitri. That cannot be that.

vincent

1,972 post(s)
#12-Jun-18 14:35

I don't know what I did while preparing the subset. The offset is southward in the subset. In my main project, the offset was northward. I'll look into the datum and report if I can get the expected output

tjhb
10,094 post(s)
#12-Jun-18 18:46

That is useful.

To me it is starting to sound as if there is a problem in code Manifold 9 uses when it has to retile an image (change received or output tile size). Something like the accidental use of a one-based Y index that should be zero-based.

vincent

1,972 post(s)
#12-Jun-18 21:07

I'll post new data soon. For now, here is the workflow. I did it again.

1- Importing a surface and a drawing into M8.

2- Setting MTM 7 / Nad 83 Canada projection for both. Both items have been exported from ArcGIS with this projection.

3 - Saving the project. Importing it into M9. Projections seems ok to me. Details are good. Both layers are lined up.

4- Exporting the drawing and the image from M9.

5- Importing them again in M8. The drawing now is labeled with Transverse Mercator Projection and Geodetic Reference 1980 Datum. The surface is labeled with an Orthographic projection and a WGS84 datum.

6 - Assigning MTM 7 projection with Nad 83 Canada Datum for both. The drawing is lined up. The image is offset northward about 10 meters.

vincent

1,972 post(s)
#12-Jun-18 21:11

Test data

https://drive.google.com/open?id=18IgCCEkk_jMPDITwWVGpN3U6vlxJEk--

Dimitri


7,413 post(s)
#13-Jun-18 06:48

Thanks for the link. It seems to me there are two sets of questions:

1. What happens when you import data written by Arc directly into 9? Does it import correctly with everything lined up?

2. Multi-step interoperability issues. What happens when you do a multi-step chain: Arc->8, 8->9, 9->8, 9->Arc, etc.?

It seems to me that we first must answer 1. above, for these reasons:

a) 9 users who want to import files written by Arc do so directly. They do not first import into 8 and then import from 8 into 9.

b) If 9 imports the Arc-written data correctly, the primary task is done.

c) If 9 does not import the Arc-written data correctly, that can be fixed in a day.

d) If the interest is not getting Arc data into 9 but instead is testing interoperability between 8 and 9, knowing what happens Arc -> 9 and Arc -> 8 helps narrow down problems when doing 8 -> 9.

If we assume the initial export from Arc was correct (a risky assumption, but let's start with that), the first thing to verify is if pairwise interchange Arc -> 8 and Arc -> 9 are both correct.

I've asked before and for some reason have not gotten an answer: What do you mean by "Everything is fine in M9, all is aligned" ?

When importing the sample data set from the link, there are no dots in the c914_mnt raster against which alignment of the points in the buf drawing can be checked. I see no obvious way to tell if in the direct import into 9 that c914_mnt and buf are correctly aligned or not.

Overlaying c914_mnt with partial transparency over a google terrain layer seems to indicate it is in the correct place, but not to a level of detail where you can say plus or minus 1 meter it is exactly correct.

So, let's rephrase that key question: when you import the shapefile and the tif directly into 9, are they imported correctly, does everything line up as expected, and, how can you tell whether everything is correctly aligned or not?

Attachments:
test_data_in_9.jpg

vincent

1,972 post(s)
#13-Jun-18 14:05

Hi Dimitri,

You asked for the original data. That is what I sent. I pass from Arc to M8, because I need to modify the surface. I use queries in M8 to do so. The circles in the drawing are selected and the selection is transfered to the surface. The value of the selected pixel is changed by the queries. I go in M9 after, because it can export the image directly in .tif format. M8 cannot export a surface in .tif format. Using ESRI grid is longer to export from M8 and longer to import in Arc and it needs to be exported again in .tif format once in Arc. M9 allows to save a lot of time, but something in the image export or import seems odd.

About your question 1 :

I imported the original image in M9. The detected initial projection is Transverse Mercator. The detected Datum is Lat/Long WGS84.

I made a shapefile in Arc that contains a line which follows a river, visible on the image. I imported the original shapefile in M9. The detected initial projection is Transverse Mercator. The detected datum is Nad 83 (CONUS).

I modified both datum to NAD83 Canada, because this is the original datum. I made a map with Tranverse Mercator, Nad 83 Canada datum. You can see the parameters on the attached picture.

You can see on the attached picture that the river and the image are not lined up. The black vector line is the South-East side of the river. The offset of the image is southward (or maybe the shapefile is the one that is wrongly placed ?)

I did the whole process again, but this time applying the real coordinate system of the inputs (Canada NAD83 MTM zone 7) to all components. Result is the same. Offset is the same.

Attachments:
misaligned.jpg

Dimitri


7,413 post(s)
#13-Jun-18 15:28

You can see on the attached picture that the river and the image are not lined up. The black vector line is the South-East side of the river. The offset of the image is southward (or maybe the shapefile is the one that is wrongly placed ?)

Yes, I can see. Suppose you just take the shapefile as exported from Arc, and the image as exported from Arc, and import both into 9. Do they correctly align if you don't change anything (no changing of datum, etc.)?

If a drawing exactly overlays a raster in Arc, and then if Arc exports that drawing as a shapefile correctly, and if Arc exports the raster as a .tif correctly, then both should import into 9 exactly correctly and overlay just like they did in Arc, with no need to manually change anything.

If that does not happen, that is a bug in 9 to be fixed. If that is what is going on, please file a bug report and offer to provide sample data that shows the bug.

If a vector and a raster align perfectly in Arc, but then when you export them from Arc using routine formats like a shapefile and a tif, they should align perfectly upon import into 9. If they do not, that's an error exporting from Arc, an error importing into 9, or both. With samples as exported from Arc, Engineering can investigate and fix any errors immediately.

vincent

1,972 post(s)
#13-Jun-18 16:03

Suppose you just take the shapefile as exported from Arc, and the image as exported from Arc, and import both into 9. Do they correctly align if you don't change anything (no changing of datum, etc.)?

No, they don't. They do in M8.

I'll investigate the export process from Arc before filling a bug report. ESRI is creating more bug than new features these days...

I'll keep posting here about it.

Thank you.

vincent

1,972 post(s)
#13-Jun-18 16:55

I loaded the full original DEM in M9, and it is aligned with the drawing.

Then I removed any Arc's auxilliary file created in the export process (we don't keep aux. files with the full DEMs) I loaded the TIF without aux. files in M9 and the problem is solved.

Any hint about what can be wrong ?

Auxilliary files are : .tfw, .xml and .ovr (for pyramids). Which one M9 reads ?

Dimitri


7,413 post(s)
#13-Jun-18 19:27

Auxilliary files are : .tfw, .xml and .ovr (for pyramids). Which one M9 reads ?

I had to look that up:

Some file formats utilize accessory files such as "world" files, or .PRJ files to convey coordinate information for formats, such as shapefiles, which do not automatically convey coordinate system information. When reading formats Manifold will automatically read coordinate system information from accompany files, if they exist, in the following order: "world" files, .PRJ files, .XML files (Manifold System 8.00 accessory file) and .MAPMETA files (Manifold accessory files). In the event of conflicting coordinate definitions the last read file wins, since the read order is in order of more rigorous definitions.

However, if Arc wrote the internal tags as a GeoTIFF I would have expected that the internal tags within the GeoTIFF would take precedence over any external .tfw or .xml files. So this is puzzling:

I loaded the TIF without aux. files in M9 and the problem is solved.

That could not have happened, I don't think, if the tif were not a GeoTIFF.

My totally inexpert guess is that something in the auxiliary files does not agree with what the tif says. 9 should be able to handle that. I think it is worth reporting.

Thanks a lot for your patience!

tjhb
10,094 post(s)
#13-Jun-18 19:43

Regardless of what happens with data out of Arc, with or without a .tfw worldfile (interesting), there are still at least two issues (which may or may not be the same bug) using just Manifold (8 and) 9.

See my points (2) and (3) above. Easily reproduced.

vincent

1,972 post(s)
#13-Jun-18 20:06

I have to leave. I'll resume testing/reporting on Friday.

Dimitri


7,413 post(s)
#13-Jun-18 20:30

Easily reproduced.

If 9 does not correctly consume something 9 itself has written, that's a bug. If it can be reproduced it is best to file a bug report, providing example data. I file a lot of bug reports and I'm always happy to file another one, but it is usually best for those people who are closest to a specific issue to report.

Likewise with 2. The ideal is perfect interoperability 8->9. The problem could arise with how 8 consumes anomalous data, how 8 exports such a situation, or how 9 imports that, or any combination of all three. Real world bug reports with example data help a lot.

tjhb
10,094 post(s)
#13-Jun-18 21:17

I will file a report referring to Vincent's thread.

adamw


10,447 post(s)
#14-Jun-18 10:01

I am late to the thread.

(2) looks very much like our bug. I get a different shift, but still a shift, the image read from 8 seems to be put into the wrong place.

We will investigate this, then check the rest of the thread and I will try to report the results here.

tjhb
10,094 post(s)
#19-Jun-18 03:16

Thanks Adam.

I made a simpler test file which I hoped would show the same issue. Instead it seems to show a different issue (somehow related?).

The Manifold 8 project has two images and two surfaces, one each at 128x128 pixels and one at 160x160 pixels. (Sizes chosen to need one even tile, and one tile plus residue, in Manifold 9, at the default tile size of 128x128. I thought I might get a different result with one size compared with the other. As it turned out, not really.)

There is also a grid of lines and buffered points, used to mark pixels in the rasters for comparison (as Vincent did).

Everything is in the default Manifold 8 coordinate system, Orthographic/WGS84, with local offsets of 0, local scales of 1.

Opening the project in Manifold 9.0.163.3 shows...

The Rect properties for each 128x128 image should be [ 0, 0, 128, 128 ], and for the 160x160 images [ 0, 0, 160, 160 ]. Instead we have [ 0, 32, 128, 160 ] and [ 0, 32, 160, 192 ] respectively.

We see only part of each image--there are 32 pixels missing along the upper and right edges. In both X and Y, even though the Rect values in X are apparently correct.

On the other hand, correcting the Rect properties does not fix the issue--apparently no change.

And yet... here the marks in the images line up exactly with the marks in the drawings.

Attachments:
128x128_m8.png
128x128_m9.png
160x160_m8.png
160x160_m9.png

tjhb
10,094 post(s)
#19-Jun-18 07:32

Sorry, I meant to attach the Manifold 8 project and thought I had. Here it is.

Attachments:
20180619 test rasters m8.map

tjhb
10,094 post(s)
#20-Jun-18 23:38

Results for this issue in 9.0.167.4:

Rect properties are now correct, [ 0, 0, 128, 128 ] and [ 0, 0, 160, 160 ].

Images now display differently, still not correctly. There are now bands of 32 empty pixels at the bottom and right edges.

Attachments:
128x128_9.0.167.4.png
160x160_9.0.167.4.png

tjhb
10,094 post(s)
#21-Jun-18 00:55

In 9.0.167.4, the symptoms that I was calling issue (2) above no longer appear.

The points and point markers in Vincent's Manifold 8 project now line up, when it is opened in 9.

That seems to be fixed.

adamw


10,447 post(s)
#21-Jun-18 16:40

Thanks for this other test, it steps onto a slightly different issue. We will fix it.

vincent

1,972 post(s)
#05-Jul-18 14:05

Everything seems to be fixed. However I'd like to report the following issue (already reported upper here http://www.georeference.org/forum/t143839r143997#143865). I attached files to see the problem and to test it.

Importing an image in M9 with or without auxiliary is leading to different results. The provided .tif has been produced with ArcGIS 10.6.1. If I import the .tif with the aux. files, run a Traces Areas transform and export the resulting shapefiles, the shapefiles is off northward by 5 kilometers. Not using the .aux files leads to correct results. Initial coordinates system is ok and used in both processes and exports.

Attachments:
fire_example.pdf
image_feu.zip

tjhb
10,094 post(s)
#10-Jul-18 01:21

Everything [else] seems to be fixed.

Not quite. This issue is also still present in 9.0.167.5. A fix is no doubt coming.

adamw


10,447 post(s)
#01-Aug-18 09:54

We fixed this. The fix is coming in 9.0.167.9 in a couple of days.

tjhb
10,094 post(s)
#10-Jul-18 01:26

If I import the .tif with the aux. files

What do mean by "the aux. files"?

There is one file called image_feu.tif.aux.xml, another called image_feu.tfw.

Which? Or only both?

tjhb
10,094 post(s)
#10-Jul-18 02:17

It looks as if it is the world file, .tfw. I think the .aux.xml file is a red herring.

To reproduce, open two instances of 9.0.167.5:

(A)

Import image_feu.tif, from a folder that also contains image_feu.tfw.

Open mfd_meta. Inspect 'image_feu Tiles' > 'FieldCoordSystem.Tile'.

EPSG:32187,mfd:{ "LocalOffsetX": 350165.3977791705, "LocalOffsetY": 5437603.857325403, "LocalScaleX": 50.368675765618775, "LocalScaleY": 50.3093835835618 }

(B)

Import only image_feu.tif, that is, from a folder where that is the only file.

Open mfd_meta. Inspect 'image_feu Tiles' > 'FieldCoordSystem.Tile'.

{ "Accuracy": 4, "Axes": "XY", "Base": "NAD83 (EPSG:4269)", "CenterLat": 0, "CenterLon": -70.5, "Eccentricity": 0.08181919104281579, "FalseEasting": 304800, "LocalOffsetX": 350165.3977791705, "LocalOffsetY": 5442584.48630016, "LocalScaleX": 50.3686757656, "LocalScaleY": 50.3093835836, "MajorAxis": 6378137, "Name": "NAD83 \/ MTM zone 7 (EPSG:32187)", "Rect": [ -172.54, 23.81, -47.74, 86.46 ], "ScaleX": 0.9999, "ScaleY": 0.9999, "System": "Transverse Mercator", "Transform": "Molodensky-Badekas", "Unit": "Meter", "UnitScale": 1, "UnitShort": "m" }

The most obvious difference is that in A we have plain EPSG plus local metrics, while in B we have many more things specified.

In particular:

  • In B we have an extra parameter "Accuracy". Does this indicate an export from the geodatabase, with resolution/precision set to 0.0001? I can't test, since I don't use ESRI. In any case, I expect that this parameter is ignored by Manifold 9.
  • More importantly, while the LocalOffsetX values match exactly, the other metrics differ:
    • LocalOffsetY: 5437603.857325403 ≠ 5442584.48630016
    • LocalScaleX: 50.368675765618775 ≠ 50.3686757656
    • LocalScaleY: 50.3093835835618 ≠ 50.3093835836

Ideally these should all match exactly. The differences in the LocalScale* values look like rounding/truncation/representation errors or limitations. OK.

But the LocalOffsetY parameters look worrying. The difference is 4980.628974757, closely matching Vincent's measured offset of 5 km.

So, an ESRI problem or a Manifold problem?

tjhb
10,094 post(s)
#10-Jul-18 03:47

It is worth noting that the pixel scale for this image is not the same in X and Y. That may be material.

tjhb
10,094 post(s)
#10-Jul-18 04:53

It would be good to test this some more, but I have sent in a bug report in the meantime.

adamw


10,447 post(s)
#01-Aug-18 10:15

This is our problem.

This particular image should import similarly with and without the world file and it does not. We will try to fix this in 9.0.167.9.

Thanks a lot to both you and Vincent.

Dimitri


7,413 post(s)
#12-Jun-18 14:59

Just been able to download and try...

I imported a drawing and a surface in M8. Then I saved the project and open it in M9. I exported the drawing and the surface (now an image). The projection has been defined in M8 and is the same for both. Everything is fine in M9, all is aligned.

When I opened the Release 8 .map saved within 20180611.zip in Release 8, the Buf Drawing and Mosa 2 raster align in the Map. When I opened the the Release 8 .map saved within 20180611.zip in Release 9, the Buf Drawing and Mosa 2 raster were not aligned but were slightly off. When you reported "Everything is fine in M9, all is aligned" what did you do?

This definitely needs looking into and should be reported. Out of curiosity, do you have the original raster and the original drawing that were imported into 8? What happens when you import those into 9?

vincent

1,972 post(s)
#27-Jul-18 16:38

I tought this was solved. Problem again today. See both jpeg. There is an offset of around 23m.

Project done in M8, then imported in M9. In M9 (see jpeg), circles in the drawing line up with image. Tiff is exported from 9 and imported in 8 : offset (see other jpeg).

Original project in M8 is attached. Image has been reduced in size for upload purpose.

I tried with the initial coord. sys. in M9. And also by changing to EPSG 32187. Same issue.

Same offset in ArcGIS 10.6

Tested with 167.7 and 167.5.

Attachments:
back_in_M8.jpg
m9.jpg
mosa3.zip

vincent

1,972 post(s)
#27-Jul-18 17:20

original shapefile attached. Original .tif in the next post.

Attachments:
buffers.zip

vincent

1,972 post(s)
#27-Jul-18 17:27

Subset of the original image/surface :

https://drive.google.com/open?id=1wTXWmGBF164QPdAh8LbaueJIMTnPAYWb

adamw


10,447 post(s)
#01-Aug-18 09:18

We cannot reproduce the issue.

If we open MOSA3.MAP in 8, we don't see a shift similar to the one in BACK_IN_M8.JPG, we see the vector objects being right over the dark pixels in the raster. If we then open the same MAP file in 9 (9.0.167.8), this keeps, the vector objects are still right over the dark pixels in the raster.

Example image, 8 on the left, 9 on the right, the difference in shade comes from different formatting:

If we import the SHP and the TIFF into 8, there are no dark pixels in the raster to see whether they overlay correctly, but the relative position seems to carry over into 9 unchanged as well.

Attachments:
m8-m9.png

vincent

1,972 post(s)
#08-Aug-18 18:26

Again today, new project (168.0 version). I imported the surface back in M8, after exporting it from M9 in tif. The offset is 30m northward.

How can I delete all the favorites coordinates system from M9 ? I have 4 that are greyed out.

adamw


10,447 post(s)
#10-Aug-18 15:27

Will check the export shortly.

If we could get back for a second to an earlier issue with MOSA3, here is what I am doing:

Launch Manifold 8. Open MOSA3.MAP from MOSA3.ZIP above. Open Map. I see this screen:

I then zoom in closer to the second point and see this screen:

Your file BACK_IN_M8.JPG shows that there should be a shift, but I cannot see it. Is it that MOSA3.MAP is the wrong MAP file and does not exhibit the issue? Or do you see a different picture than I do when you open it on your machine?

Attachments:
shot-1.png
shot-2.png

adamw


10,447 post(s)
#10-Aug-18 15:29

Forgot.

You can edit the list of favorite coordinate systems by pressing the button at the right end of the coordinate picker control and selecting Edit Favorites in the menu.

adamw


10,447 post(s)
#10-Aug-18 16:41

OK, we reproduced the shift during the export. This is our issue (the export to TIFF should adjust the coordinate system written into the file but does not). We will fix it.

vincent

1,972 post(s)
#10-Aug-18 20:27

Great !

What I found strange is that it was fixed at 167.4 I think and was erroneous in the next versions.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.