Subscribe to this thread
Home - General / All posts - objects inverting themselves when added to a Map
TomG29 post(s)
#12-Jan-17 19:58

I'm having a strange problem never experienced before:

Creating a simple map consisting of 2 separate elements: a property boundary (simple polygon drawing) overlaid on a topographic map (raster image).

- Both of these elements display properly in their own window.

- Either of these elements display properly when it is the only element in a Map.

- However when the two elements are combined in a Map, one of them is inverted.

- The element that is added first retains its normal orientation; the element added to the first is shown inverted (upside down and backwards).

- e.g. - Create the Map with just the image, and the drawing is inverted when added. Create the map with just the drawing, and the image is inverted when added. If both elements are selected when creating the Map, then the drawing is inverted and the image is correct.

Both elements display geospatially-correctly (when not inverted).

The raster topo map is a GeoTiff from Canadian Dept of Natural Resources - I have used this data source many times before without problem.

I tried starting over with a fresh instance of Manifold and new file, with the same result.

I've generally been able to work through my Manifold problems myself in the past, but this one is a head-scratcher for me.

For reference, the computer is Win7, i7 4-core 3.4 Mhz CPU, Quadro GPU, 24 GB RAM, 256 GB SSD. I don't think that's a limitation in this case though - this is a very lightweight task. There is obviously something strange happening when combining the image with the drawing element.

Any suggestions are appreciated.

Thanks.

atomek

422 post(s)
#12-Jan-17 20:08

have a look at your component's projection settings and see if there's something out of ordinary (i.e. 0, 0) in "false easting/northing" fields

Dimitri


7,413 post(s)
#13-Jan-17 07:26

Almost certainly, the projections parameters for one or both of the components is wrong. Here's how to find out:

1. Link an image from an image server like Google or Bing.

2. Create a map from that.

3. Zoom into the region where the property boundary drawing is located.

4. Drag and drop the property boundary drawing into the map.

5. Does it overlay in the right position? If so, it is *not* the component with the wrong projection. Your task is now to figure out why the GeoTIFF was imported with a wrong projection.

6. If it does not overlay in the right position, then you know the projection assigned to the property boundary drawing is wrong. Fix that. It still could be that the GeoTiff *also* has the wrong projection assigned to it.

But let's begin with the easiest step, since the property boundary drawing will reproject on the fly into the map created using the image server quicker than (presumably) some big GeoTIFF.

Also....

I have used this data source many times before without problem.

... have you used this particular GeoTIFF exactly as imported many times before without problem or do you mean other files downloaded from that web site? If it is the latter, that doesn't help much.

If it is the former, that depends on what you mean by "used" and "without problem." When debugging such things you can't take any small detail for granted. :-)

Here's what I mean by that: the world is full of broken data published by well-meaning people who don't realize they are publishing flawed data because when they use it it appears OK. For example, if something has the wrong projection but it is only used by itself (for example, both components you use look OK when opened by themselves) or in combination with other data sets that are all broken in the same way the user will not notice the data is broken. They go on to publish it to a web site, people download it and from there it multiplies over and over to feed the world's torrent of junk data.

A great example of this is how GeoPackage and plenty of other amateur packages claim to observe EPSG coordinate system definitions but in fact ignore the required XY ordering for coordinate systems such as EPSG:4326, a particularly egregious crime given that EPSG:4326 may be the most frequently-encountered EPSG definition used by people who do not understand EPSG: it is the EPSG code generally used for "latitude/longitude unprojected" data. Gpkg blockheadedness on this is so intense that geopackage actually publishes a broken data set as a "test" case on their GitHub site.

Such errors are obvious in a package that uses EPSG correctly, such as Radian, but if you only use applications that repeat the GeoPackage error or which otherwise ignores the coordinate system definition you'll never notice that you are working with, and possibly publishing, junk data.

Here's a note from the Radian beta that warned users about the issue:

If you try the GPKG dataport with the "small_world.gpkg" test data set that is published on github the drawing comes out rotated 90 degrees.

That is NOT an error in the dataport: it is an error in the small_world.gpkg "test" data published on github... (!)

The specific error in the data is that the drawing layer reports that it is in EPSG:4326. That is an error because coordinate ordering in EPSG:4326 is Y first and X second, hence the rotation.

As inconvenient as it is to have test data that itself is broken, it is a good reminder not to be too credulous that whatever one downloads from the web is OK...

By the way, this particular error is very common. It is amazing how many slackers don't bother to pay attention to key details such as axes ordering defined by a particular EPSG code, which EPSG *insists* must be respected (OGC does too). So the world is full of data sets which claim to be in EPSG:4326 or other (Y,X) systems but which in fact use the wrong, (X,Y) ordering.

Unfortunately for us all, systematic ignoring of coordinate systems standards like the above is becoming more common. But even when using scrupulously accurate software it is still easy to accidentally introduce broken data into workflow. It can happen to anyone and even expertly run sites like the Canadian site or USGS in the US or IGN in France may have some coordinate system bugs in the archives.

mdsumner


4,260 post(s)
#13-Jan-17 11:59

Seems helpful Dimitri, up until "A great example". This is fear, uncertainty and doubt, casting unrelated muck around to lend weight to informed guesses. You're burning to tear them to shreds? Tear us slackers apart who remain silent since we don't really get it? Do it in the open, address it to the people responsible, share the link to where you do that and report back on the discussion. Name them, point to the actual repo, why leave us guessing?

I'm a slacker, I still don't understand if the order you talk about is a visible convention that users will see, or somehow baked into the stream of bytes in the blobs, or what. Only in WKT/WKB? Only in WKB? I don't understand why this can't be illustrated, it's pretty much the way things are done these days. Transparently. We could do this in free software in minutes, pick out a blob here and there, unpack it and format it explain the structure, read it and write it and transform it in different ways and pick it apart, then publish a reproducible document to the web, tear that apart together and provide real understanding. There's no explanation in your rant, no pointing to where this happens and how it manifests.

This is absolutely clearly unhelpful and unrelated to the question posed, and really not what this forum is intended for as I understand it. I can't recommend it to friends, which is unfortunate since there's pretty much nothing anywhere else. I do hope that changes because Manifold is really good and more people should know about it, I can't see this kind of discussion helping that mission whatsoever and I'm confused as to why it's welcome here. Maybe it's ok given the community user agreement, but it certainly doesn't pass the common sense decency test.


https://github.com/mdsumner

Dimitri


7,413 post(s)
#13-Jan-17 15:51

There's no explanation in your rant,

Whoa there! This is a technical forum, not about religion. Sounds like I hit a nerve by mentioning a technical error in gpkg that produces symptoms similar to those in the problem that launched this thread.

By the way, I didn't slam gpkg or point out the technical error that gpkg commits. I simply pointed out that a file which gpkg provides on their web site is, indeed, inaccurate in that it wrongly claims the wrong EPSG code for the data. Getting latitude and longitude backwards is a big error.

I respectfully point out I did provide an explanation. That would be this bit:

So the world is full of data sets which claim to be in EPSG:4326 or other (Y,X) systems but which in fact use the wrong, (X,Y) ordering.

GPKG does indeed ignore EPSG-required coordinate ordering and it really does write what it claims to be EPSG:4326 in X,Y order instead of the required Y,X order. That is simply a technical mistake, nothing more and nothing less.

In layman's terms, it gets latitude and longitude backwards when using that code. Noting that GPKG gets that wrong or that a file published on their site as a "known good" test sample is indeed in error, is a simple technical observation, not a commentary on religion, politics, decency, etc.

I still don't understand if the order you talk about is a visible convention

No, it's not a convention, it is part of the EPSG standard that gpkg claims to use. As noted in my post it is what EPSG *insists* must be respected in their standard.

Errors like misapplying EPSG codes are exactly the sort of errors which result in data sets appearing flipped in various ways, the problem in this thread, hence the particular example.

I know you are a fan of gpkg but getting angry at the messenger who points out a simple technical mistake in gpkg is not what this forum is about. But if you believe I stated something that is technically incorrect you can check that for yourself by comparing what GPKG does to what the standard says it should do.

Read the EPSG standard, study what gpkg does and you'll see how it gets it wrong. Not too difficult in this case given the obvious blunder of getting latitude and longitude backwards. Use the sample data set they provide that I cited as a test case so others can help you or duplicate your work. I have no idea why gpkg doesn't fix this error but it is not my cross to bear to fix errors in other people's programs.

Instead, my cross to bear is to provide technical means so people can use Manifold to deal with errors in broken data, like flipped coordinate ordering.

As I recall, Radian added a dataport for gpkg at your request, only to discover in testing, using gkpg's own sample data, that gpkg uses EPSG incorrectly. Instead of simply dropping gpkg, Manifold looked into the matter and learned that it is very common for people to misuse the EPSG standard, and that as a result the world's data archives contain much data that claims to be in EPSG:4326 but is not. It's a real problem.

There are many people who would take a more rigorous approach to standards and say that if the data is broken it is best not to import it. But that's not what Manifold did in this case. Instead,Radian added an option to flip coordinate ordering for data that gets it wrong. That's doing a good turn for users so they can deal with errors in data, which we all know happen all of the time, and convert the data into correct EPSG ordering.

That's a perfectly fair technical solution to a simple technical problem. Nothing to get heated up about.

mdsumner


4,260 post(s)
#13-Jan-17 11:41

TomG can you share the drawings? At least enough of them, i.e. the Map with the drawing and one polygon showing the problem, and when you open the drawing alone you see it normally.

I just want to see the problem for myself and dig into it enough to see transparently exactly what's going on. The issues are explained well in Assign Projection in the manual, but really you need to carefully check each component independently and verify it against known targets for yourself, before comparing to layers like this.


https://github.com/mdsumner

TomG29 post(s)
#13-Jan-17 19:19

Thank you for everyone's response.

I'm diverted to a different project for today, but I will try to digest the various comments and recommendations over the weekend. If I'm unsuccessful, then I can post the problematic file and see what the greater and more experienced minds than mine think. Thanks, Tom

TomG29 post(s)
#14-Jan-17 18:38

Thanks to all of your suggestions, I think I identified the reason for this problem. I believe Dimitri is indeed

correct in flagging bad projection data.

First I must correct a statement I made in the first post: I have indeed used GeoTIFFs from Natural Resources Canada successfully, however this problematic map actually came from a different source - a Provincial government agency called GeoBC. I have in fact not used map data from this source before.

The NRCan maps come as a single GeoTIFF, which I believe contains the projection data intrinsically. The GeoBC map is supplied with a TFW file for projection, and a text file with projection info that I can read.

I reviewed your posts, re-read all the Projection sections in the manual, and looked at the projection information for the suspect GeoTIFF map. I appears that there are a number of errors in the projection info that are apparent even to me, who is not a projection expert.

The map area is actually in UTM zone 10 and is entirely in the area of the -123° meridian, however the GeoTIFF projection information showed zone 11 and center meridain of -117° even though the map corners are shown as -123.xx°.

I used the Assign Projection dialog to change the UTM zone and Center Lat and Long. This fixed the problem, using a trial-and-error process to visually match the problem map to a known good map (from NRCan). I also had to rotate the whole map by a few degrees to align the meridan lines N-S. I did this by rotating all pixels in the bitmap with the rotate transform. My boundary drawing is exactly where it should be now.

So I think the problem is solved, however I welcome comments about my process - is there a better or more streamlined way to make these corrections? The way I did it seems fine for a few maps, but it would be tedious for a mosaic of many.

The upside is that I learned from the experience: What to look for if I encounter this again, and to not naively assume the projection data is good just because it comes from a big government organization.

mdsumner:

Thanks for the offer to look at the file for me. I think the problem is solved now, but if you are still interested in seeing my specific case, I can upload an example.

tjhb
10,094 post(s)
#14-Jan-17 21:19

Well done for achieving a usable fix.

However, to me at least it seems worth getting to the bottom of the issue.

You've ascertained that the supplier's UTM projection information was wrong.

You've switched to a more likely UTM zone, then made other adjustments, including manual shifts (I think, possibly scales too) and rotation.

Those adjustments shouldn't be necessary. They, and the original symptoms you were seeing (especially apparent XY reversal), suggest that the projection is not UTM, but something else.

My hunch is that the supplied data is in a conic projection, possibly BC Albers or a Lambert Conformal Conic.

This would be quick to try, if you have another moment.

With the original image, as freshly imported into Manifold (no adjustments), try assigning the projection National Grids > Canada > British Columbia. This is BC Albers (EPSG 3005). The built-in parameters are below (no need to enter these manually).

<preset>

<name>British Columbia</name>

<category>National Grids*Canada</category>

<system>Albers Conical Equal Area</system>

<datum>North American 1983 (Canada)</datum>

<centerLat>45</centerLat>

<centerLon>-126</centerLon>

<firstStdLat>50</firstStdLat>

<secondStdLat>58.5</secondStdLat>

<scaleX>1</scaleX>

<scaleY>1</scaleY>

<falseEasting>1000000</falseEasting>

<falseNorthing>0</falseNorthing>

</preset>

Leave local offsets and scales unchanged (in the hope that these are correct as supplied) along with default datum North American Datum 1983 (Canada).

If that lines up with no extra effort, great.

If not, you could also try the Statistics Canada implementation of Lambert Conformal Conic (EPSG 3347). To assign this projection, choose Standard > Conic > Lambert Conformal Conic (the first version), and enter these parameters:

Center Latitude: 63.390675

Center Longitude: -91.86666666666666

1st Standard Latitude: 49

2nd Standard Latitude: 77

False easting: 6200000

False northing: 3000000

Manually assign datum North American Datum 1983 (Canada).*

Leave other parameters unchanged, including local offsets and scales (again, hoping...).

(*It's possible that data uses not EPSG 3347 but EPSG 3348, with the newer NAD83 (CSRS) datum, which we don't have in Manifold 8. No point worrying about that for now.)

[A couple of edits made regarding datums for the Lambert Conformal Conic projection.]

TomG29 post(s)
#15-Jan-17 23:35

Thanks for the additional feedback, tjhb.

The map I have been using is indeed originally UTM projection, however it appears to have the wrong zone and central meridian specified. Those are the parameters I changed manually to make it plot correctly.

There is also an Albers version of the map available which I downloaded to have a look at and compare with your notes. The Albers version also plots incorrectly (inverted), like the UTM version did before I corrected it.

I tried the projections you suggested but it didn't make any difference. By looking at the Albers text file, the problem isn't as apparent to me as in the UTM text file. I may try some further experimentation as I get the time, at least for my own education on the topic, but for now I'm satisfied with the fix I was able to do to the UTM map.

I have attached the text files for the UTM and Albers maps for your interest. Note that in the UTM file, the zone is shown as 11N when it should in fact be 10N, and the "Central Meridian" is shown as -117.0 when it should be -123.0. These are the parameters I changed using Assign Projection.

Attachments:
092J063 albers.txt
092J063 utm.txt

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.