Subscribe to this thread
Home - General / All posts - M9 Import ECW file - Projection
dchall8
614 post(s)
#01-May-19 21:34

This topic picks up where a previous topic about importing JPG files left off. I received an ECW file from the EagleView contractor. When I open the ECW in ESRI, it seems to be identical to Google Satellite imagery. When I open it in Manifold 9, it is different as shown in the following image. The really good news is that the file did not appear inside the Arctic Circle. But not all was perfect.

For reference I used Google Earth and the ground based walkway at the middle school to draw a faux monument. The red lines are the cross hairs. I also linked to the ECW and left the projection alone. In addition I imported the ECW and edited the projection as Tim suggested in the linked topic. I used the same approach and to abbreviate the the result I call it EPSG 2278. The middle image below shows the ECW file as linked with no alterations to the projection. The right side image shows the ECW as modified by the edited projection.

The PRJ file that came with the ECW is attached. I tried to upload the ECW to Imgur, but it doesn't like ECW files. Got any ideas how to bypass that restriction? The file is 239,221 KB. That's quite small compared to the rest of the files they sent.

There are a couple of puzzles to me.

  • Why does the raw file import perfectly into ArcMap and not Manifold 9?
  • Why does the adjustment to the ECW projection not get it quite perfect?

Attachments:
ECW Imported and Linked to M9.jpg
TXBAND019039.prj

Mike Pelletier


1,602 post(s)
#01-May-19 22:49

Cannot answer your questions about the internals, but can say that Manifold is aware of the inability to change the initial assigned projection to a linked ECW file. If it could, looks like that would solve the problem. Your ECW links at least close by. Mine comes in across the globe somewhere :-)

tjhb

8,771 post(s)
#01-May-19 23:05

I'll come back with more detail, but can you try EPSG 2919 instead of 2278?

tjhb

8,771 post(s)
#02-May-19 02:32

If I understand it, the difference between EPSG 2278 and EPSG 2919 is that, while they use the same projection and the same ellipsoid, their ellipsoidal coordinates are referenced to slightly different datums. EPSG 2278 uses the original NAD83 (EPSG 4269) and references a standard Earth centre. EPSG 2919 uses the later NAD83 (High Accuracy Reference Network) datum (EPSG 6152), referencing a centre specifically adjusted for its own State Plane system (in this case Texas South Central).

The .prj file you attached refers to "NAD 1983 StatePlane Texas South Central FIPS 4204 Feet", which is the name of an ESRI definition that should correspond to EPSG 2278. So if EPSG 2919 is a better fit, that may indicate a metadata error. (Happens.)

Second question. Can you say more about your workflow? I'm curious how you drew the crosshairs (great idea by the way), in which software and in what projection? Then how you shared them between Google Earth, ArcGIS and Manifold? How do we know that the crosshairs themselves are correctly positioned, in all three screenshots above?

dchall8
614 post(s)
#02-May-19 20:30

You raised concerns that made me double check the cross hair lines. I have historically relied on Google Earth imagery knowing that there are imperfections in the data. I like it because GE makes a good viewer for our taxpayers to use at home. I export the KML file for the county once a week and make it available to download from our website. Once I got the HTML coding into the KML format, it makes it easy to export and view details about the property. So, good or bad, GE is my go-to master background. Using the cross hairs created in Manifold 8 (State Plane Texas South Central - North American 1983 (mean for CONUS)) I can export them as a KML file and look at them in GE. Then I can go back in time to see how the imagery is different from image to image. The images definitely move around a little.

However for the cross hairs I used in M9, I created those in M9 State Plane - TexasSouth Centraln(NAD 83, feet). You have to be careful using cross hairs to trace a line that is on the surface of the Earth. If you use a fence or top of a building, for example roof lines, ridges, etc., those are not appropriate to make cross hairs due to parallax in the imagery. Sidewalks are good because they rarely move around, but sometimes they are covered with a shadow or trees. Sometimes a sidewalk is not straight, so you have to look around. Hard sports courts like tennis, basketball, or even 4-square are good because they rarely get repainted and are usually out in the open with no trees. Also they are practically guaranteed to be level from one end to the other. The longer lines you can make the better.

So I redid the cross hairs and took notes. First I imported the ECW into a fresh M9 project. Then I brought in the Google Satellite view and created a map. I dragged the ECW and the Goog Sat image into the map. Then I created a drawing, assigned it the Texas South Centraln(NAD 83, feet) projection, and dragged it onto the map. With the Goog Sat imagery in view, I found the school tennis court, drew lines in the Drawing layer and colored them green. Then I duplicated the drawing and table and dragged it into the map. I turned off the Goog Sat layer so all I could see was the ECW. Of course it was way off, I'm guessing 30 feet to the south in the native imported projection. I changed the projection to EPSG 2919 and refreshed. Then I copied the lines from Drawing and pasted them into Drawing 2 and adjusted them to align with the tennis court lines as viewed in the ECW. The ECW cross hairs are colored yellow. The ECW has 3-inch resolution so I was able to tune up the end points of the lines slightly. In M8 I would use the crossed lines to make a point and measure the distance and direction between the two points, but M9 doesn't have that transform yet. Instead I selected all the lines and did a Bounded Areas transform to make an area with vertices to measure between. The offset from green Goog Sat crossing point to the yellow ECW point is 2.816 feet at 167.8236 degrees. The attached image is not the easiest to see.

Images

1. Raw Goog Sat

2. Goog Sat with green lines at center service line and the service line

3. ECW image with the Goog Sat green lines. Really hard to see at this size.

4. Goog Sat with green lines, yellow lines, and a gray area inside the four lines.

The image tagged with a 2 at the end of the label shows the tennis court closer in.

In my initial post on this topic I complained that even after adjusting the projection in M9, the ECW image still did not align with the image in ArcMap. Ahem. Well, testing that theory shows I was premature in my judgement. The ArcMap attachment is basically identical to the image out of M9, so...my second question above is no longer a question. To make the shape file I exported the drawing from M9 and dragged it into ArcMap. Then I dragged the ECW image in. Two points about ArcMap are the ECW file opened instantly and in the proper location without fiddling with projections. The ECW alone contained enough information to put it in place. Importing the ECW into M9 took 168.113 seconds.

Attachments:
ArcMap Goog Sat and ECW comparisons Tennis Court.jpg
Goog Sat and ECW comparisons Tennis Court 2.jpg
Goog Sat and ECW comparisons Tennis Court.jpg

Mike Pelletier


1,602 post(s)
#02-May-19 20:54

Sorry if you already get this but just in case. Arcmap is linking the data. M9 can also link and should be instant as well, but cannot alter projection at this point. Importing the ECW in M9 converts it to M9 internal format with full control over projection and modifying the image.

dchall8
614 post(s)
#02-May-19 21:33

That makes sense, speedwise. Thanks. Yes the M9 link is very speedy. Started doing that with LiDAR data, but as soon as you try to do anything with it like style it, M9 does the long import.

Dimitri


5,460 post(s)
#05-May-19 17:40

but as soon as you try to do anything with it like style it, M9 does the long import.

You can style images from linked data sources. There's an example at Style Applied to an Image Server Image. That also works for images linked from ECW and other data sources.

There are advantages and disadvantages to both linking and importing. If you want total control, import. If you can live within the limits of what is possible leaving the data in the original format and linking to it, link.

adamw


8,584 post(s)
#06-May-19 11:05

In my initial post on this topic I complained that even after adjusting the projection in M9, the ECW image still did not align with the image in ArcMap. Ahem. Well, testing that theory shows I was premature in my judgement. The ArcMap attachment is basically identical to the image out of M9, so...my second question above is no longer a question.

We'll try to get rid of the first question as well - by reading PRJ / similar files associated with ECW, if they exist, and using their info to override the coordinate system stored in ECW. ECW allows storing coordinate system info, but if it, says, stores wrong info, that's not easy to fix, so being able to override it by using a PRJ or something similar is helpful.

Dimitri


5,460 post(s)
#06-May-19 12:25

by reading PRJ / similar files associated with ECW, if they exist, and using their info to override the coordinate system stored in ECW.

The fundamental problem, it seems, is an ECW that specifies a slightly inaccurate coordinate system.

I'm not 100% convinced that is what is going on in this particular instance (I've uploaded the ECW with a detailed report to tech support, so they can see if Manifold is failing to import some coordinate system details from the ECW that it should). But based on the extensive detail in the coordinate system (drill into the base, etc., etc., to see) you get with the import it seems that Manifold is correctly harvesting all coordinate system info the ECW says. So let's go with that as a working hypothesis.

As I mentioned in my other post, in the past there have been instances of ESRI people using PRJ files to get around ESRI inability to read coordinate systems specified by the ECW. Some of that may have been because ECWs have a limited ability to specify coordinate systems.

An ECW does not have the internal ability to specify all projections that can be handled either by ESRI or by Manifold. I suspect the ESRI community's use of PRJ files as a non-standard sidecar file to override whatever is in the ECW evolved as a way to specify projections used that the ECW standard itself could not handle. That is just a guess. [From a purely anthropological perspective it's a plausible guess, since ESRI people evolved PRJ files to get around the inability of shapefiles to specify projections... so why not apply the same kludge to other formats?]

If that guess is true, using a PRJ file to override what is in the ECW is a wretched kludge, a truly irresponsible way of using ECW, because it means you are willing to write an ECW that contains fake projection data, that tells a lie. Sooner or later that fake info in the ECW is going to cascade into yet more inaccurate work products created from that wrong ECW file. The right way to use ECW is to re-project your image into whatever projection an ECW can handle, and not to use a PRJ file to override wrong info deliberately placed into the ECW.

I strongly disagree that Manifold should say "we support ECW format" and then ignore what the format says. I understand the argument that if somebody did not want a PRJ to override what the ECW says, they wouldn't put a PRJ in the same folder as the ECW. But that assumes people understand that ECW doesn't use PRJ so having a PRJ is an override. What if in the same folder they had a different image format that does use a PRJ with the same stem, prior to re-projecting into a projection that ECW could handle?

In that case, by assuming the user does not want to respect the ECW standard, automatic use of PRJ to override would get it wrong. Such second guessing of users that when they explicitly say to use a standard, like ECW, they should not be trusted, well, such disrespect for what users say they want to do is a formula for chaos.

Look, if the problem is that an ECW has an error in it, specifying the wrong coordinate system, the solution is to make it easier to fix that error. There are three ways to do that:

a. The existing way: If you think the ECW you have needs a repair to the coordinate system because the wrong one was specified in the ECW file, you choose Repair Coordinate system, drill down to the Coordinate System dialog, and in one click choose the right system. This isn't rocket science.

b. If you want to consume a PRJ file, OK. Add that as another tab to the Coordinate System dialog, so in addition to Standard, EPSG, and Custom, you also have an Import tab. If you want to repair a wrong coordinate system specified by the ECW, you now have a fourth choice, of harvesting a coordinate system from a PRJ file, or from other sources. You could click the Import tab and get a dialog for importing the coordinate system from various different sources: pick some other component in your project, or choose a TFW file, or paste a JSON specification, or... (drumroll...) pick a PRJ file.

c. Less pretty, because it wrecks the simplicity and ease of use of automatic imports using file extensions, is to have a separate ECW import dialog that provides a checkbox captioned: "Override ECW coordinate system with PRJ file."

Personally, I don't like c. because I think it's a mess to wreck a clean, easy to use dialog with a special case control that works only when the format contains fake data. Deliberate misuse of a format is a fairly rare thing, so it seems that b. above would be the better way. Why punish everybody with a complication because a very tiny minority of users deliberately wreck a perfectly good format?

dchall8
614 post(s)
#08-May-19 19:23

I got a little more information from Eagleview today. They..."use QGIS and ArcMap when making the files, mostly QGIS."

dchall8
614 post(s)
#09-May-19 16:48

Here's a new wrinkle. Don't know if this is helpful, but I've been using M9 for these ECW discussions. Today I linked into M8 and it aligned perfectly without fiddling. It is projected as NAD 83 Texas South Central.

Also interesting is that I cannot put the linked ECW and a link to Google Satellite in the same map in M8. Incompatible projections.

adamw


8,584 post(s)
#10-May-19 17:02

The latter is a limitation of 8. The former we will look at, thanks for the note.

Mike Pelletier


1,602 post(s)
#08-May-19 22:53

Yes your items a and b work great for imported images of any type. What is needed is a way to override linked images. I'd be happy with Adam's suggestion of using a prj to override or a new option to override within the software. Importing large ECWs takes too long and creates storage issues.

Dimitri


5,460 post(s)
#02-May-19 10:16

Got any ideas how to bypass that restriction?

Use Google Drive, Dropbox, or Box. All provide free options in the gigabyte range. Find more free file storage/sharing sites by using Google to search for "free file sharing" and similar terms.

Some sites limit the amount of bandwidth for free use, so if you publish a link on this forum and it stays up for years and people from all over the world start downloading that file you might exceed the free limit. That's easy to deal with by putting a link on this forum and then after some reasonable time, say a week, deleting the file from the storage service.

dchall8
614 post(s)
#02-May-19 16:58

Here is a link to the 233MB ECW file on MyAirBridge. The link is good for 3 days.

RAR6 post(s)
#04-May-19 11:39

I downloaded this ecw file - Pict_Band_3-inch_2019.ecw - and using the latest 9.0169.0 I imported the file and I see only a table but no image.

Linking though is OK.

Do I miss something or it is supposed to be that way with ecw files?

Thank you,

RAR

Dimitri


5,460 post(s)
#05-May-19 18:15

I imported it just fine. Could it be the filter box in your project pane is set to just show tables?

By the way, the easy way to test alignment with Google is to just use a layer of Google transparent streets above the ECW, like this:

You can see in the above that the intersection of 11th St and Cherry St in Google is slightly offset to the south form the ECW. It's the same in Bing, etc., as can easily be seen using partial transparency/enhanced contrast/etc for layers.

The ECW imports/links with a coordinate system of...

That is not an EPSG number, it is a Lambert Conformal based on what the ECW specifies. I haven't had time to compare every detail of the PRJ to the coordinate system achieved on import. But a rough guess based on the detail coming in from the ECW import or link, it seems that Manifold is reading what is inside the ECW.

The offset between the ECW and Google/Bing is about 180 feet, which is within the ballpark for an offset issue cause by using different datums. With no info other than what is in this thread, my guess was that ESRI was reading the PRJ to get the coordinate system but what the ECW provides was not exactly the same.

So... I went looking for a coordinate system in the EPSG list that had "Texas" and south Central in the name and I found one. Specify that coordinate system using the "repair" dialog and the ECW lines up perfectly:

The coordinate system used:

As to why the ECW seems to contain slightly wrong projection info, to know that we have to know how the ECW was created.

By the way, if you search the globalmapper forums you'll find cases where people have issues with projection specifications in ECW and ESRI files, where they use a PRJ to get around that. Just guessing, but if ESRI was used to create these ECWs it could be there is a reverse issue where the ESRI software understands what it intends but does not write that correctly into the ECW. Just a guess.

Attachments:
ecwLCC_1.png
ecwLCC_2.png
ecwLCC_3.png
ecwLCC_4.png

RAR6 post(s)
#06-May-19 02:35

Dimitri,

There is not filter. I tried few ecw files and all of them did not load completely in Build 9.0.169.0 and showed only the table - I guess only part of it as it was too fast for normally 2.5gb decompressed file. When I tried to import them using the last Build 9.0.168.12 though, they all load OK.

Thank you,

Dimitri


5,460 post(s)
#06-May-19 04:45

I tried few ecw files and all of them did not load completely in Build 9.0.169.0 and showed only the table - I guess only part of it as it was too fast for normally 2.5gb decompressed file.

I do not understand what you mean by "too fast". I used 9.0.169.0 for the screenshots I posted. All imported OK. Are you using a 32-bit machine or 64-bit machine?

adamw


8,584 post(s)
#06-May-19 10:57

In addition to Dimitri's question, are there any errors in the log window?

RAR6 post(s)
#06-May-19 12:02

Dimitri,

I am using a 64bit windows 10.

Manifold 9.0.169.0 Log reported for the tested file 'No memory' after 1 sec .

Manifold 9.0.168.12 for the same file loaded without any problem after 191.032 sec.

Thank you,

Dimitri


5,460 post(s)
#06-May-19 12:40

I am using a 64bit windows 10.

OK. Could you say a few words about your system? How much RAM? How much free space on disk? How much free space on the disk drive that hosts your TEMP folder? Running a virtual machine?

It could be you had enough space when you ran .12, but then when you ran 169 you were out of space.

RAR6 post(s)
#06-May-19 13:14

Dimitri,

This is the computer specifications:

Processor Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz 2.39 GHz

Installed RAM 4.00 GB

C: 107.6GB free space (holds TEMP folder)

K: 137.9GB free space

I am not running a virtual machine.

I run Manifold 9.0.169.0 first and had that 'No memory'; I then run Manifold 9.0.168.12 to import the same file and it loaded correctly.

Thank you,

adamw


8,584 post(s)
#06-May-19 14:10

4 GB RAM seems very low for a 64-bit Windows 10 system. It seems like we ask the operating system to allocate some memory and the operating system tells us no. Try increasing the size of the pagefile. I am not sure which specific change between 168.12 and 169 might result in different behavior here - this might not be the result of any specific change, but rather related to changes in the execution environment - eg, if there were less applications running in one case than in the other. Try running the 32-bit version of 169, too, this might help in low RAM situations like this one.

RAR6 post(s)
#06-May-19 15:55

Dimitri,

Increasing the size of the pagefile or running the 32-bit version did not solve it.

I probably need to increase my RAM.

Thank you Dimitri.

Dimitri


5,460 post(s)
#05-May-19 18:25

Quick answers (see my other, illustrated post in this thread):

  • Why does the raw file import perfectly into ArcMap and not Manifold 9?

I'm not sure what you mean by "raw file." Importing the ECW that you posted works perfectly in 9. It imports it exactly, as far as I can tell, as the ECW file itself says it should be imported.

That creates an import slightly off from what Google and Bing show. That seems to happen because the projection the ECW says to use seems to be slightly incorrect. Adjust it to the correct projection and it lines up perfectly.

ArcMap does not use the projection the ECW seems to specify. Why it uses a slightly different projection might be because it is adjusting using a PRJ file, or using the name, which is an ESRI name. Software often looks for the first tag that makes sense and then uses that, ignoring all that follows. If Arc is just looking at the name and stopping there, it won't use anything else which might lead to slight differences.

  • Why does the adjustment to the ECW projection not get it quite perfect?

Most likely, because whatever adjustment was used was incorrect. See my illustrated post for the correct adjustment.

dchall8
614 post(s)
#06-May-19 14:58

Thanks for looking at this.

I was hoping that ArcMap would not open it perfectly, so I set it up to give it the least information. I dragged only the ECW into another folder and used that to drag into ArcMap. Would Arc be able to find the PRJ file in another folder?

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.