Subscribe to this thread
Home - General / All posts - M9 Import JPG File - Projection
dchall8
1,008 post(s)
#29-Mar-19 20:59

This is going to be a projection question. I have a bunch of good resolution, .jpg files for our county. They came with three accompanying files (examples attached). I brought them all into M9 at the same time. I would like to merge the image tiles and export as .ecw files for use with other software. The .jpg files are 8.6-18 MB each, so not attached.

The files import quickly but they are not located where they belong. Upon initial import they are in the Sør-Spitsbergen National Park well north of Murmansk. When I reset the projection to the one specified in the .xml file, the images jump to a few hundred miles west of the Galapagos Islands. Changing the false easting and northing don't make a significant difference.

It seems like this is almost there, but something isn't set right to make this property go where it belongs. What am I missing?

Attachments:
jpg import screenshot.jpg
TXBAND007008OrthoSectorTile.jgw
TXBAND007008OrthoSectorTile.jpg.aux.xml
TXBAND007008OrthoSectorTile.pmi

tjhb
10,094 post(s)
#29-Mar-19 21:47

[Partly wrong. Back soon.]

tjhb
10,094 post(s)
#29-Mar-19 22:28

Here's what should work:

Import all the images.

Open each one in turn (we may be able to automate this later...).

For each image, open the Contents pane. (The initial coordinate system will probably be Pseudo Mercator, in red.) Click the Coordinate System button at bottom right.

Choose Assign Initial Coordinate System, then...

Don't choose "US State Plane NAD27 Texas South Central", if that is in your list of favourite projections, because that will mess everything up. This is a trap when importing images. The problem is that the default metrics will override what Manifold has already read from the world file, and your image will wind up in entirely the wrong place...

Instead, from the Assign Initial Coordinate System list, choose Edit Coordinate System.

You'll now be in the Custom tab. Activate the Standard tab instead.

Notice, at the bottom of the panel, that the Metrics are already correctly set (all going well). They should say, in your case, 0.75 ft US, which is the pixel size, followed by two figures in square brackets, which are the X and Y offset values. Manifold should have read these from the worldfile (.jgw in this case). Normally don't change them, but if you want to inspect the detail (or if you do ever need to adjust them), click the XY button to the right, then choose Edit Metrics. (Don't choose Use Default Metrics, unless you really want to set scales to 1 and offsets to 0. Not usually good for images. This is the same mechanisim as in the trap mentioned above.)

In the Standard tab, type part of the projection name, e.g. Texas or texas south, and choose the right projection from the filtered list. Press OK.

That's it. (But again, it seems almost necessary to speed this up when importing a large number of images with worldfiles.)

tjhb
10,094 post(s)
#29-Mar-19 22:47

Correction. I should have said

Don't choose "US State Plane NAD83 Texas South Central", if that is in your list of favourite projections, ...

The point of course is not to choose any projection from your list of favourites, after importing an image.

dchall8
1,008 post(s)
#30-Mar-19 21:30

Contents>Component. Indeed it comes in as Pseudo Mercator.

Following the instructions puts me in the Arctic Circle. Just for grins I went to the Map and looked at the Component pane. I clickedd on the Projection button and selected Repair Initial Coordinate System. I reselected the projection from my list of saved projections and the images popped back to the Galapagos. Having fun with the Projection button, I clicked it again and edited the projection using the Standard tab and selecting the South Texas projection again. I expected to see the image back in the Arctic, but it stayed in the warm waters of the Galapagos.

The image I used this time is 3-inch resolution. I believe the previous one was 9-inch. I will look at Dimitri's approach on Monday.

Attachments:
jpg import screenshot redo.jpg

tjhb
10,094 post(s)
#30-Mar-19 21:42

You should not be looking at any map.

You should be applying the sequence above to each and every image, individually.

After following the sequence for the first image, open it (not in a map), and check the location in the status bar. If that is wrong, stop.

Otherwise, import the other images, and apply the sequence above in each case.

Then you can create a map.

tjhb
10,094 post(s)
#30-Mar-19 22:07

[Sorry, 9-inch resolution was right. 0.75 US survey feet. How you guys put up with not being metric I have no idea.]

tjhb
10,094 post(s)
#30-Mar-19 23:03

If you followed my instructions to the letter, per image, and your data appeared in the wrong place, then the data itself is wrong. That is quite possible.

But please retry. Follow the sequence exactly, in a new project, doing nothing else.

Import all of the the .jpg files (together is fine). Then for each one, follow the sequence.

If you get any problem at all, you should post some source data.

(If the source data is too big then there is any easy way around that. You can just replace the whole image with zeroes, which will compress to nearly nothing. Let me know if you need the SQL for it.)

dchall8
1,008 post(s)
#01-Apr-19 15:18

After following the sequence for the first image, open it (not in a map), and check the location in the status bar. If that is wrong, stop.

STOP

It's wrong. It is showing 21 degrees west ant 87 degrees north. Arctic circle.

(If the source data is too big then there is any easy way around that. You can just replace the whole image with zeroes, which will compress to nearly nothing. Let me know if you need the SQL for it.)

If it requires SQL, then yes I would need that. I tried changing the Rect size to 0,0,0,0 and still have a 75MB mxb file.

Attachments:
Erroneous Latitude in M9 jpg Import.jpg

tjhb
10,094 post(s)
#01-Apr-19 21:27

I tried changing the Rect size to 0,0,0,0 and still have a 75MB mxb file.

Nice try. This adjusts how much of the data is displayed (in a particular image), but does not affect the data stored in the source table.

Here's how to reset everything to zero in SQL:

Let's assume the table is called [Image 1 Tiles] and has an image called [Image 1].

Open [Image 1 Tiles], and note what is shown in the tile column. For example, it might be <tile, 128x128, uint8x3>. We need to know the tile size (here 128x128) and its type (here uint8x3). If your image has different tile size or type then you will need to adjust the SQL below.

Open a Command window: View > New Command Window > SQL, or just Ctrl-~ (Ctrl-`).

Paste in

UPDATE [Image 1 Tiles]

SET [Tile] = TileMakeNew(128, 128, VectorMakeX3(0, 0, 0))

;

Adjust the table name (NB the table, not the image). If the tile size is not 128x128, adjust the two 128 values to suit.

If the data type does not end in 'x3', change VectorMakeX3(0, 0, 0), e.g. to VectorMakeX4(0, 0, 0, 0) if it ends in 'x4'.

Run the code (F5). That may take a few minutes if the image is very large.

Now open the image, [Image 1]. You should see the warning bang at the right side of the tab. Hit F7 to bring up the Messages window and click Update Data to rebuild intermediate levels. Wait for that to finish (no progress window is normal here).

The image should now be blank.

Close all windows, delete any components you don't need, then export the project to .mxb. It should be tiny.

Can you also include the original metadata for the same image?

dchall8
1,008 post(s)
#01-Apr-19 22:32

Only did one image. Hope that's enough. The three metadata files are attached.

Just to scope this, there are 180 files in the 3-inch resolution, 859 at the 9-inch resolution, and 859 at the 1-meter resolution. I believe the 9-inch covers all the same area as the 1-meter. The 3-inch files were a special order for two areas in the county.

Attachments:
JPG Image Project.mxb
TXBAND019039OrthoSectorTile1.jgw
TXBAND019039OrthoSectorTile1.jpg.aux.xml
TXBAND019039OrthoSectorTile1.pmi

Dimitri


7,413 post(s)
#02-Apr-19 06:33

Only did one image. Hope that's enough.

Please don't make this like pulling teeth.

You have posted the results of an obviously wrong import, but you haven't posted the original .jpg and you don't report every detail of what you did.

Why don't you just post the original .jpg? If you did that with your original post this thread would have been just two posts long, question and answer, and the issue would have been resolved days ago.

Imports are not rocket science. To fix a problem, you go down a very short list:

1. Is the original set file OK or does it contain errors?

- If we don't have the original files, we can't check that.

2. What is the workflow to import it correctly?

- Reporting " I followed those instructions" isn't enough, because people often overlook details without realizing they missed something. That's especially true if more complex workflow is required to get around some problem with the original files.

If you post the original files - all of them - then other people here in the forum can report the workflow they used to import it correctly, to get around any errors in the files and so on.

3. Bugs can happen, but to find them you have first get 1 and 2 above sorted out.

The easy way to fix this is for you to provide all of your starting files and then somebody here who is an expert at such things can say "No problem.. here's how to do it step-by-step," or "You've found a new bug...".

dchall8
1,008 post(s)
#03-Apr-19 18:19

I'm not trying to prolong this. Everyone is busy. I worked on this for days before I frustrated myself enough to ask for help.

1. As I mentioned in the introductory post, the .jpg files are 8.6-18 MB each. I'm under the impression there is a much lower limit to what I can attach here.

2. You are correct that the workflow is key to resolving this. I followed Tim's process but could have been more clear about it. Here's how I did it. File>Import where I selected all the files attached in the introductory post. Then, opened the image from the Project pane, went to Contents>Component and selected the Calipers button in the lower right. Calipers>Repair>Edit. Switched from Custom tab to Standard tab and selected US State Plane NAD 83 Texas South Central. Turns out this is in meters, so it is likely the wrong projection. Adam's idea to use the EPSG tab looks better.

If I post all 900 files you're going to hate me more than you might already for making this a dental visit.

3. My first thought is ALWAYS that Manifold has no bugs. There may be features I'd like to see, but no bugs. My first thought, then, is that the problem is with me not knowing how to use it. At this point I still don't see that there is a bug. Isn this case the problem might be with the import meta files, which is not a Manifold bug. Once I got the projection correct, or at least close, (thank you, Adam) there seems to be a 28.5 ft offset in the first image I compared to Google Earth. It could be that the vendor for the images is off or it could be a datum discrepancy.

2 cont'd. Once I have the workflow to get one file projected correctly, then I will work on how to repeat it 1,000 times.

Dimitri


7,413 post(s)
#04-Apr-19 08:01

Take advantage of the forum. If a quick approach to import a file using what the user manual suggests doesn't work, don't spend days on your own. Post a sample zip file with the image file and all sidecar files and ask for help. You don't have to post all the files. Just post a representative sample of one or two images.

If the zip file is bigger than can be attached directly to a forum post, use a free, third party alternative site: dropbox is popular, as is imgur for images (could post the .jpg to imgur and the sidecar files as an attachment here), as is google drive (drive.google.com) where you get up to 15 GB of free storage and you can put files that are available to the public.

I agree it's not likely about bugs. It's probably about the usual hassles of interoperability using formats that are highly unreliable for GIS use, because they lack reliable standardization in matters like conveying projections. That gets especially messy if multiple different packages have been involved in the creation or processing of such files, so dissonant bits and pieces (in terms of sidecar files) from different packages have accumulated, or where at some point an import or reprojection process was done incorrectly so that whatever was done in subsequent packages perpetuated that error.

I'd still try to do it the GDAL way. If GDAL was the package that wrote these files, and if the images were correctly georegistered when GDAL wrote them, there is a very good reason to believe GDAL accurately will import what it wrote. It could be, of course, that GDAL was not the last package to have touched these, and that something else altered them from how they were created by GDAL (if, indeed, it was GDAL that created them).

dchall8
1,008 post(s)
#04-Apr-19 17:36

Great idea to use file sharing. I tried using Firefox Send but the upload hung up at 29%. While that was trying to upload I signed up to Imgur and uploaded the four images I'm starting with. Here are links to the jpgs.

Tile 1

Tile 2

Tile 3

Tile 4

If these don't work right, I won't be surprised as this is my maiden voyage with Imgur.

The tile reference numbers match the meta files included in the attached zipped file.

If someone can get these to read using GDAL, then I'll spend more time working on that problem.

Here's another approach I just thought of. If all these tiles interlock in their own projection space, perhaps I could bring them all in together, merge the images, adjust the local offset (while shooting myself in the foot) and then export the merged image as an ecw file.

Attachments:
TXBAND19-TILES-3INCH.zip

dchall8
1,008 post(s)
#04-Apr-19 19:10

Here's another approach I just thought of. If all these tiles interlock in their own projection space, perhaps I could bring them all in together, merge the images, adjust the local offset (while shooting myself in the foot) and then export the merged image as an ecw file.

Just to comment on my latest thought: that seems to work. I used concrete curbed plots in the cemetery (Tile 3) to find non moving monuments on the ground . This resulted in a Local Y offset of 29.385 feet and no apparent X offset. Trying to use roof top ridges does not work due to the photo angles from different fly overs. The Tile 1 image aligns in the Y but still shows an unresolved offset in the X direction. Perhaps georegistration would work better with the merged image file using a vector parcel file for reference.

Dimitri


7,413 post(s)
#04-Apr-19 19:33

Thanks for posting the images on imgur. I downloaded them, but the image names don't match the names used in the sidecar files in the .zip.

I'm guessing, for example, that the Tile 1 image should be renamed from T1PoW2c.jpg to TXBAND019039OrthoSectorTile1.jpg - is that correct?

Edit: Tried that and GDAL imports the image, but not with projection info (apparently GDAL doesn't read the .xml either...). I tried various combinations with/without the various sidecar files. So... no easy solution using GDAL.

Out of curiosity, do you know how these images were created? If they came from some public source there may be info on the originating website that could be useful.

dchall8
1,008 post(s)
#04-Apr-19 22:37

These files come from a company we know as Pictometry, but now they call themselves EagleView. They fly custom aerials.

To follow up. Using M8 it is relatively easy to make the merged images fit to the Google Earth images and, hence, my vector parcels. I tried nudging it by altering the local offset and that worked okay, but on the average over a few miles there discrepancies of several feet at the far end. Then I tried georegistration in M8. That may have worked better in that the average of all the errors seem to average out giving passible registration with the vector parcels. Expect errors of less than 2 feet just about everywhere. However, the program that needs to use the ECW file is not working with the file. The ecw comes in without the nudge or geo alterations. That program (PACS Appraisal) relies on ESRI integration. It brings in ECW file with the un-nudged location in both instances. So getting it right in Manifold is only step 1. I thought that was all the steps, so I'm glad I started with a small subset of the images. EagleView provided both jpg and .sid files. I am asking if they can replace the .sid files with .ecw files. The .sid files import into PACS and are properly located, but the images are washed out after the import.

Dimitri


7,413 post(s)
#05-Apr-19 07:27

EagleView provided both jpg and .sid files. I am asking if they can replace the .sid files with .ecw files.

Ask if they can provide in GeoTIFF. GeoTIFF contains the projection info within the TIFF itself. Both .ECW and .SID are wavelet compression that almost inevitably uses lossy compression. It's a shame to degrade such nice imagery by throwing out content through lossy compression.

Thanks for the explanation, as that clears up much of the mystery. Pictometry uses a proprietary format which usually is .jpg with required metadata in their proprietary .pmi file. The solution for easy import of Pictometry images is to read the .pmi file to extract necessary coordinate system info.

It's interesting to note that very few, if any, GIS packages can do that. GDAL, for example, which has very broad coverage of even very niche formats, does not parse .pmi to extract coordinate system info.

I'm mildly surprised that any county (Pictometry's market, for the most part) would in modern times agree to provision of data in such a highly closed format without requiring Pictometry to provide some tool to at least convert data into more open form. That bedeviled MrSID in many government markets where, over time, buyers realized they were violating their own open access laws by utilizing MrSID, so Lizardtech provided utilities to open up the data, including the API now used by Manifold.

It might be worth seeing if your county has some tool provided by Pictometry that could be used to batch convert the .pmi metadata into something more standard, that is accurate.

dchall8
1,008 post(s)
#05-Apr-19 14:59

EagleView provides a hard drive with the images along with licenses and ... software.

Before I elaborate, the main reason to use the Pictometry imagery is for the oblique view of buildings from four sides. Their controlled process allows the user to measure area, length, height, and slope of land and buildings. In the Texas appraisal biz there are a lot of properties where the owner will not let us inside their gate to measure the buildings. When all we can see is a roof, it's a guessing game as to what is under the roof. With the oblique imagery we can see doors, windows, number of stories, etc. and even measure them fairly closely. Their customer base seems to be ESRI users. As a service to the people who pay our salaries, we make the imagery available to our county schools, city and county offices, and emergency services including police, fire, and ambulance. The 3-inch imagery is special and they are very proud of it. We only order 3-inch resolution for a few square miles of the county. The rest of the county is now shot at 9-inch resolution. The 1-meter resolution used to be their standard, but competition is changing that.

The software provided by EagleView seems to be oriented to people who need to manage the imagery. They provide ArcMap and ArcGIS extensions including older versions of Arc and the following software:

  • ALOHA 5.4.3
  • RoboCopyGUI

  • SectorPlanner 2.9.3.32
  • NIW (Network Image Warehouse) 2.2.0
  • EFS (Electronic Field Study 2.7)
  • Configuration Editor 2.01

    The installation help files that come with these do not have details as to the capability or even use of the software. I'll see what I can find out from our rep.

    Attachments:
    Pictometry Height measurement.jpg

  • adamw


    10,447 post(s)
    #02-Apr-19 10:39

    I think I know what's going on and how to fix it.

    Open the MXB. Open the image. Switch to the Component pane (Ctrl-1).

    The coordinate system is displayed as 'Pseudo-Mercator', in red - that's fine, we know it is missing (was not imported from JPG.AUX.XML because we cannot read the format used there at the moment).

    Click the button next to the readout (the second one which is the readout for the image, the first readout is for the map and it should not have a button near it because the map is temporary, we opened the image on its own) and select Assign Initial Coordinate System - Edit Coordinate System.

    Let's make two adjustments:

    1. Switch to the EPSG tab and type 'Texas South Central' into the search filter (that's the wording from the JPG.AUX.XML). Out of multiple systems presented, let's pick the first that seems suitable: NAD83 / Texas South Central (ftUS) (EPSG:2278). Because JPG.AUX.XML mentions both NAD83 and US Survey feet.

    2. Click the XY button at the bottom near the units readout. Select Edit Metrics. Change the XY unit from Meter to US Survey Foot. Because JPG.AUX.XML mentions US Survey feet again. Click OK to close the coordinate system metrics dialog.

    Click OK to save changes to the coordinate system. The status bar should now report that the image is around 99W 29N (well, closer to 30N). That seems to be in Texas (verified by creating a Bing data source, creating a new map, dropping the Bing image into it, then dropping this image into it, then zooming to this image from the layer tab context menu and zooming out a bit).

    As I said, we'll try to read what's in JPG.AUX.XML in this particular case.

    dchall8
    1,008 post(s)
    #02-Apr-19 19:41

    This is 99% perfect. The general differences between each image is the local offset, which is different for each image. I've looked at the metadata for the images and cannot see where the local offset is called out. Fiddling with the local offset, ad hoc, has been deemed 'shooting yourself in the foot' in other topics in this forum. Tim has noted that the problem is more likely to be with the original datum being mistaken. Pointing that out helped me find a datum problem in Manifold 8 two years ago. With that one I switched to Radian (back then) and imported with Radian interpreting the datum correctly the first time. I've looked at these new images and compared datums (data?) and don't see an obvious datum difference like there was in the previous problem. These new images are from a different vendor than those in the previous problem. Here is the visual showing the difference between Google Earth, above, and the jpg overlay.

    By using the roof ridge lines to create a cross type monument near the ground, the offset measures 28.4 feet south and 5.9 feet west from the Google Earth reference. It has also been pointed out that the quality of GE positioning is variable, but that a 30 foot error is more likely to be a datum issue.

    Looking at the mfd_meta table, specifically at the FieldCoordSystem.Tile property, here are the data from image tiles 1-4

    Tile1 EPSG:3674,mfd:{ "LocalOffsetX": 591127.4588400001, "LocalOffsetY": 4210593.9918, "LocalScaleX": 0.0762, "LocalScaleY": 0.0762, "LocalScaleZ": 0.3048, "Unit": "Foot", "UnitScale": 0.3048, "UnitScaleZ": 0.3048, "UnitShort": "ft", "UnitShortZ": "ft", "UnitZ": "Foot" }

    Tile2 EPSG:2278,mfd:{ "LocalOffsetX": 591664.9584, "LocalOffsetY": 4210585.000200001,

    Tile3 EPSG:2278,mfd:{ "LocalOffsetX": 592203.2352, "LocalOffsetY": 4210585.000200001,

    Tile4 EPSG:2278,mfd:{ "LocalOffsetX": 591126.7578, "LocalOffsetY": 4210048.2474,

    I included the entire record for Tile 1 and truncated the records for all the identical text. Tiles 1-3 are side by side with Tile 2 being east of Tile 1 and Tile 3 east of Tile 2. Tile 4 is south of Tile 1. Here they are on top of Google Earth. The 28-foot local adjustment gap between Tile 1 and Tile 4 is visible.

    It would appear that Adam's process could be used once, and then the meta table could be used to copy and paste projection data for the rest of the tiles. The local offset looks like a bugaboo in that you can't copy that from one tile to the next.

    Attachments:
    Jpg image tiles on Google Earth.jpg
    Offset in jpg image.jpg

    adamw


    10,447 post(s)
    #02-Apr-19 10:16

    A side remark regarding filling an image with zeros - instead of using TileMakeNew, you can use TileFill:

    --SQL9

    UPDATE [image] SET [tile] = TileFill([tile], 0);

    This way you don't have to remember or know the dimensions (will be taken from the tile) or even the pixel type (if it is an xN, the fill value will get auto-extended - with zeros!).

    Dimitri


    7,413 post(s)
    #30-Mar-19 05:29

    Where is a sample of the actual .jpg image? Given a sample of the original .jpg such issues are usually easy to solve in terms of providing a step-by-step procedure. Without the original there is guesswork involved.

    But... since we are guessing I'd guess from the presence of an .xml file that declares itself to be a <PAMDataset> that somebody used GDAL to create these images. OK.

    My rule of thumb is that if somebody uses GDAL there's no need to figure out what GDAL has done... just use GDAL and let GDAL itself figure out what it did.

    Manifold has a GDAL dataport which makes that easy. Import the original .jpg using GDAL as shown in the GDAL / ORG topic. Should do the trick. GDAL normally does a great job of correctly reading GDAL-created sidecar files.

    If you have any influence on whoever created these images, try to talk them into using GeoTIFF, which has some reasonable standardization around it. That's better than using JPG plus a hodgepodge of pseudo-standards. You can see that in action in this case where the <PAMDataset> declaration seems to conflict with what is written in the .jpw "world" file, as a quick look seems to indicate.

    Another consideration is that when people write to TIFF they often use packages which do that using lossless compression, so the images are not degraded. The problem with JPG is that most packages which write JPG do so with some lossiness to the compression, so the image is automatically degraded. It's a shame to lose any content with such beautiful data.

    adamw


    10,447 post(s)
    #01-Apr-19 11:00

    A common typo: OGR, not ORG. :-)

    gdal.org is a home of GDAL/OGR. Easy.

    dchall8
    1,008 post(s)
    #01-Apr-19 15:53

    I think I did the GDAL correctly all the way through to adding the path for the .dlls. When I hit the import button, nothing changes. Nothing imports. Is there something else I need for this?

    I have been withholding information. I also have Mr SID files for the same areas. They take forever to import to Manifold, but I'll try that approach again and then export as ecw.

    Attachments:
    GDAL Import Setup.jpg

    dchall8
    1,008 post(s)
    #01-Apr-19 16:41

    I quickly linked to the sid file and it shows up in the arctic, too.

    Dimitri


    7,413 post(s)
    #01-Apr-19 17:15

    To import through GDAL, you just iclick the .jpg, not all the sidecar files.

    dchall8
    1,008 post(s)
    #01-Apr-19 19:50

    I still get nothing.

    Dimitri


    7,413 post(s)
    #01-Apr-19 19:55

    Would love to see the original file... :-)

    adamw


    10,447 post(s)
    #01-Apr-19 10:56

    From the files that you posted, the coordinate system is specified partly in JGW (world file, always contains local offsets / scales) and partly in JPG.AUX.XML (PAM data, the particular file that you posted contains the definition of the coordinate system, we are lucky). We read the former, but we do not read the latter. We will see if we can extend our code to read the latter as well. If we do that, you won't have to change the projection at all, we'll just read it automatically.

    One thing to note here is that PAM data can contain coordinate system info decoded in many different ways, including vendor-specific ones, so supporting it all is a bit like supporting "all" of GML - not really possible with the decidedly open-ended approach as to how the data can be organized. But, as with GML, we can absolutely support the most common cases - eg, we can support the case of PAM data containing an embedded PRJ, like in your case.

    Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.