Hi,

As forecast in my previous thread, Aerial Raster for urban vegetation cover analysis, I have received some files to help me (hopefully) carry out a canopy cover analysis for a given area near Melbourne, Australia.

I would like to differentiate between vegetation layer height classes of my project area between 2 different years and determine percentage area changes.

What I want to achieve is a simple version of something like this report.

At the time I posted the previous thread I was not sure what form the data would arrive in, or if it would arrive at all. But, it did and it is ER Mapper Grid format - massive images with header files with the extension ers. I'm hoping it has the data I need for what I want.

To recap where the previous thread was going I've pasted Dimitri's help from the previous thread here where he is responding to my original post.

I'm hoping the data I get is already classified and has infrared data incorporated otherwise I see there are ways to do it, at least partly, in software such as Global Mapper and perhaps QGIS.

As I understand it, your task is simple, almost trivial, in Manifold: take a raster in ERS format that your client has provided and to color it by heights:

The analysis should ideally differentiate between layer height classes: grass (0-0.5m); shrub (0.5 - 3m); small tree (3 - 10m); medium tree (10 - 15m); and large tree (15m+) as can be seen here.

OK. Easy enough. Manifold reads ERS format, so once you have your surface, it sounds like the contour transform, set to areas, would do that in one step. Or, you could just style it, as in the Style: Contouring using Colors topic.

If there's something else, it's not clear what the task is. The "as can be seen here" link in the quote above jumps to an "Urban Monitor™" web page that has plenty of generalities but no specifics, so it's hard to tell exactly what they're selling. [Usually a sign they're trying to sell something simple for a too-high price... :-) ]

The only specific item is the illustration on that page showing a hill shaded surface that in addition to hill shading is colored by height with one palette in regions where land use indicates parks and vegetation and colored with a gray palette in regions where land use is structures and urban land cover.

If that's the task, that's simple to do in Manifold.

For example, it's easy to trace out polygonal areas from clumps of different raster pixels, classified as to land use (there's an example of that in the user manual). You can then use those polygonal areas as you like to manipulate the LiDAR points that fall or don't fall within them. Or, if you're working exclusively with raster layers you can do arbitrarily complex math between different raster layers if you like.

Once you know what data you're working with, if you run into any problems, ask here in the forum. State clearly the data you're working with and what you want to do, possibly posting links to examples to the data, and you'll get plenty of tips.

This will take a couple of posts to unpack so I'll start with issues getting the data into M9 in the correct location.

The files I received are around 4.4GB (a couple being 17.4GB) with no explanation or metadata beyond what is contained within them (and I'm unlikely to get a response from the source provider - I have put the question out there to them anyway - we live in hope).

The file names all contain various references to their projection and dates with one possible clue as to their contents being grs_raw, nsm_raw, tre_raw, vht_raw and msk_nod (the last one just appears to be rectangles off the coast of Melbourne). When I open them it can be seen that they are isolated height layers such as tree canopy, grass, etc. Some layers seem to have a mixture.

The files are expected to be in Australia GDA94 MGA55 as per the info in the ers header files with each reporting the dataum as "GDA94" and the projection as "MGA55".

This has been my work flow for a sample file so far:

After I link (File > Link) the ers file it imports the image with the default projection shown as Transverse Mercator. When opened it is about 90k to the east of where it should be.

If I try to reproject the image (Reproject Component) the dialog won't allow me to choose a Conversion Type to reproject it with to GDA94 55 so I allow the default.

Manifold chews on it for a while an then spits out an identically (as far as I can see) located copy.

If I link to the same file through QGIS it doesn't complain but does assign it a Papa New Guinea projection (EPSG:5551).

I can then use QGIS's 'Set CRS' to set the layer to GDA94 55 okay. It comes out in the correct position.

As an experiment in M9 I tried 'Repair Initial Coordinate System' to EPSG:5551 and then reproject it to GDA94 55. It moves but still nowhere near where it should be.

I exported a smaller clip of the file from QGIS in GDA94 55 and imported them into M9 but it still came out about 1800m north of where it should be. Interestingly the same file imported into M8 with the correct projection and in the correct place.

I experimented with importing the file into M9 rather then linking it and nothing changed as far as the above projection issue.

Global Mapper v.22 is available as a demo version so I tried that. I'm guessing that ers grid files are part of its staple diet because it imported whole files and clips, correctly projected them and applied colour to elevations. Looks pretty cool. I'm happy to buy Global Mapper if this is the best way to do this.

I'm sure M9 can do this too though so that's what I want to achieve as soon as I can get the projection issue sorted.

BTW, the tables only have x, y and geom data - no other fields.

Dale, if you read this, I did try several times to e-mail you but had no response. I'm sure you are a busy too!

Interesting.

Could you confirm the clipped file exported from QGIS displays correctly in MF8, but not 9?

Checked email spam folder, found your emails! Will call.

Lots to unpack. :-)

It's not clear from your post what three letter extension the file names you have are using, given the citation of names such as "grs_raw". If you haven't prevented Windows from hiding three letter extensions, as advised in the Read Me First topic, do that now so you don't have to guess.

That's important, because ERMapper ERS grid files usually end in an .ers extension, not a .raw extension. If what you have ends in a .raw extension, you probably have a "raw" file, not an .ers file. To consume .raw files, see the RAW, RWB, RWT, BIN topic and the links it contains.

By the way, QGIS itself does not have built-in code for imports/links. Instead, it calls GDAL for imports/links. Manifold can use GDAL as well if you prefer GDAL to native Manifold dataports. See the GDAL / OGR topic and the links it contains to learn how to import into Manifold using GDAL.

The following by the way, sounds wrong for two reasons:

After I link (File > Link) the ers file it imports the image with the default projection shown as Transverse Mercator. When opened it is about 90k to the east of where it should be.

If I try to reproject the image (Reproject Component)

1) If the projection as originally assigned is wrong, you don't reproject the image, you must repair the projection. See the Repair Initial Coordinate System topic.

For example, if it is supposed to be in GDA 94 55 , then use repair to specify that. If it's in the wrong place, there's something wrong with the projection. There are useful topics to provide tips for troubleshooting. For example, it could be you have various sidecar files that disagree with what the projection should be, and in the priority tree for picking which last one wins, a sidecar file that is specifying the wrong projection has won (see the "Projections from Accessory Files" section of the Projectionstopic). Can't say exactly without all the details on the files you have.

2) Why link? Use import. You then can do whatever you want without being limited to what the format allows.

Thanks for the quick response Dimitri.

Sorry, I should have been explicit about the extension which is ".ers".

I can't imagine why anyone would have extensions turned off in Windows!

The ers extension is how I know they are ER Mapper ers files: 1 large file (a BIL format file I expect) with no extension and a header file with the .ers extension - both with the same name otherwise. The header file is an ascii file with the coordinate system names and cell info in it amongst other things. It does indicate it is "GDA94" datum and "MGA55" projection. It also has a line "RegistrationCoord Begin" with Eastings and Northings lines with coordinates which I think are supposed to be the top left cell coordinates.

mdsummer indicates a possible issue in an old post to do with coordinates in ers files:

... Manifold works with georegistration from the bottom left corner, which is different from many other contexts in which interpretation is done from the top left

The "grs_raw" is just a part of the file name. The actual names are 74 characters long with numbers, dates, coordinate datum, projection and other references in them. Since I can decipher all but the part that goes like "grs_raw", etc. I thought that could be a clue to the actual layer contents - say "grs" could be grass? (The file does seem to have only very low elevation objects in it.) Perhaps "tre_raw" could be trees? Is there a standard naming convention like CAD people seem to have?

Dimitri said:

1) If the projection as originally assigned is wrong, you don't reproject the image,

Sorry, my bad. I did actually "Repair Initial Coordinate System" first time (and a few times after) before I experimented with leaving it at Transverse Mercator and Reprojecting. Using Repair takes the image from ~90k east to 14k north of the project area in M9.

... it could be you have various sidecar files that disagree with what the projection should be, and in the priority tree for picking which last one wins

In my post I said:

The files are expected to be in Australia GDA94 MGA55 as per the info in the ers header files with each reporting the dataum as "GDA94" and the projection as "MGA55".

I expect the header file does the job of a sidecar file here.

Dimitri said:

2) Why link? Use import. You then can do whatever you want without being limited to what the format allows.

In my post I said:

I experimented with importing the file into M9 rather then linking it and nothing changed as far as the above projection issue.

I do see the sense in importing the files and I will take your advice once I get the projection issue sorted - thanks!

Dale said:

Could you confirm the clipped file exported from QGIS displays correctly in MF8, but not 9?

I tried this again this morning just to be sure with M8 and yes, the clips display in the correct position in M8.

The original files also display correctly when imported into M8 (no reprojection or projection repair required). I have found before that M8 and M9 don't always read or treat sidecar files the same so that could be part of it.

And, erratum: I said in previous post "BTW, the tables only have x, y and geom data - no other fields."

I actually meant X, Y and Tile fields (not geom).

I expect the header file does the job of a sidecar file here

No, that's incorrect, as sidecar files are often used to correct inaccurate information within whatever is the main file. Please read the discussion I recommended:

(see the "Projections from Accessory Files" section of the Projectionstopic)

I do see the sense in importing the files and I will take your advice once I get the projection issue sorted -

Forgot to mention... One reason I advised importing and not linking is to help sort out the projection issue.

When linking, the data remains in the original file. If that file is read-only, that affects what you can do with the data. Manifold tries to get around limitations, what with using cache and so on, but all the same leaving stuff in the original file introduces unknowns. When you import data, there are no dangling unknowns like whether the source file is read-only or not (I'm just using that as an example of a possible unknown).

In terms of eliminating unknowns, it would be a real help if you could post a link to an example .zip file that contains one of the data sets. People could then download it and, if they can import it correctly, report the workflow they used.

Thanks again Dimitri.

I read the projections topic and can see that sidecar files can correct (or confound) info within the image file. I'm sure the ers ascii header file could also be incorrectly attributed.

I found this regarding ers files in the Manifold 9 Help. I then added 15000m to the False Northing in the Coordinate System dialog for the full image and 1400m for the clipped image and it seems to have fixed the issue (moved the images to the correct locations). It appears you need to add the height of the image to correct it so perhaps in this particular case mdsummer's post was correct about the top left corner vs bottom left corner reference.

I'll now try out your suggestion of contour transform using area.

Any further help or suggestions from anyone who has done this sort of thing before are welcomed!

It appears you need to add the height of the image to correct it so perhaps in this particular case mdsummer's post was correct about the top left corner vs bottom left corner reference.

That's still a projection issue, as dataports automatically handle different top-left/bottom-left referencing by different formats. It's also a classic example where sidecar files can confound what otherwise would be a correct import.

If I have problems importing a file and getting incorrect georegistration, the first thing I do is move into some other folder all files except the bare minimum file required, and then I try importing that. That often works, because one of the sidecar files was commanding some incorrect adjustment.

Thanks Dimitri.

I always keep client's data separate according to its type and, in this case, there are only 10 files - 5 unnamed (BIL) files and five .ers header files. I have double checked them and they are all in Australia GDA94 Zone 55.

I know from past discussions on this forum that there are issues with data format standards, lack of standards and vague standards (e.g. world files) but it is interesting that Manifold 8 (and Global Mapper) import the files into the correct locations without problem. Perhaps they would both import other files incorrectly while M9 would import them correctly (?).

Anyone:

I would love some hints at a workflow to isolate objects (elevations) within my test file at given elevations. The files appear to have pixel elevations in millimetres so perhaps as an example, how would I isolate trees between 3000mm and 10,000m height. When I run the contour transformation using areas (as Dimitri suggested) I get contour areas around lower elevations too. Is this because there are a few pixels of higher elevation within these lower elevation areas too? Is it possible to generate areas around just the 3000mm to 10,000mm (average?) range elevations leaving out the lower elevations?

I would like to isolate areas for each category so I can then calculate the percentage area cover.

Any help at all greatly appreciated!

Ok, I think I'm working this out.

I just deleted all records with a ValueMin of 0. That seems to have done what I want.

I've added a calculated field for metres area for each area.

Can anyone help with a query that will calculate the 'Area' field for all selected areas?

Perhaps they would both import other files incorrectly while M9 would import them correctly (?).

Hard to say without looking at the files. In 9, by the way, you can also import those files using GDAL if you want, for yet another option. :-)

As you pointed out, there are often gray areas due to vague standards where whichever way you guess it should be, you'll get some of the files as intended and others not. But it still helps to get as many examples as possible for the zoo of "gray area" files, because you might be able to improve guessing, add a check box to handle common differences of opinion and so on. I know that in this case you have to keep those confidential, but in the future if possible any examples of strange imports are great to have.

Sorry Dimitri. I didn't reply to the request for an example zip file.

I've had to sign an agreement regarding how I use the data so I can't provide an example.

I did think of taking a snip but that would sort of defeat the purpose because it wouldn't be the same file anymore.