Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.166.6
adamw


8,217 post(s)
#04-May-18 19:00

9.0.166.6

manifold-9.0.166.6-x64.zip

SHA256: 0d1c455a289c701cbde2d3d26a6a7e67e0afd5d63ac8ffcb095b545743967fc9

manifold-viewer-9.0.166.6-x64.zip

SHA256: 5b30fd1d61c67f0c634ac188aa612e2a4b68705507f3a4b96a5b83383f99f564

adamw


8,217 post(s)
#04-May-18 19:02

Changes

(Fix) Fixed btree indexes in MAP files to no longer sometimes (very, very rarely) desync after deleting data.

(The above failure could only happen with variable-length field values of certain shape. After the failure, the index was refusing to operate and was doing so loudly, throwing error messages. If you never saw any error messages regarding records not being found, you very likely never had the error happen. The failure was not propagating. The failed index could have been safely deleted and recreated. The fix does not automatically heal failed indexes, but prevents future failures in both existing and new indexes.)

There are three new commands to merge data in existing components of different types. All commands work on layers in an existing map. All commands support merging layers from different data sources.

The Edit - Merge - Merge Drawings command allows merging one or more drawings into a single component. The new drawing and the accompanying table are created in the data source of the map. The coordinate system of the new drawing is initially set to the coordinate system of the first layer and can be edited. Right-clicking a layer and selecting 'Use Coordinate System' sets the coordinate system of the new drawing to that of the layer.

If the data source for the new drawing is a MAP file, the drawing table includes the system MFD_ID field and MFD_ID_X index. The values of the MFD_ID field in the source drawings are ignored.

The 'Copy all fields' option controls whether to copy fields other than geometry (on for drawings by default). Fields of the same name and type from different layers are merged together. Fields of the same name but different type are still merged together as long as field types are compatible, with the type of the resulting field being made big enough to hold all values (eg, if field X in drawing A is INT16 and field X in drawing B is INT32, the system will create a single field X in the merged drawing and set its type to INT32).

The 'Save source component for each record' option allows saving the name of source layer into a separate field (on for drawings by default).

The 'Skip records with null geometry' option allows skipping records with null geometry values (off for drawings by default). Null geometry values might occur in the original data or they might be produced when projecting original data to the coordinate system of the new drawing.

The Edit - Merge - Merge Labels command allows merging one or more labels into a single component. The options are the same as for drawings, but 'Copy all fields' is off by default and 'Skip records with null geometry' is replaced with 'Skip records with null geometry or text' and is on by default.

The Edit - Merge - Merge Images command allows merging one or more images into a single component. The new image uses the exact pixel scale specified in the coordinate system (shown in coordinate system metrics readout in the coordinate system dialog). If source images use different pixel types, the new image uses the type big enough to hold all values. In particular, if source images use different number of channels, the new image uses the maximum number of channels. (We are going to better indicate such cases in the dialog, because they are frequently an error.)

Merging data from multiple images follows Z order of layers specified in the dialog, with pixels from upper layers overwriting pixels from lower layers.

Merging data from images includes special provisions for cases where the coordinate system of the original image coincides with that of the new image up to pixel scale and only differs in offsets. In such cases, pixels in the original image are not reprojected and keep their exact values.

Merging data from images uses multiple threads if this is beneficial. (Currently, we are only using a single thread only if all source images are copying their pixels without reprojection. Whenever there is scaling or any other operations, we are using multiple threads.)

Merging images into a MAP file generates intermediate levels for the new image at the end of the merge.

Merging components of all types reports progress and allows canceling.

Geometry values with mixed 2d and 3d coordinates in GML, GeoJSON, TopoJSON are automatically converted to 3d with 2d coordinates padded with zeros. (Previously, such geometry values were rejected.)

End of list.

Dimitri

5,085 post(s)
#05-May-18 15:15

A quick tip for anybody playing around with merging images, such as terrain elevation data like SRTM....

Here is a step by step procedure:

1. Create a map, and drag and drop a Bing streets layer into it for background context.

2. Import your SRTM files. You might have a few dozen you've downloaded from USGS, all of which are in Lat/Lon.

3. Drag and drop those SRTM images into the map. To do this quickly, I set the Project pane filter to showing only Images. I can then highlight the long list of SRTM images quickly without having to click around their tables. Once the long list of SRTM images is highlighted, I can drag and drop them all in one move into the map.

4. You now have a map with a Bing layer at the bottom and a few dozen SRTM layers, the layers coming together to cover your area of interest.

5. Launch Edit - Merge - Images. The dialog shows a list of all the layers in the map, and it shows the Pseudo-Mercator coordinate system of the map (same as Bing).

6. Scroll down to the bottom and un-check the Bing layer, since you don't want that merged into the combined image.

7. Right click on any of the SRTM layers that's in the list in the merge dialog and choose Use Coordinate System. That changes the coordinate system of the merged image to Lat/Lon [and, more importantly, using the unit scales for pixels that the SRTM layers use].

8. Press the Merge Components button. On my machine, an older Core i7, it takes about five seconds to merge a couple of dozen SRTM3 layers.

I recommend the above because it is easy to not set sensible unit scale values. The easiest way to make sure the unit scale values make sense for the merged image is to use whatever the unit scale is for one of the SRTM's being merged, and the easiest way to do that is to right click on one of them and choose Use Coordinate System.

The Merge dialog is very compact but between it and how it interacts with the other dialogs that can be called from it, like from the coordinate picker button, it gives you very many capabilities. That's going to require a fair amount of discussion, examples and maybe a video or two to explain.

hugh
159 post(s)
#05-May-18 18:24

merge drawings great also. Separate cadastral boundary dawings for 17 counties (4.3 GB):

Merge: [TaxRollGISboundariesCombined Drawing] (40.528 sec)

my older Core i7. Boundaries and ID fields only so no data fields to merge figuring types values etc. Checked both save source component [name as field] for each record, skip records with null geometry.text

artlembo

3,096 post(s)
#06-May-18 01:23

Hugh,

Have you given this a try with GDAL or ArcGIS? I would love to see how it compares. I have some Maryland data I may try it on Monday.

Are you saying that you merged 4.3GB of vector data in 40s? That is approaching a Windows copy command.

Dimitri

5,085 post(s)
#06-May-18 07:40

Are you saying that you merged 4.3GB of vector data in 40s?

Not bad, but it is much faster with raster data. Try putting about 30 SRTM layers into a map and merging them. It seems to happen instantly.

If you check the log it's really about four or five seconds but still... quick enough you don't even think about it.

I have to say, this is something you get used to instantly. It always really bugged me that USGS provides elevation data in tiles that seem designed to cut right down the middle of your area of interest. It also bugged me that to get good elevation coverage of an area of interest I'd have to download about 30 files to get the NED, SRTM, ASTER DEM or whatever data I needed to cover it, and then I'd have to copy/paste the Style to get them all looking the same.

No more problems like that. I just download a few dozen files, load them all up, merge them with one click, and now I have one big elevation data set I can style however I like. Just a day of that and you forget what it was like before.

hugh
159 post(s)
#06-May-18 13:30

What is most impressive for me is not the 40 sec to do the merge but the ~15 min it took me to figure out what to do, import the shp files and put in a map, have some confidence IDs and indexing capability are there to bring in more data fields later, and press the button to do the merge. Even if GDAL or ArcGIS merge as fast it would take me forever to figure out how to do it with them.

artlembo

3,096 post(s)
#06-May-18 14:36

The ArcGIS merge tool is super easy to use. I would be interested to see the speed differences. I’ll try some data on Monday and let you know.

artlembo

3,096 post(s)
#06-May-18 17:08

Hugh,

Thanks for the email. I think the people here would be very interested in hearing the results you had if you don’t mind sharing.

hugh
159 post(s)
#06-May-18 18:18

tried merge with ArcGIS Pro. Failed after 10 min probably due to lack of temp space. Screen shot attached.

Attachments:
2018-05-06_10-07-35.jpg
2018-05-06_10-07-51.jpg

artlembo

3,096 post(s)
#07-May-18 14:37

I just gave this a shot with soil data: NY, PA, and MD SSURGO data. 3 million complex polygons. 14GB of data and tables.

I used an ESRI geodatabase as a data source, linked from my external USB drive:

Merge: 123s

I then decided to move the data onto my SSD:

Merge: 120s

So, the SSD didn't seem to make a difference.

Finally, I used ArcGIS to merge the data from my SSD:

Merge: 517s

So, all-in-all, not too bad. It is great to see 9 run 4x faster, but even waiting 8min for ArcGIS isn't too terrible.

maybe once the semester ends, I'll give this a go with GDAL (ogr2ogr), but for now, I can't do it.

Dimitri

5,085 post(s)
#07-May-18 15:24

When you leave the data in the ESRI geodatabase you're not really giving 9 a chance to run. I understand that is very convenient but GDB isn't the world's fastest data store. PostgreSQL would probably be faster.

It's great it is 4x faster even using ESRI's own native storage, but still... I think it would be significantly faster if you had the data in a native Release 9 .map.

... I'd love to see how fast the Merge would be with everything (TEMP, .map, etc.) stored on the SSD. :-)

artlembo

3,096 post(s)
#07-May-18 17:16

I can check later. But remember, you're going to have to also include the time to import the data into 9 from the geodatabase. That will take many minutes.

artlembo

3,096 post(s)
#07-May-18 17:22

Decided to check before my next appointment:

I imported the soils into 9. It took about 3 minutes to bring in NY, PA, and MD. Not horrible, I suppose.

As you can imagine, the 3 states draw in under 2 seconds.

Then I did the merge: that completed in 99s. So, about 30% faster.

Dimitri

5,085 post(s)
#07-May-18 18:35

Thanks Art! That's very interesting data. Were all three drawings in the same coordinate system or in different systems?

artlembo

3,096 post(s)
#07-May-18 19:32

all the same. I am so tempted to create a soil map for the entire US. The semester is almost over, so that might be a fun project to try out.

dchall8
512 post(s)
#07-May-18 23:26

My patience has been rewarded. This is amazing.

My DEM files are 17MB each having resulted from a 2015 LiDAR fly over. Resolution is 1 meter. I just merged 256 of them in about 30 seconds. However, Manifold took longer than expected setting them up to style. I did not check time on that if it is even logged. Then we come to saving the file. Saving took 34 minutes. The resulting file is 27.5GB. While it seems the original Z values are missing from the table, the Style function shows the Z values. Subsequent saves are faster.

The results, especially with hill shading, are spectacular. Otherwise flat looking Google Earth imagery comes alive. See attached pics. I am seeing features I never noticed because now they are geographically set off from the flatness. I can use the telephone poles in Google Earth images like a sundial to tune up the azimuth and altitude of the shading to match Google's shadows. This is really cool.

Attachments:
Pipe Creek Google Earth.jpg
Pipe Creek Hill Shaded.jpg

tjhb

8,348 post(s)
#08-May-18 00:02

You’re right, that is amazing. A great idea to post both images—the shaded relief makes all the difference in the world. Very cool!

Dimitri

5,085 post(s)
#08-May-18 03:38

Agree with Tim... amazing, and thanks for the images! It's a great idea to combine the shaded relief with the images.

Some commentary to help guide others...

DEMs these days are often provided in compressed formats like GeoTIFF. If so, a 17MB file will expand to a larger size in pixels. 256 * 17 is about 4.3 GB, so with some expansion it could end up being way more pixels. Is the 27.5 GB with the original 256 images still in the project or is that just the combined image after merging the 256 original images and then deleting them as no longer needed?

The setup time to style is the first time through when it computes statistics. It's getting statistics on all the pixels to allow precise computation of intervals using various methods and breaks. Once done it should not have to be repeated for further Style changes. Also, the saving takes time only the first time. After that you can make changes and saves generally are instant.

Z values are each pixel's elevation value. Since you're working with images and not points, the table shows tiles as records and not individual pixels. You can get the value for any specific pixel, but to do that outside of pre-built methods like tracing areas or creating contours is intricate.

Hope that helps!

...great images! :-)

dchall8
512 post(s)
#08-May-18 21:24

No, I did not think to delete the files before saving. Since that was more less a large scale proof of concept, I deleted the project to do one that I will keep. Today I started over with a slightly larger scale job. Yesterday I calculated the number of DEM tiles based on four 8x8 tiles. I thought they were all 8x8 clusters of DEMs, but they are not. Today I counted tiles and came up with 241 comprising the entire east part of the county. Thus yesterday's test had to be fewer than 256 DEMs. I merged the 241 tiles (in 502 seconds (8 minutes 22 seconds)) and immediately deleted the folder containing the smaller DEMs. Deleting the folder took 3 minutes. Then I selected the merged DEM file, went to Style pane, and selected Channel 0. Running the Style statistics took 9 minutes. Rendering the Styling took a fraction of a second. Then I brought in Google Satellite to make a new Map. I checked the general nature of the DEM and then saved. Saving took 10 minutes. The process absolutely slammed my computer's memory. I had three Firefox profiles open, and they simply stopped. When the streaming music stopped I knew Manifold was challenging the computer. I checked the resource usage and noted that the RAM was maxed out but the cores were practically idling. Closing the browsers during the Save was even difficult. Anyway the file size for this merged file only is 6.8GB. If I was in the DEM conversion business I would get a lot more than 16GB of RAM. I might need to upgrade the computer once I convert over to M9 (hate to get into Win 10, though).

This capability to quickly merge DEMs is amazing (did I mention that already?). I will willingly sacrifice a few minutes of streaming oldies to do this.

Having 1 meter resolution on the DEM allows you to see very subtle changes in elevation with hill shading turned on. 2 more pix attached.

Attachments:
11th Hole 12th Tee hillshaded .jpg
11th Hole 12th Tee.jpg

KlausDE

6,233 post(s)
#05-May-18 22:38

German UI file for 9.0.166.6

Attachments:
ui.de.txt

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.