Subscribe to this thread
Home - General / All posts - New video - Creating Terrain Elevation from a NASA PDS Table
Dimitri


5,994 post(s)
#06-Nov-19 11:12

There's a new video on the video gallery page. This one shows how in six minutes you can do very many steps in Manifold, including a Kriging operation to create a raster terrain elevation surface from a table/drawing. It shows a variety of moves that an experienced Manifold user can do to make workflow very fast.

rk
374 post(s)
#06-Nov-19 13:50

Good video. I noticed that screen capture did not include mouse cursor.

Dimitri


5,994 post(s)
#06-Nov-19 14:28

Yes... don't know why that happened, so the video's been redone. The new version is now up.

dchall8
710 post(s)
#11-Nov-19 22:01

I find myself no longer employed by the central appraisal district, so I'm going to ask a question that would be easy to answer if I still had M9 and the LiDAR data on my computer. Can the Kriging image be exported as a .dem file? ...or is that a feature we can look for in the future?

When the CAD received their LiDAR data, it came with .dem files for each geographical tile. The .dems were only for the ground surface. Inside the LiDAR cloud the points were categorized as soil, low shrubs, medium trees, tall trees, buildings, culverts, and miscellaneous. The CAD's interest was to find the buildings hiding below the trees. It would have been nice to run Kriging on the buildings data with hill shading to see the angles of roof tops. I believe M9 had not yet included Kriging back when I was working with the LiDAR.

Dimitri


5,994 post(s)
#12-Nov-19 08:10

Can the Kriging image be exported as a .dem file?

Not as DEM file, but in 24 other formats. See the list for images in the Exporting topic. I recommend using TIF, which Manifold will export automatically as a GeoTIFF, or ESRI GRD. GeoTIFF in particular is a much more popular interchange format these days than DEM.

StanNWT
173 post(s)
#12-Nov-19 21:08

When I saw this video I decided to experiment with some LiDAR (xyz) files I had. I was able to import them and merge them very quickly. I'm using 9.0.169.9. Importing took some time but the merge of 3.5 GB of Lidar data took 1.76 seconds. I'm going to bring in the rest of a large area of LiDAR data and see how fast it imported. I can post a full log file sans directory paths, if you're interested. I did notice that the Google Earth Hybrid v2 imager server data wouldn't load in 9.0.169.9, but I used Bing maps instead. However, in my neck of the woods Google satellite view is far better for up to date and high resolution coverage.

dchall8
710 post(s)
#12-Nov-19 22:06

Wait, wait, WAIT! Before you try importing a number of LiDAR tiles, try "linking" to one, instead of importing, and see if it loads practically instantly. After that if you try to do something with the LiDAR, like merge it, you may have the same problem with the import time.

I had the same issue today with Google Earth Satellite v2. Try the other one right underneath Bing Maps Street Map.

StanNWT
173 post(s)
#13-Nov-19 23:44

Yes I did use linking, it was significantly faster. The only semi-slow part was going into the schema for each (xyz) file and changing the projection. Yes I'm sure there's a programatical way to do it in one go. But the merge took 1.876 seconds on linked data. It saved including the cache for each linked (xyz) LiDAR image in 20 seconds. The resulting (map) file is 1.21 GB

The Google maps hybrid v2, streets v2, terrain, all don't render in the map at all. Bing maps hybrid does render.

I'm now using 9.0.169.10.

I was previously using 9.0.169.9

I did install the new vc++ x86 and x64 runtime installers as suggested for the new build.

Dimitri


5,994 post(s)
#14-Nov-19 06:04

The Google maps hybrid v2, streets v2, terrain, all don't render in the map at all.

If instead of using the Favorites choices, you use File - Create - New Data Source and then in the image servers list choose the original Google versions (not v2), those still work.

tjhb

9,223 post(s)
#14-Nov-19 06:29

Nice deduction!

dchall8
710 post(s)
#14-Nov-19 18:11

When I was working with LiDAR, I linked to 20 or so LiDAR tiles, dragged the contents of each into a new drawing, and then made adjustments to the new drawing. I don't know if there is a time savings with that or if it just would take 1.876 seconds x 20 tiles.

StanNWT
173 post(s)
#14-Nov-19 21:10

Well the total time to save the 1.2GB file, with the merged grid and 20 linked LiDAR tiles was 20 seconds. I don't think that's too bad considering my storage is a Drobo 5D with a 250 MB/s read/write speed. Yes I'm sure it would likely be much faster on an M.2 SSD, but I'm trying to keep all my files in the place where I have the room, which is the RAID. I do have to spend the time editing the schema to change the projection of each linked data set, from the default to what it's supposed to be, but once done the linked or even imported tiles work great.

I'm going to be demonstrating the speed of the LiDAR merging later this afternoon to a colleague. I will also show him the ability to play with Bing Maps or Google Maps web services when you copy the image layer into Manifold, then you can change the styling of the imagery to suit your needs/wants. There's nothing else that I've used in remote sensing software that can do that on Bing Maps or Google Maps streaming imagery. So looking at the LiDAR with hill shading after processing it with the free viewer or just Manifold 9, and playing around with the streaming imagery will be enlightening. They use PCI Geomatica or ENVI to do their remote sensing work, they also will use remote connections into some NASA servers to process imagery or even a server within the organization but not locally in town, so it's been problematic and they can't get the performance they'd like.

tjhb

9,223 post(s)
#14-Nov-19 21:24

I do have to spend the time editing the schema to change the projection of each linked data set, from the default to what it's supposed to be

As you suspect, you can pretty easily do that with a simple query--in theory one click once it is written.

In the simplest case, just UPDATE mfd_meta using a WHERE filter on some combination of Name, Property and Value.

For example if the default projection assigned is always wrong (i.e. if there are no other components sharing that projection, for which it is correct) then you can just filter on Property and Value, and replace Value for all matched cases.

(Assuming there is not a fixable reason why the wrong projection is assigned by default in the first place.)

StanNWT
173 post(s)
#15-Nov-19 17:22

Hi Tim,

I think the problem is that the default coordinate system is EPSG: 3857. I needed all the linked data to be in EPSG:6337. So I would liked if there was an easy way to set the default for that session to be EPSG: 6337, so I wouldn't have to reset the schema for each tile individually. Hopefully if the default was altered for that session it would make the default for each tile EPSG: 6337. Of course once each tile in a directory on disk has had it's coordinate system set to EPSG: 6337, if you've chosen to save the cache option, you won't need to set it again, if you use the tiles again for another project and redo the linking or importing and finally merging, or whatever else you want to do. Some might not want to save the cache to save disk space though.

StanNWT
173 post(s)
#15-Nov-19 17:17

OK an update to the LiDAR data processing.

I decided to grab a larger data set.

163 tiles, 15.42 GB in size. It took 14.921 seconds to process. That's roughly 1GB/s

I in order to make having all those linked tiles more easy to organize I decided to make folders for each group and place the tiles for each folder using the same folder name in Manifold as the folder on disk so I can keep track of things. I moved the linked tiles into those folders, I think based on the final (map) file size, this copied the tiles. Yes this map file is now 11.1 GB, that includes 1 m contour lines that I generated. It took 46 seconds to generate 1 m contour lines on that merged LiDAR data set. It took 74 seconds to save the file. Each linked directory of tiles I chose to save the cache for.

One interesting thing I noticed is that when you have a list of tiles, that you're copying into a folder, is that you get a list of the images in the folder below the folders of each tile, for each tile except the last folder in the list (see attachment). You can see that be_117d01j01 doesn't show up in the list at the bottom. This is reproduced for every directory that I have for all the other groups of LiDAR tiles. Can someone confirm this is the case when they try a similar exercise of linking a large group of images or LiDAR tiles, then grabbing that group and moving it into a folder?

Attachments:
Mackenzie_Delta_List_of_Image_Tiles_Issue_With_Last_Image_vs_Last_Folder_v2_Nov14_2019.jpg

adamw


9,144 post(s)
#16-Nov-19 13:18

Moving or copying data source components between folders does not copy data in those data sources, what gets copied is just the info needed to connect / open that data.

In the screen though, there is data copied - the Area_01_be_1m_XYZ folder contains both the data sources for linked files as well as tables and images copied from them. Regarding the last table + image pair not copying over - how exactly are you copying them? Open all data sources, Ctrl-click all tables and images, then copy / paste (or drag and drop) into a folder in the MAP file? If you are doing something else, one tempting thought is that maybe the last table + image pair somehow ended up being unselected and thus did not copy, but that has to be established.

StanNWT
173 post(s)
#04-Dec-19 22:41

Well some news for you folks that have helped me test performance on large data sets. I'm building a new home workstation.

  • AMD Threadripper 3970x
  • 128 GB DRR4 3200 MHz RAM
  • 2 x 2TB Samsung 970 EVO Plus NVME drives
  • ASUS Strix 2080Ti
  • ASUS Zenith II Extreme motherboard
dchall8
710 post(s)
#05-Dec-19 00:51

That should handle some large data sets. 4,000+ CUDA cores on the Strix

Can't wait to see envy your performance reports.

Dimitri


5,994 post(s)
#05-Dec-19 08:43

The new 3970x (32 cores / 64 threads) is a huge win for AMD. As PCMag put it: "AMD's 32-core Ryzen Threadripper 3970X performs so far ahead of the curve that it practically creates a new class of consumer-accessible CPU." If you can afford it, the 3970x is the best there is today.

For those on a budget, there are lower cost alternatives that also provide good bang for the buck. Prices on prior generation Threadrippers, like the 24 core / 48 thread 2970WX at about $900, are starting to tumble, and the new 12 core / 24 thread Ryzen 9 3900x at $500 is a great deal (the 16 core Ryzen 9 3950x is still too hard to find...).

Release 9 to date has been tuned to optimize up to around 20 or 30 threads on CPU, depending on what is being done, given that covers what most users to date have been configuring. Performance isn't going to be worse with 48 or 64 threads, but it's not going to scale to being dramatically better, at least not for now. Engineering is working away (using Threadrippers, mainly, as test platforms of course...) to expand automatic tuning for many more threads.

Executing many threads on many CPU cores is a very different deal than executing on many GPU cores, so optimization/algorithms for that has to be adjusted as the number of threads that might be used gets bigger and bigger. For most users, the high end today is 24 or 36 threads, but as StanNWT's "home" rig shows, a breakout into 48 or 64 threads is starting to happen.

It's not just about supporting rare, but totally cool, super-machines like Stan's: as Manifold begins adding distributed capabilities (big jobs dispatched to multiple machines working in parallel on your local net or in the cloud), even if the machines are only eight core or four core machines when you get a few dozen of them working together you can have a lot of threads. That too, is a different deal for optimization / algorithms because of different latency as compared to having all those threads within the same chip, but automatically optimizing for many threads, 64, in the same CPU is a great and necessary first step.

Bottom line: don't be surprised if execution for now with 64 threads isn't a whole lot faster than with 24 threads, but expect good things as Engineering dials up effective use for many threads in upcoming builds.

Congrats on that new machine! It's definitely the best. :-)

StanNWT
173 post(s)
#05-Dec-19 20:16

Don't forget the 3990x will be announced in January 2020, likely at CES, and the 3980x is still a likely product. So if the Manifold engineers are interested in building out to 128 threads, that's a great target. I'm glad to see the possibility of network processing across multiple machines, like the 3D animation software has render nodes. I'm hoping that we can have an instance of the main machine being the licence and using other machines not having to have a full licence, i.e. no GUI, but using the full hardware capabilities of those processing nodes to add to the collective processing power of Manifold. Cloud based rendering sounds interesting as well, but where I work and live, the internet is very slow and can and does easily go down.

Have any Manifold engineers got access to a dual EPYC 7742 or 7H12 server to try to stretch Manifold 9 a bit?

My home internet is 15 Mb/s down, 1 Mb/s up, and that's as fast as is available for home use and it's ADSL. Work internet/network bandwidth is not fast either so, the cloud is an expensive waste of time in my opinion.

StanNWT
173 post(s)
#05-Dec-19 20:47

Once I get the new computer set up and running nice and stable I'll be uninstalling Manifold 8 and 9 from my old laptop. I'm assuming when I install them on my new workstation, it'll just use one of my remaining 4 tokens?

Dimitri


5,994 post(s)
#06-Dec-19 11:34

I'm assuming when I install them on my new workstation, it'll just use one of my remaining 4 tokens?

Yep.

Dimitri


5,994 post(s)
#06-Dec-19 11:33

I'm hoping that we can have an instance of the main machine being the licence and using other machines not having to have a full licence

Current thinking is that the free Viewer will be on worker nodes. That would make it easy to install on however many systems you wanted, either on your local network on in the cloud. Cloud providers like that too, since no SN/activation is required.

Have any Manifold engineers got access to a dual EPYC 7742 or 7H12 server to try to stretch Manifold 9 a bit?

Manifold has way more hardware than necessary. That's good, but it's only part of the picture since optimizing use of hundreds of CPU cores depends on what is being done.

So far, very many cores in a box (dual-CPU motherboards, running two EPYC 7742 CPUs with 64 cores each = 128 cores / 256 threads) have mostly been used by what AMD calls "Hollywood creators," running rendering/special effects. The parallelism in that business is very straightforward, basically a lot of the same thing done over and over for more and more frames. It's easy to scale from a dozen cores to many, so AMD has been very successful just dropping in new CPUs with bigger and bigger core counts.

GIS is different in that manycore CPU parallelism can involve significantly different approaches for the very different types of parallel processing that are done with vectors, rasters, databases, SQL, data access, computation, etc. Part of the challenge is not just getting parallelism done right for one thing, like a raster calculation, but for many different things, and also for a mix of those many different things that can change from moment to moment. 9 already does that for typical core counts, but it would be a mistake to just go forward blindly without tuning for the significantly larger core counts now coming into use.

Simulations and other lab work can go a long way but you can't beat real life experience in such totally new environments as distributing a wide variety of GIS tasks over hundreds of CPUs. Good feedback from the community on real life applications will help the system wring all possible performance out of lots of cores.

Those people who are running 32, 64 or 128 core machines, or (when distributed computing comes out) who are launching tasks in networks with hundreds of worker CPU cores, should report their experiences in different tasks and maybe volunteer to try specialized builds that can probe different optimizations. That will help tune lab results to a wide range of real life uses.

I expect that just as 9 already has had a steady stream of performance improvements and optimizations based on real life scenarios reported by the community, just so as core counts increase that will continue. It's already a pretty wild world out there, as technology has shifted gears from increases that amounted to no more than a couple of more cores per year to suddenly, jumps of 24, 32 and 64 cores at a time. It's really a great time to be running parallel, to take best advantage of all that new power.

Dimitri


5,994 post(s)
#13-Dec-19 06:12

Progress on that front... yesterday's build 9.0.170.1 includes advances to better use more threads. It's just a first step, but a good first step.

Dimitri


5,994 post(s)
#05-Dec-19 09:40

Here's a cool video on your new CPU, from AMD: https://youtu.be/ll2cipUZWPQ

oeaulong

242 post(s)
#05-Dec-19 15:07

Made a wish list machine with these specs last weekend. Schpeedy!

adamw


9,144 post(s)
#13-Nov-19 06:35

+1 to dchall8, try linking LiDAR files instead of importing them, that should be even better. The LiDAR dataport now implements a specialized index optimized for point clouds, which is much more efficient than the regular spatial index. We are going to allow MAP files to use a similar index for points, but this did not happen yet, so currently linking LiDAR has an advantage over importing it.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.