Subscribe to this thread
Home - General / All posts - New video - Creating Terrain Elevation from a NASA PDS Table
Dimitri


5,598 post(s)
#06-Nov-19 11:12

There's a new video on the video gallery page. This one shows how in six minutes you can do very many steps in Manifold, including a Kriging operation to create a raster terrain elevation surface from a table/drawing. It shows a variety of moves that an experienced Manifold user can do to make workflow very fast.

rk
341 post(s)
#06-Nov-19 13:50

Good video. I noticed that screen capture did not include mouse cursor.

Dimitri


5,598 post(s)
#06-Nov-19 14:28

Yes... don't know why that happened, so the video's been redone. The new version is now up.

dchall8
662 post(s)
#11-Nov-19 22:01

I find myself no longer employed by the central appraisal district, so I'm going to ask a question that would be easy to answer if I still had M9 and the LiDAR data on my computer. Can the Kriging image be exported as a .dem file? ...or is that a feature we can look for in the future?

When the CAD received their LiDAR data, it came with .dem files for each geographical tile. The .dems were only for the ground surface. Inside the LiDAR cloud the points were categorized as soil, low shrubs, medium trees, tall trees, buildings, culverts, and miscellaneous. The CAD's interest was to find the buildings hiding below the trees. It would have been nice to run Kriging on the buildings data with hill shading to see the angles of roof tops. I believe M9 had not yet included Kriging back when I was working with the LiDAR.

Dimitri


5,598 post(s)
#12-Nov-19 08:10

Can the Kriging image be exported as a .dem file?

Not as DEM file, but in 24 other formats. See the list for images in the Exporting topic. I recommend using TIF, which Manifold will export automatically as a GeoTIFF, or ESRI GRD. GeoTIFF in particular is a much more popular interchange format these days than DEM.

StanNWT
145 post(s)
#12-Nov-19 21:08

When I saw this video I decided to experiment with some LiDAR (xyz) files I had. I was able to import them and merge them very quickly. I'm using 9.0.169.9. Importing took some time but the merge of 3.5 GB of Lidar data took 1.76 seconds. I'm going to bring in the rest of a large area of LiDAR data and see how fast it imported. I can post a full log file sans directory paths, if you're interested. I did notice that the Google Earth Hybrid v2 imager server data wouldn't load in 9.0.169.9, but I used Bing maps instead. However, in my neck of the woods Google satellite view is far better for up to date and high resolution coverage.

dchall8
662 post(s)
#12-Nov-19 22:06

Wait, wait, WAIT! Before you try importing a number of LiDAR tiles, try "linking" to one, instead of importing, and see if it loads practically instantly. After that if you try to do something with the LiDAR, like merge it, you may have the same problem with the import time.

I had the same issue today with Google Earth Satellite v2. Try the other one right underneath Bing Maps Street Map.

StanNWT
145 post(s)
#13-Nov-19 23:44

Yes I did use linking, it was significantly faster. The only semi-slow part was going into the schema for each (xyz) file and changing the projection. Yes I'm sure there's a programatical way to do it in one go. But the merge took 1.876 seconds on linked data. It saved including the cache for each linked (xyz) LiDAR image in 20 seconds. The resulting (map) file is 1.21 GB

The Google maps hybrid v2, streets v2, terrain, all don't render in the map at all. Bing maps hybrid does render.

I'm now using 9.0.169.10.

I was previously using 9.0.169.9

I did install the new vc++ x86 and x64 runtime installers as suggested for the new build.

Dimitri


5,598 post(s)
#14-Nov-19 06:04

The Google maps hybrid v2, streets v2, terrain, all don't render in the map at all.

If instead of using the Favorites choices, you use File - Create - New Data Source and then in the image servers list choose the original Google versions (not v2), those still work.

tjhb

8,926 post(s)
online
#14-Nov-19 06:29

Nice deduction!

dchall8
662 post(s)
#14-Nov-19 18:11

When I was working with LiDAR, I linked to 20 or so LiDAR tiles, dragged the contents of each into a new drawing, and then made adjustments to the new drawing. I don't know if there is a time savings with that or if it just would take 1.876 seconds x 20 tiles.

StanNWT
145 post(s)
#14-Nov-19 21:10

Well the total time to save the 1.2GB file, with the merged grid and 20 linked LiDAR tiles was 20 seconds. I don't think that's too bad considering my storage is a Drobo 5D with a 250 MB/s read/write speed. Yes I'm sure it would likely be much faster on an M.2 SSD, but I'm trying to keep all my files in the place where I have the room, which is the RAID. I do have to spend the time editing the schema to change the projection of each linked data set, from the default to what it's supposed to be, but once done the linked or even imported tiles work great.

I'm going to be demonstrating the speed of the LiDAR merging later this afternoon to a colleague. I will also show him the ability to play with Bing Maps or Google Maps web services when you copy the image layer into Manifold, then you can change the styling of the imagery to suit your needs/wants. There's nothing else that I've used in remote sensing software that can do that on Bing Maps or Google Maps streaming imagery. So looking at the LiDAR with hill shading after processing it with the free viewer or just Manifold 9, and playing around with the streaming imagery will be enlightening. They use PCI Geomatica or ENVI to do their remote sensing work, they also will use remote connections into some NASA servers to process imagery or even a server within the organization but not locally in town, so it's been problematic and they can't get the performance they'd like.

tjhb

8,926 post(s)
online
#14-Nov-19 21:24

I do have to spend the time editing the schema to change the projection of each linked data set, from the default to what it's supposed to be

As you suspect, you can pretty easily do that with a simple query--in theory one click once it is written.

In the simplest case, just UPDATE mfd_meta using a WHERE filter on some combination of Name, Property and Value.

For example if the default projection assigned is always wrong (i.e. if there are no other components sharing that projection, for which it is correct) then you can just filter on Property and Value, and replace Value for all matched cases.

(Assuming there is not a fixable reason why the wrong projection is assigned by default in the first place.)

StanNWT
145 post(s)
#15-Nov-19 17:22

Hi Tim,

I think the problem is that the default coordinate system is EPSG: 3857. I needed all the linked data to be in EPSG:6337. So I would liked if there was an easy way to set the default for that session to be EPSG: 6337, so I wouldn't have to reset the schema for each tile individually. Hopefully if the default was altered for that session it would make the default for each tile EPSG: 6337. Of course once each tile in a directory on disk has had it's coordinate system set to EPSG: 6337, if you've chosen to save the cache option, you won't need to set it again, if you use the tiles again for another project and redo the linking or importing and finally merging, or whatever else you want to do. Some might not want to save the cache to save disk space though.

StanNWT
145 post(s)
#15-Nov-19 17:17

OK an update to the LiDAR data processing.

I decided to grab a larger data set.

163 tiles, 15.42 GB in size. It took 14.921 seconds to process. That's roughly 1GB/s

I in order to make having all those linked tiles more easy to organize I decided to make folders for each group and place the tiles for each folder using the same folder name in Manifold as the folder on disk so I can keep track of things. I moved the linked tiles into those folders, I think based on the final (map) file size, this copied the tiles. Yes this map file is now 11.1 GB, that includes 1 m contour lines that I generated. It took 46 seconds to generate 1 m contour lines on that merged LiDAR data set. It took 74 seconds to save the file. Each linked directory of tiles I chose to save the cache for.

One interesting thing I noticed is that when you have a list of tiles, that you're copying into a folder, is that you get a list of the images in the folder below the folders of each tile, for each tile except the last folder in the list (see attachment). You can see that be_117d01j01 doesn't show up in the list at the bottom. This is reproduced for every directory that I have for all the other groups of LiDAR tiles. Can someone confirm this is the case when they try a similar exercise of linking a large group of images or LiDAR tiles, then grabbing that group and moving it into a folder?

Attachments:
Mackenzie_Delta_List_of_Image_Tiles_Issue_With_Last_Image_vs_Last_Folder_v2_Nov14_2019.jpg

adamw


8,750 post(s)
#16-Nov-19 13:18

Moving or copying data source components between folders does not copy data in those data sources, what gets copied is just the info needed to connect / open that data.

In the screen though, there is data copied - the Area_01_be_1m_XYZ folder contains both the data sources for linked files as well as tables and images copied from them. Regarding the last table + image pair not copying over - how exactly are you copying them? Open all data sources, Ctrl-click all tables and images, then copy / paste (or drag and drop) into a folder in the MAP file? If you are doing something else, one tempting thought is that maybe the last table + image pair somehow ended up being unselected and thus did not copy, but that has to be established.

adamw


8,750 post(s)
#13-Nov-19 06:35

+1 to dchall8, try linking LiDAR files instead of importing them, that should be even better. The LiDAR dataport now implements a specialized index optimized for point clouds, which is much more efficient than the regular spatial index. We are going to allow MAP files to use a similar index for points, but this did not happen yet, so currently linking LiDAR has an advantage over importing it.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.