Subscribe to this thread
Home - General / All posts - M9 point elevation data (LiDAR) to DEM to contour lines
tonyw
528 post(s)
#09-Apr-19 03:52

In post http://www.georeference.org/forum/t141745.13#141746 Dimitri wrote with respect to starting with point elevation data and creating contour lines:

#10-Feb-18 19:34

That's a very ambitious project, not for the faint of heart.

The SQL Functions topic lists the arsenal you have if you want to use SQL and the API doc is here if you prefer scripting. SQL in 9 is way more capable than the surface transform pane in 8, but that doesn't mean you have to use SQL for project like this. (Always check the query builder in the Command Window for the most up to date list of SQL functions).

I have LiDAR data, specifically .laz files with point elevations. Recently in M9 I found I was able to generate contour lines, my process is described below. So were some surface tools quietly introduced to M9 over the last year?

I import .laz files and add a spatials index. This is described in the manual here: Start > Formats and Data Sources > LAS, LAZ LiDAR Then about 2/3 of the way down that section of the manual, it reads:

"Zooming in, we can see better detail. However, viewing LiDAR as colored points is neither efficient nor appealing. We can do better by interpolating the vector point layer into a raster image using the Kriging transform template, as discussed in the Example: Vector to Raster using Kriging topic".

I want ground contours so I select in the table of LiDAR point elevations where Return_number = Number of Returns to get the last return which should be the reflection off the ground. The link to Kriging Transform brings me to Start> Examples> Transform> Example: Vector to Raster using Kriging

Applying Transform > Interpolate, Krigging, set th Z to field with heights, accept the rest of the default values, Restrict to Selection (LiDAR reflections from the ground only, not tree foliage) and Add Component. That generates a raster image and in the next Transform, step I can Transform > Contour Lines (or Contour Areas). And I have my contour lines (or areas).

Before I proceed further with the rest of the 59 .laz files that this is the process to generate contour lines.

Attachments:
M9 LiDAR to DEM to contours.JPG

Dimitri


5,452 post(s)
#09-Apr-19 09:33

So were some surface tools quietly introduced to M9 over the last year?

Many new tools have been introduced. It's a lot to keep up with, so the best approach is to switch to each new Cutting Edge build as it comes out. You then can adjust to the list of new items published in this forum with the release notes, and also published within a day or so of the release in a slightly more expanded summary in the Changes and Additions topic.

In the days after a new Cutting Edge build comes out, topics that need adjusting in the usual manual get adjusted, and new topics get added for major features, like the Example topic you mentioned. Features that involve changes in many places can take a week or two to propagate through the documentation, like the big Schema changes currently bubbling through many topics. Videos will also often get done to show new features. See the videospage for a list of videos.

Kriging and contours have been around for a long time. There are two contours videos, but I don't think there's a Kriging one. Such features are not called "surface tools" in 9 since they are just considered part of the general package, dealing with vectors or rasters and interchange between them.

The illustration you attached seems to do the trick. My advice would be to take advantage of the Edit Query button to turn what you are doing into a query. You can then modularize the query like in the RGB example to make it easier to apply the query over and over to many files, instead of having to point and click your way repetitively through the same dialogs.

tonyw
528 post(s)
#09-Apr-19 19:57

Thanks for the tip to use the Edit Query button to be able to re-use the steps.

Dimitri


5,452 post(s)
#09-Apr-19 15:14

Forgot to mention: there's another example that does kriging.

tonyw
528 post(s)
#09-Apr-19 19:52

Thanks Dimitri. I started writing a much longer reply here. I've erased it and will keep it shorter. I get frustrated because I don't have the technical adeptness others have in geoinfomatics. I'm just a Manifold end user much like I use a word processor. I type and my text appears as I wish on the screen. I don't have to code each sentence or paragraph (yes, I used LaTEX at one time for a thesis). It seems being able to go from point elevation data to contours existed in Manifold for some time, I see Kriging first appeared last July 2018 in build 9.0.167.4. However I have no idea what Kriging is for other than interpolating where you have no or little data. But in my mind I have great data, LiDAR data at great resolution so who needs Kriging? I want to generate contour lines. The demos in the manual start with contour lines or DEM. So right away that's Catch 22 for me, I can't start with contour lines because I don't have contour lines.

The videos are great, I've learned alot about navigating Manifold dialog boxes. The manual is very detailed. I'm sure the adept folks instantly knew, "hey Manifold has Kriging, now I can process the ton of LiDAR data I have". That's not me, I'm not in that rarefied atmosphere. If I can make a suggestion, sometimes you just have to dumb things down and state the obvious. Something like, "Now with release 9.0.167.4 users can import elevation point data such as LiDAR .laz files, generate digital elevation models and contour lines or contour surfaces. And here's a demo from beginning to end".

I've had two projects since July 2018 where i had to ask a favour from someone in an ESRI shop to provide me contours. I only discovered last week that I could generate contours myself because I was looking in the Start> Formats and Data Sources section of the manual and saw the .laz format which had a description of using Kriging to create a raster from point elevation data. But then the demo starts with contour data so I gave up. However I thought later what the heck, let me power through Kriging on point data and see what happens. I'm glad I persevered. To my delight I generated a DEM from which I could generate contours. There is probably a lot more I can do with Manifold...if it was just a bit more obvious to the non-technical users.

Dimitri


5,452 post(s)
#10-Apr-19 05:44

There's a lot in Manifold, I agree. But it's like most things in that if you take one step at a time you can cover a lot of ground.

LiDAR data is point data. It's not continuous data, it's just a lot of points. When you zoom out, sure, it looks tightly packed, but when you zoom in those points can seem far apart from each other. What's in between? The LiDAR data doesn't say.

Kriging just converts a scattering of points into a surface, interpolating in a reasonable way to fill in the surface in between LiDAR points, and also interpolating in a reasonable way to create an evenly-spaced raster where LiDAR points are not necessarily evenly spaced. The topic I suggested has a biographical sketch of Krige and how he came to invent the technique named after him.

Krige was dealing with natural resource exploration, where he had a scattering of drill holes that sampled what was underground, and he needed to interpolate between those locations to try to figure out how an ore body was laid out underground, so he came up with the statistical interpolation method now called Kriging.

Once you have a surface, you can create contour lines on that surface. So, the process is:

LiDAR drawing -> Surface made with Kriging -> Drawing of contour lines

You might ask, why doesn't Manifold provide a command that collapses those two steps into one?

The answer is to give people simpler ways of getting the big flexibility they want. Kriging isn't just used for the sole and exclusive purpose of making surfaces from LiDAR points that will be used for contours. It's used in a huge range of different circumstances. So there's a Kriging facility that works well for all of those.

Likewise, contours aren't just generated on surfaces created by Kriging from LiDAR data. Contours are created from a very wide range of different surfaces that have been created in different ways from many different types of data. So there's a contours tool for that.

Using the two commands in sequence allows enormous variability and flexibility. For example, you might want your LiDAR data as a surface not for contouring but for other purposes, such as creating hill shaded terrain elevation displays. By providing a large, but comparatively limited, set of general purpose commands that can be combined and re-combined as you like, you can do millions of different things exactly as you prefer.

In contrast, by having a special purpose command for each possible path people want to take, you end up with millions of special purpose commands that nobody will ever be able to learn or use (if even it were practical to create a special command for each possible pathway or task somebody might want to do with a GIS).

There really isn't all that much to learn in Manifold - it's not like learning, say, French, where you need to memorize about 3000 core vocabulary verbs, nouns and adjectives, plus about 100 core rules for grammar and around 1000 irregularities.

In contrast, Manifold has about a dozen major dialogs and, for most GIS work, only about 50 key commands that are highly regular and which cover almost all work, and none of which needs to be memorized, since they are always at hand in menus or transform templates. That can be learned in about four days of attentive study. SQL - not required, but a great assistant when you want it - at the simple level used in GIS can easily be learned in another two days of attentive study.

Where people get it wrong is they fail to invest a day or two into serious study - and I mean a full eight hour day of attention, not seven hours of walking around, having lunch and checking emails - into getting the very fundamentals down. Do that, build a sound foundation of the basics, and all the rest falls into place very quickly.

dchall8
614 post(s)
#11-Apr-19 14:51

Good analogy with the word processor, Tony. Mathematically you could think of language as a piecewise function with punctuation marking where the thoughts become modified and the future direction of the thought goes, but who does that? I was with you on the use of Kriging...until you wrote this, so THANKS because I've wanted to make contour lines for 2 years. I do recognize that M9 is a work in progress and eventually this sort of application will surface, so to speak.

I understand the oil and gas industry is using extraordinarily dense LiDAR data from poles to calculate material fill requirements for drilling pads. They must be using Kriging to make the surface.

Just out of curiosity, what do you use contour lines for? All the Realtors in my area are using Google Earth to view terrain except for one who cannot seem to make it work.

tonyw
528 post(s)
#11-Apr-19 21:52

Hi dchall,

Just out of curiosity, what do you use contour lines for?

I use contour lines at a couple of different scales. One use is to determine boundaries for new properties or lots in remote unsurveyed areas. The property sizes can range from 50 ha to several thousand hectares, carved from watersheds that have no other surveyed properties. So some criteria are to choose areas with access to roads or safe moorage, access to potable water, areas that are relatively flat to build buildings and to layout roads. I have a lot of latitude on where I draw the boundaries. Using orthophotos, Google Earth, and provincial contours at 20 m intervals I draw my proposed boundary lines for the lots. I use visual cues such as slide runouts, where vegetation changes, where people have built roads, etc. as indicators of topography. In the rare instances where I can get LiDAR data the more precise contour lines give me more information to better draw the boundary lines.

The other use is at a larger scale on smaller sized areas for instance where to site infrastructure and to have an idea of ground conditions. This narrows down the candidate areas before going to check them out on the ground.

Without contour lines from LiDAR I only have contour lines at 20 m intervals which is too coarse and based on much less accurate methods such as stereoscopic interpretation of air photos.

I never thought about it, I suppose my task is like Real Estate, I'm creating future properties.

dchall8
614 post(s)
#17-Apr-19 15:26

Have you tried using the hill shading effect on the DEM view of the LiDAR? To my eye the hill shading is much more revealing and informative than contour lines.

Attachments:
Bandera Area with contour lines.jpg
Bandera Area with hill shading.jpg
Bandera DEM without hill shading.jpg

tjhb

8,760 post(s)
#17-Apr-19 15:57

Excellent thinking. Great example.

Mike Pelletier


1,602 post(s)
#17-Apr-19 16:47

Also, while not as pretty, converting DEM to % slope (9 can currently do) is informative for siting roads and buildings. Aspect is also helpful for sun exposure information.

Dimitri


5,452 post(s)
#11-Apr-19 18:42

users can import elevation point data such as LiDAR .laz files, generate digital elevation models and contour lines or contour surfaces. And here's a demo from beginning to end".

Sounds like a great idea for a video! :-)

mikedufty

843 post(s)
#12-Apr-19 02:28

You could also consider using interpolation-triangulation rather than kriging. That effectively takes straight lines between your points, so should be as good if you have dense data. I tend to prefer it because I feel I understand what it does.

Kriging has long been on my list of things I should learn about, but I haven't succeeded yet. The default kriging in surfer does a good job of extrapolating an aesthetically pleasing surface between sparse data points and we use it quite a bit. I have been unable to reproduce the same results in Manifold 8. I tried copying the kriging parameters from surfer, but manifold doesn't seem to take quite the same parameters.

Trying to use it in M8 for groundwater levels it always seems to come up with levels well outside the range of the data points. I should have another go in 9 and see if I can do better. Would be nice if M9 can replace surfer.

Dimitri


5,452 post(s)
#12-Apr-19 05:16

Would be nice if M9 can replace surfer.

The key to that is to send in suggestions with what you are doing in Surfer that you want to see in 9.

Don't over think it or get grandiose. Break it down into specific commands or workflow steps. Start with the top three commands/facilities you use in Surfer. See if you can do those in 9. If not, send them in as suggestions.

Most of what people do with a package over and over tends to be from a relatively small set of commands. Start with the smallest possible set of commands, ideally just two or three, you use in Surfer and send those in. That helps get the ball rolling in terms of thinking specifically about individual functions that can be quickly added.

adamw


8,579 post(s)
#19-Apr-19 13:19

Regarding this:

But in my mind I have great data, LiDAR data at great resolution so who needs Kriging? I want to generate contour lines.

We are planning to add means to generate contours directly from points / points+lines / lines, using triangulation as an intermediate step. This has both pros and cons.

Pros: you don't have to create an intermediate image, you don't have to balance between performance (high resolution harms performance) and quality (low resolution harms quality).

Cons: you don't have a lot of control over the intermediate surface - it's just triangulation with no fancy parameters and smoothing techniques, you cannot post-process the generated surface to improve its quality before generating contours.

tonyw
528 post(s)
#21-Apr-19 05:02

Adam, how much of the point elevation data needs to be loaded into a .map project? I found that in order to add an index to the table, I needed to import the point elevation data in order to add an index. Linking did not allow me to add the index. Importing takes considerable time. On a current project, I have 59 .laz files. Is it possible to link to a .laz file and create/add the index to a table that only exists in Manifold with a relational link to the linked data file? At that point, M9 should be operating with the data as if it had imported the point elevation data in the process to create the raster elevation image.

I may not have gotten the terminology correct but in other words is there a way to link to the point elevation data and avoid the time needed to import that data while still adding an index to the data? Assuming the original point elevation data isn't changed, all this is a precursor to the next step to generate the raster elevation image. Once the raster elevation image is created, I no longer need and can dispense with the point elevation data.

On the other hand, maybe nothing is gained because for the Kriging or Triangulation, the point elevation data needs to be processed so eventually needs to be imported. Anyways I ask because importing the point elevation data takes some time. Linking is relatively fast.

Dimitri


5,452 post(s)
#21-Apr-19 10:21

I found that in order to add an index to the table, I needed to import the point elevation data in order to add an index.

Correct. LAS files are not database files. You cannot create an index within those files. When you link a LAS file into Manifold you are operating within whatever are the limitations of LAS files.

Manifold will do as much as possible to present that data source using familiar user interfaces, such as showing the contents as a table, but, as discussed in the Importing and Linking topic, that only goes so far. Manifold cannot add capabilities that the file format does not have. I suggest a re-reading of that topic.

If you want to have an index on the data in your LAS file, you must put the data into some format that allows indexes. For example, you could copy the data from the LAS file and save it into a PostgreSQL database. Or, more simply, just import it into a .map project.

Leave the data in a format that does not allow for an index and you can't get around the limitations of the format. Link to it in some way? Sure. That will be slower than importing it. Just bite the bullet and import it. That's a one-time process and then you have it in super-wonderful .map format.

The general rule is that if data is worth working with, it is worth taking the time to import into a .map file and then saving the .map. Then, forever more, you get speed and all the power of Manifold facilities (not the least of which is nested links to .map files that lose zero performance while greatly facilitating archival handling of data...) that formats such as, for example, CSV, shapefiles, LAS/LAZ, do not provide.

You may think "well, I only plan on doing this once so I'll tolerate the slow performance and other limitations of leaving it in the original format..." but that usually ends up being short-sighted. If you have to touch that data more than once you end up spending more time, and it could be that the overall time to do what you want to do just once will be faster if you first import than if you work with the data left in the original format and linked into the project.

adamw


8,579 post(s)
#23-Apr-19 08:53

We have plans to extend the dataport for LAS / LAZ to do a specialized external index, so that you can avoid importing everything but still get fast spatial operations. We are currently working on other things, but as we cross various t's and dot various i's, this comes closer and closer to the top.

Technically, kriging / triangulation don't really need a spatial index. It's rendering that needs it. So, once we allow doing contours from vectors directly, it would be possible to use this on linked data even before we extend the dataport to support a specialized index - possibly with some minor use of Edit Query and editing component names.

tonyw
528 post(s)
#23-Apr-19 15:34

Hi Adam

So, once we allow doing contours from vectors directly, it would be possible to use this on linked data

That would be amazing, linking to rather than importing point elevation data would significantly reduce the time required to arrive at contours. I often use Manifold in real time in meetings to support the discussions. With this feature it's feasible to answer the question "can you show us contours and break out slope into categories"? "Sure, give me a moment".

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.