Subscribe to this thread
Home - General / All posts - Moving/reprojecting data in Manifold Viewer
dchall8
1,008 post(s)
#07-Aug-17 18:06

I'm importing 22 million records of LiDAR point data. When I import the data into Manifold Viewer it comes in with a red flag message that the drawing cannot be rendered because it has no spatial index. When I click the Build Temporary Index button, the red flag goes away. I have selected all the records classified as buildings according to the LiDAR source. There are 669 of those, so this is a small set and hopefully manageable to practice with. The problem is that when I overlay a Google Satellite image the data are in the Sahara Desert instead of Texas. I think the problem is an XY origin issue where Manifold Viewer starts at lat/long 0/0, but my data starts somewhere else. I have xml data which I hope will help me to move or offset the data into place. I just don't know how to do that.

Here is the data from the data table for the first record.

mfd_id 33935

Geom <geom, point>

X 44110863

Y 328430614

Z 50900

intensity 570

return_number 1

number_of_returns 1

scan_direction_flag 0

edge_of_flight_line 0

classification 7

scan_angle_rank 27

user_data 0

point_source_ID 69

gps_time 72715625.4370279

The Info pane tells me

Geometry field: Geom

Geometry field type: geom

Geometry index: Geom_x

Geometry index type: rtree

Coord system: Pseudo Mercator

Selection set: 14

I am attaching an image of the Map showing the Manifold Viewer coordinates. Here are some excerpts from the xml metadata for this set of LiDAR data.

 <abstract>This data has been acquired and developed by the Texas 

Natural Resources Information System (TNRIS) for collection and delivery

 of topographic elevation point data derived from multiple return light 

detection and ranging (LiDAR) measurements for the 2256 sq. mile project

 area encompassing Bandera and Lampasas Counties, Texas. Fugro 

EarthData, Inc. acquired 305 flight lines in 36 lifts from December 18, 

2013 to January 25, 2014. LiDAR data collection was performed with a 

Cessna 310 twin engine aircraft, utilizing a Riegl LMS-Q680i LiDAR 

sensor; collecting multiple return x, y, and z as well as intensity 

data. The classified LiDAR point cloud data are delivered in LAS 1.2 

format: 1, unclassified, 2, bare-earth ground, 3, low vegetation, 4, 

medium vegetation, 5, high vegetation, 6, buildings, 7, low 

points(noise), 9, water, 13, bridges, and 14, culverts. Specialized 

in-house and commercial software processes the native LiDAR data into 

3-dimensional positions that can be imported into GIS software for 

visualization and further analysis. The horizontal datum for the data is

 in UTM 14N Coordinates, NAD83(2011) in meters and the vertical datum is

 the North American Vertical Datum of 1988 (NAVD88) in meters using 

GEIOD12A.</abstract>

<spdom>

      <bounding>

        <westbc>-99.609375</westbc>

        <eastbc>-99.593750</eastbc>

        <northbc>+29.687500</northbc>

        <southbc>+29.671875</southbc>

      </bounding>

    </spdom>

 <spref>

    <horizsys>

      <planar>

        <gridsys>

          <gridsysn>Universal Transverse Mercator</gridsysn>

          <utm>

            <utmzone>14</utmzone>

            <transmer>

              <sfctrmer>0.999600</sfctrmer>

              <longcm>-99.000000</longcm>

              <latprjo>+00.000000</latprjo>

              <feast>500000.000000</feast>

              <fnorth>0.000000</fnorth>

            </transmer>

          </utm>

        </gridsys>

        <planci>

          <plance>coordinate pair</plance>

          <coordrep>

            <absres>1</absres>

            <ordres>1</ordres>

          </coordrep>

          <plandu>Meters</plandu>

        </planci>

      </planar>

      <geodetic>

        <horizdn>North American Datum of 1983(2011)</horizdn>

        <ellips>Geodetic Reference System 80</ellips>

        <semiaxis>6378137.000000</semiaxis>

        <denflat>298.257222</denflat>

      </geodetic>

    </horizsys>

    <vertdef>

      <altsys>

        <altdatum>North American Vertical Datum of 1988</altdatum>

        <altres>1</altres>

        <altunits>Meters</altunits>

        <altenc>Explicit elevation coordinate included with horizontal coordinates</altenc>

      </altsys>

    </vertdef>

  </spref>

Attachments:
Screenshot 2017-08-07 11.25.54.png

Dimitri


7,413 post(s)
#08-Aug-17 06:38

When I click the Build Temporary Index button, the red flag goes away

Correct. Building a spatial index is resource intensive so the system will not automatically do that for things like LAS, where table sizes can be very big. It's better to do this once in the table with big data than to build a temporary index:

http://manifold.net/doc/radian/index.htm#example__add_a_spatial_index_to_a_table.htm

I think the problem is an XY origin issue where Manifold Viewer starts at lat/long 0/0

Viewer will use whatever coordinate system you tell it to use for a newly-imported component, if the import format does not automatically specify the coordinate system. If you don't tell it otherwise, Viewer will use whatever is the default projection to apply when using projection-unaware formats.

You did not specify the initial projection after import. The data doesn't use the default projection (which you can set as desired, pseudo Mercator by default.) so you must specify what the initial projection should be. See

http://manifold.net/doc/radian/index.htm#projections.htm

danb

2,064 post(s)
#08-Aug-17 06:49

I think it would be a great shame if Radian didn't ultimately build indexes for point clouds. I see Adam mentions adding a specialised index for point clouds at some point here.

http://www.georeference.org/forum/t136445.60#136447

Though I guess this is still in the pipeline?


Landsystems Ltd ... Know your land | www.landsystems.co.nz

Dimitri


7,413 post(s)
#08-Aug-17 07:22

Though I guess this is still in the pipeline?

Sure. Adamw is a very serious and reliable person and you can count on whatever he says to be as accurate as humanly possible.

A tip intended in a friendly way: When adamw or anyone else shares confidential info or advance info in a public forum take it for what it is. Don't make them regret offering insights by being impatient. Instead, apply your skills and experience to contribute razor sharp suggestions on that theme to improve whatever was discussed.

danb

2,064 post(s)
#08-Aug-17 07:27

No impatience here. I just thought I remembered Adam sharing some info in one of the cutting edge change logs which was relevant to the discussion.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


10,447 post(s)
#08-Aug-17 07:48

I see Adam mentions adding a specialised index for point clouds at some point here. [...] Though I guess this is still in the pipeline?

Yes, this is one of the things we are planning to add.

Right now, we can (a) access LAS/LAZ data using queries, (b) show it without optimizations specific to point clouds, (c) write the results of a query back as LAS/LAZ - including without going through import / export, just by linking the source file, running a query in the command window, then using Edit - Export Results to save the results as LAS/LAZ (not available in Viewer).

We are planning to add specialized indexes which will make (b) fast and help (a) as well.

danb

2,064 post(s)
#08-Aug-17 07:57

Brilliant, thanks Adam. I am increasingly thinking that Radian map file storage may be the solution to our ongoing lidar/bathymetry storage problems as it is blindingly fast once the data is inside the map. I see the index as a key component of this so am glad it is still in the pipeline.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

Dimitri


7,413 post(s)
#08-Aug-17 07:00

Forgot to mention... see also the discussions of LiDAR in the release notes for the build and predecessors you are using. (as danb notes below)

adamw


10,447 post(s)
#08-Aug-17 07:42

I think the problem is an XY origin issue where Manifold Viewer starts at lat/long 0/0, but my data starts somewhere else.

I don't think the problem is this.

The metadata you pasted seems to say that the coordinate system is UTM 14, and the false northing of 0 indicates UTM 14N rather than UTM 14S, so let's just set that first:

Open Drawing, Edit - Initial Projection. In the Standard tab, set filter to 'Universal Tran', then select 'Universal Transverse Mercator Zone 14 (N)'. Click OK.

If this will not be sufficient, post a link to the data and we will take a look.

dchall8
1,008 post(s)
#08-Aug-17 15:12

Excellent Adam. I had to build the temporary index first. When I ignored or dismissed that index dialog then applied the UTM 14N I got nuthin. When I built the temp index and applied the UTM 14N, it popped right into place.

I will check out Dimitri's links.

Also as a SQL novice, I am loving the query builder. One of my issues has been with spelling and syntax. I haven't made those errors since getting corrected on select-into-from-where ordering issue I had last week.

dchall8
1,008 post(s)
#08-Aug-17 15:56

THIS IS SO COOL

Now I have something to show the boss to convince her to get both Radian Studio and a CUDA graphics card. GIS mapping is like black magic to appraisal business management, so getting the necessary tools can be an uphill battle. With Radian Studio I will finally be able do what I need to do with the LiDAR data I have been sitting on for 2 years. See attached pix.

Attachments:
Hunting Cabins in Utopia with LiDAR.jpg
Hunting Cabins in Utopia.jpg

dchall8
1,008 post(s)
#08-Aug-17 16:08

And furthermore...

since all I'm interested in is the buildings, I will be able to cut my point clouds down from 25 million points per tile to 20,000 to 50,000 points. I should be able to see the entire county in one layer. There's likely more wizardry at my beck and call in Radian Studio to make this even more efficient. I'm clearly pretty excited about this.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.