StanNWT196 post(s) |
You are correct, that it makes it difficult to help someone when they're not using the precise language and steps involved. I took another run at the process of taking the rivers polyline layer from the File GeoDatabase and creating a point for each vertex. The rivers polyline layer is dragged from the File GeoDatabase source onto an existing map. The polyline feature class in in geographic coordinate system NAD 83 CSRS, which shows up in the layer contents as Latitude / Longitude. The other existing layers in the map are Lambert Conformal Conic. If I double click the table for that particular river polyline layer to check out the number of records, it takes around 8 minutes to read the table at a rate of 48,000/s. I then highlight the rivers polyline layer in the map, which has 15,310,631 records. I then select "Edit" -> "Transform". I then select decompose to coordinates. The code from the "Edit Query" window is below: -- $manifold$ -- -- Auto-generated -- Transform - Decompose to Coordinates - Add Component -- CREATE TABLE [HD_1470009_1 Decompose to Coordinates] ( [OBJECTID] INT32, [Shape] GEOM, [CODE] INT32, [VALDATE] NVARCHAR, [PROVIDER] INT16, [ID] NVARCHAR, [DATANAME] NVARCHAR, [ACCURACY] INT32, [THEME] NVARCHAR, [DEFINITION] INT16, [PERMANENCY] INT16, [ACQTECH] INT16, [CONCISCODE] INT32, [GEONAMEDB] NVARCHAR, [LANGUAGE] INT16, [NAMEEN] NVARCHAR, [NAMEFR] NVARCHAR, [NAMEID] NVARCHAR, [Shape_Length] FLOAT64, [mfd_id] INT64, INDEX [mfd_id_x] BTREE ([mfd_id]), INDEX [FDO_OBJECTID] BTREEDUP ([OBJECTID]), INDEX [FDO_Shape] RTREE ([Shape]), PROPERTY 'FieldCoordSystem.Shape' 'GEOGCS["GCS_North_American_1983_CSRS",DATUM["D_North_American_1983_CSRS",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],VERTCS["NAVD_1988",VDATUM["North_American_Vertical_Datum_1988"],PARAMETER["Vertical_Shift",0.0],PARAMETER["Direction",1.0],UNIT["Meter",1.0]]' ); CREATE DRAWING [HD_1470009_1 Decompose to Coordinates Drawing] ( PROPERTY 'Table' '[HD_1470009_1 Decompose to Coordinates]', PROPERTY 'FieldGeom' 'Shape' ); FUNCTION splitgeom(arg GEOM) TABLE AS (SELECT GeomMakePoint([XY]) AS [Geom] FROM CALL GeomToCoords(arg)) END; PRAGMA ('progress.percentnext' = '100'); INSERT INTO [HD_1470009_1 Decompose to Coordinates] ( [OBJECTID], [CODE], [VALDATE], [PROVIDER], [ID], [DATANAME], [ACCURACY], [THEME], [DEFINITION], [PERMANENCY], [ACQTECH], [CONCISCODE], [GEONAMEDB], [LANGUAGE], [NAMEEN], [NAMEFR], [NAMEID], [Shape_Length], [Shape] ) SELECT [OBJECTID], [CODE], [VALDATE], [PROVIDER], [ID], [DATANAME], [ACCURACY], [THEME], [DEFINITION], [PERMANENCY], [ACQTECH], [CONCISCODE], [GEONAMEDB], [LANGUAGE], [NAMEEN], [NAMEFR], [NAMEID], [Shape_Length], SPLIT CALL splitgeom([Shape]) FROM [Data Source]::[HD_1470009_1 Drawing] THREADS SystemCpuCount(); I am using all the fields and the values per polyline transferred to each coordinate pair which creates new points not for any other purpose at this stage then to test the performance of handling a large vector polyline data set converted to coordinate points. For example multibeam bathymetry or LiDAR as you know has vast amounts of data associated with each time space coordinate, so if I can at least have a few fields for each new point, I'll at least test a large number of points with a reasonable number of attributes... Not really an apples to apples comparison, but it is all I have at the moment with free data. In preview mode when zoomed in, it does show the points from the coordinates of the vertices instantly. I then click the "add component" button. It runs seemingly twice, once it gets to roughly 300 million records, which are the coordinate pairs of each vertex, it takes about 20-40 minutes the total estimated size of this operation looks like it's around 58 GB, the speed is around 130,000 records per second, then it goes through an "insert records" phase which starts from the beginning adding records, it inserts records at a rate of 21,000 records per second. I have no idea where this 58 GB of data is stored, perhaps a windows temp file, since I've not saved the project yet. After 183 minutes Manifold future 9.0.163.4 crashes before the new point layer is created in the project. It's not stored in the File GeoDatabase, it gets made in the project. I know this because the transform I was previously using in error, which was the compose to point template in the transform window, well that is what made the 15,310,631 record presumably multi-point layer. The new layer I was trying to create will create unique points and records in a table for each vertex with all the attributes of it's associated polyline. I can show screen grabs by uploads if required...
|