Subscribe to this thread
Home - General / All posts - Polyline drawing layer converted to points drawing. Both tables show the same number of records.
StanNWT
196 post(s)
#04-Oct-17 19:08

I'm using Radian Studio 9.0.163 and I have loaded as a source database an Esri File GeoDatabase built in 10.x.x. I have the entire Canada landmass can vec 1:50,000 data set as vector point line and polygon feature classes. The entire geodatabase is 46 GB. I've open as a data source the river polyline layer. The drawing and table come in fine. Drawing / rendering that drawing of the polyline layer isn't instantaneous the first time I add the drawing to the map but that's fine. As a test of the triangulation transform to further push Radian based on the 15 million point test of the California road points, I wanted to do the same thing for all the verticies of the entire river layer. There are over 15.3 million records in the polyline drawing / table. When I preview the convert to points transform it's instantaneous. When I add component it takes a few minutes, I think 1440 seconds. When I look at the table and look at the last record. Both the polyline and points tales site the same number of records in the MFDID field on the far left indicating the number of records. How can I have a point drawing / table showing the same number of records when there's clearly over 100 million and likely over 200 million points in that drawing / table? Is the the MFDID a hold over from the record IDs in the polyline drawing / table?

When I do the tranguate all or the voronoi transform via the pop-up at a large scale it's instantaneous but fails to draw when I zoom out at least within a minute. Yes I know I shouldn't be so impatient after all I work in an ArcGIS Desktop (32-bit) world all day long.

I could try it in manifold future as well. But I assume the same results.

I'm going to try this next on the full canada hydro network data set.

Dimitri


7,413 post(s)
#04-Oct-17 20:25

I know you're trying to provide full detail but you're leaving out a lot that is important. For example,

I have loaded as a source database

... is that linked in or imported? Imported means it is in Radian. Linked as a data source means the data is sitting out there in the geodatabase, so you are more testing the performance of ESRI geodatabase format, not Radian.

I could try it in manifold future as well.

Always use Future. I know it doesn't seem that long (only month) since Future forked away from Radian but a month in Manifold engineering time is hundreds of items, some of which are significant improvements in performance, bug fixes, etc.

StanNWT
196 post(s)
#04-Oct-17 20:58

Well I used the add from source, picked the fgb option. Then when I wanted to convert to points I think it automatically copied the data into radian. I could see a copy transform apparently selected and running. I don't think I chose that myself. The polyline drawing is inside the project, in the list. But would it still be linked to the file geodatabase?

If you haven't guessed I am new to radian, hence the testing but have been using GIS since Map II Processor on a Mac classic in the early 1990s.

I'm also typing this in on my smartphone at work. My radian install is in my laptop at home where I've been doing the testing. So I'm not looking at an active screen and writing it all from memory of what I did last night.

tjhb
10,094 post(s)
#05-Oct-17 00:50

Appreciated that you're not in front of Radian now, so you can't do much about this now, but it really does help if you can write each step using the names of actual dialogs and options.

I used the add from source, picked the fgb option.

I think that is File > Create > New Data Source, Type: 'File: gdb'.

But to answer this:

How can I have a point drawing / table showing the same number of records when there's clearly over 100 million and likely over 200 million points in that drawing / table?

You mention the "Convert to Points" transform. This might seem like splitting hairs, but it's not: there's no transform called "Convert to Points", but only "Convert to Point", singular.

"Convert to Point" takes a geom of any type, and converts it to a geom of point type. If the source geom has many coordinates (vertices), the result is a point geom with many coordinates, i.e. a multipoint.

That is a one-to-one transform, one object in, one object out.

It's not the transform you want, since you want individual points in the result. For that, use "Decompose to Coordinates" instead.

(While you're at it, do use the Edit Query button for each transform, for an idea of how they work--even if you can only follow a bit at first.)

Dimitri


7,413 post(s)
#05-Oct-17 07:05

"Convert to Point" takes a geom of any type, and converts it to a geom of point type. If the source geom has many coordinates (vertices), the result is a point geom with many coordinates, i.e. a multipoint.

I find that illustrations help me visualize how the different transforms work. Almost all of the templates have illustrations of starting data and the result shown in the Transform Templates - Geom topic.

To find a particular template quickly, launch that topic in your browser and then hit a Ctrl-F (works in most browsers... I tend to use Opera, Edge or Chrome and it works in all three) to open a Find box in the browser. Enter "Convert to Point" without quotes and the browser will find the template quickly for you on the page you are viewing.

The other thing that helps a lot when working with transforms is to do what that topic does: Create a drawing (takes 3 seconds...), add some lines or areas to it (10 seconds) and then try out the different transforms to see what they do. They'll all preview what they do so you can try out a dozen different transforms in a minute or two. Do that to get your head around what you propose to do because it is a lot quicker to try stuff out with a few dozen objects than with 100 million objects. :-)

Can't resist adding... if anybody wants a classic example why many people curse whoever invented the idea of "multipoints," that is, a branched, single point object that to every sensible observer looks like multiple, separate point objects, well... this thread fits right in. Multipoints have their uses, I concede, but they sure do provide opportunities for confusion.

StanNWT
196 post(s)
#05-Oct-17 07:09

You are correct, that it makes it difficult to help someone when they're not using the precise language and steps involved.

I took another run at the process of taking the rivers polyline layer from the File GeoDatabase and creating a point for each vertex.

The rivers polyline layer is dragged from the File GeoDatabase source onto an existing map. The polyline feature class in in geographic coordinate system NAD 83 CSRS, which shows up in the layer contents as Latitude / Longitude. The other existing layers in the map are Lambert Conformal Conic. If I double click the table for that particular river polyline layer to check out the number of records, it takes around 8 minutes to read the table at a rate of 48,000/s. I then highlight the rivers polyline layer in the map, which has 15,310,631 records. I then select "Edit" -> "Transform". I then select decompose to coordinates.

The code from the "Edit Query" window is below:

-- $manifold$

--

-- Auto-generated

-- Transform - Decompose to Coordinates - Add Component

--

CREATE TABLE [HD_1470009_1 Decompose to Coordinates] (

[OBJECTID] INT32,

[Shape] GEOM,

[CODE] INT32,

[VALDATE] NVARCHAR,

[PROVIDER] INT16,

[ID] NVARCHAR,

[DATANAME] NVARCHAR,

[ACCURACY] INT32,

[THEME] NVARCHAR,

[DEFINITION] INT16,

[PERMANENCY] INT16,

[ACQTECH] INT16,

[CONCISCODE] INT32,

[GEONAMEDB] NVARCHAR,

[LANGUAGE] INT16,

[NAMEEN] NVARCHAR,

[NAMEFR] NVARCHAR,

[NAMEID] NVARCHAR,

[Shape_Length] FLOAT64,

[mfd_id] INT64,

INDEX [mfd_id_x] BTREE ([mfd_id]),

INDEX [FDO_OBJECTID] BTREEDUP ([OBJECTID]),

INDEX [FDO_Shape] RTREE ([Shape]),

PROPERTY 'FieldCoordSystem.Shape' 'GEOGCS["GCS_North_American_1983_CSRS",DATUM["D_North_American_1983_CSRS",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],VERTCS["NAVD_1988",VDATUM["North_American_Vertical_Datum_1988"],PARAMETER["Vertical_Shift",0.0],PARAMETER["Direction",1.0],UNIT["Meter",1.0]]'

);

CREATE DRAWING [HD_1470009_1 Decompose to Coordinates Drawing] (

PROPERTY 'Table' '[HD_1470009_1 Decompose to Coordinates]',

PROPERTY 'FieldGeom' 'Shape'

);

FUNCTION splitgeom(arg GEOM) TABLE AS

(SELECT GeomMakePoint([XY]) AS [Geom] FROM CALL GeomToCoords(arg))

END;

PRAGMA ('progress.percentnext' = '100');

INSERT INTO [HD_1470009_1 Decompose to Coordinates] (

[OBJECTID], [CODE], [VALDATE], [PROVIDER], [ID], [DATANAME], [ACCURACY], [THEME], [DEFINITION], [PERMANENCY], [ACQTECH], [CONCISCODE], [GEONAMEDB], [LANGUAGE], [NAMEEN], [NAMEFR], [NAMEID], [Shape_Length],

[Shape]

) SELECT

[OBJECTID], [CODE], [VALDATE], [PROVIDER], [ID], [DATANAME], [ACCURACY], [THEME], [DEFINITION], [PERMANENCY], [ACQTECH], [CONCISCODE], [GEONAMEDB], [LANGUAGE], [NAMEEN], [NAMEFR], [NAMEID], [Shape_Length],

SPLIT CALL splitgeom([Shape])

FROM [Data Source]::[HD_1470009_1 Drawing]

THREADS SystemCpuCount();

I am using all the fields and the values per polyline transferred to each coordinate pair which creates new points not for any other purpose at this stage then to test the performance of handling a large vector polyline data set converted to coordinate points. For example multibeam bathymetry or LiDAR as you know has vast amounts of data associated with each time space coordinate, so if I can at least have a few fields for each new point, I'll at least test a large number of points with a reasonable number of attributes... Not really an apples to apples comparison, but it is all I have at the moment with free data.

In preview mode when zoomed in, it does show the points from the coordinates of the vertices instantly. I then click the "add component" button. It runs seemingly twice, once it gets to roughly 300 million records, which are the coordinate pairs of each vertex, it takes about 20-40 minutes the total estimated size of this operation looks like it's around 58 GB, the speed is around 130,000 records per second, then it goes through an "insert records" phase which starts from the beginning adding records, it inserts records at a rate of 21,000 records per second. I have no idea where this 58 GB of data is stored, perhaps a windows temp file, since I've not saved the project yet.

After 183 minutes Manifold future 9.0.163.4 crashes before the new point layer is created in the project.

It's not stored in the File GeoDatabase, it gets made in the project. I know this because the transform I was previously using in error, which was the compose to point template in the transform window, well that is what made the 15,310,631 record presumably multi-point layer. The new layer I was trying to create will create unique points and records in a table for each vertex with all the attributes of it's associated polyline.

I can show screen grabs by uploads if required...

StanNWT
196 post(s)
#05-Oct-17 07:57

After re-running the decompose to coordinates, I saw where what I was inferring a second run was going. The first time through is "Inserting records, Scanning records" the second time is actually inserting records. And the closest record count I saw was just over 312 million, reading at 21.5 MB/s, 124,000/s. At 43 minutes it starts the actual inserting records part. It fails when it actually tries to insert the records at around 183 minutes.

tjhb
10,094 post(s)
#05-Oct-17 08:33

More helpful:

Say what data you are using. Where from, what format, how to get it. URLs are good.

Then exactly what you do to import the data.

Then step by step, what steps you perform.

Currently you are a million miles away from that sort of standard, you are just blowing.

Can you try again?

tjhb
10,094 post(s)
#05-Oct-17 21:56

Sorry, I was more harsh here than I intended to be. (I'm not always good at that.)

You're doing absolutely the right thing, just try to take it one step at a time.

The tag for the General forum section says "no question too simple". I would add "no question can be put too simply".

Putting complex things simply is really hard work, especially in new territory.

On the plus side, many people on the forum are happy to put hours (voluntarily) into answering a good question, if you give them the right levers and tools.

Dimitri


7,413 post(s)
#05-Oct-17 15:50

OK... let's have at it. Before we do, though, I have one request: please don't use whatever text editor you were using to create text in the very wide post. :-)

Onward...

Please don't talk about queries or anything else until you tell us precisely how, step by danged step, you get your data into Manifold.

It's frustrating that we still don't know whether you have imported the data into the project or linked into the project and exactly how you did that. Please don't tell us what you infer must be the situation. Instead, tell us step by step what you actually did.

An example:

The rivers polyline layer is dragged from the File GeoDatabase source onto an existing map.

The above doesn't tell us whether you imported the table into the project or linked it into the project. It *sounds* like what you are talking about is dragging and dropping a drawing into a map window from a linked data source where all the data is stored outside the project. But it could also mean you are dragging a table from a data source and dropping it into the local part of the project.

Here is how you should describe what you might have done:

<begin example quote>

1. I read the "Example: Connect to an ESRI GDB File Geodatabase" topic.

2. Following that topic, I launched Manifold Future 163.5 and chose File - Create - New Data Source.

3. I chose Type of File: gdb, clicked the browse button, and browsed over to my C:\files\ESRIstuffGDB folder and clicked on the gdb file in there, clicked open and then clicked Create data source.

4. That created a data source with a cylinder in my project. I clicked the + icon to open it and in there I saw many drawings and tables.

5. The drawing I'm working with is one of those... It is called... everything from here on in is that drawing...

<end example quote>

Reading the above it would be clear a gdb was LINKED into the project with no data IMPORTED, and that everything done thereafter depended upon GDB as a data store.

It's not stored in the File GeoDatabase, it gets made in the project. I know this because the transform I was previously using in error, which was the compose to point template in the transform window, well that is what made the 15,310,631 record presumably multi-point layer.

Well, everything gets made in the project. The question is whether it gets made in the project stored locally in Radian storage or gets made in the project stored in the external GDB.

I can't conclude it is created in local storage from the info above because the description does not hang together: how a transform operates is not the way one knows if something is linked or imported, exactly because of the operational transparency the Radian engine provides in such situations... the operational equivalency being a very good thing, not a bad thing.

My guess - and it is only a wild guess - is that you are having problems because everything is still in the GDB. So let's start with simplifying the situation:

0. Read the example topic. Actually download the Naperville GDB from ESRI's site and duplicate the example so you get a hands-on feel for it.

1. Open a new, empty project. Link the file GDB into your project. Expand the database cylinder.

2. Find the drawing you want within that database cylinder hierarchy. Highlight BOTH the drawing and the drawing's table. Press the Copy icon in the toolbar.

3. Close the database hierarchy, and click the mouse somewhere in the big, unoccupied white space in the project below the System Data. Click the Paste icon.

4. Wait a long time while it copies the drawing and table from the GDB and pastes into the project. When the process is finally done you'll see a drawing and its table appear in the project underneath the System Data folder. As promised in the very first intro topics, imports can be slow but, mercifully, that's normally a one-time task since once you have data in a Radian format .map after that it all is fast.

5. Click on the GDB database cylinder to highlight the data source, and then press the Delete button on the toolbar. That deletes the data source. We do that to make sure there's no GDB around to sow confusion about what is linked and what is imported.

6. Is the drawing and table that you pasted into the project still there? Great! That means you successfully copied and pasted the drawing and its table out of the GDB and into the project.

7. Save the project. The first time you do this it will take a long time. After that it will be quick to open and much quicker to save.

8. Just to be sure you did all this right, close Manifold. Open it again and open the project. You should not see any database cylinders there, just the drawing and the drawing's table that you brought in from the GDB.

The above is a really convoluted way of going about this but I've tried to construct it in a way that it will be impossible for you to leave what you are doing partially in the GDB. The point of doing that is a) to avoid a situation where you have to learn enough about the system to use things liked linked tables from GDB correctly, and b) where you can get on to trying out Radian instead of doing nothing but testing GDB.

My own guess, by the way, is that it wasn't Future that crashed but GDB, and the GDB driver (ESRI's code, not ours) killed everything in which it was embedded. That's one reason we hate incorporating third party drivers because when they crash people blame us. But there's no real way to defend against kill shots from drivers when you have to let them in behind your defenses to get reasonable performance out of them.

Now, it's perfectly possible the Radian engine crashed doing what you were doing, and Engineering is always eager to find such cases. When crashes are extremely rare it is all the more exciting to find one, so that, too can be immediately fixed. So Engineering gets excited but when they hear "GDB" they know they probably won't get lucky.

Given that from what you write it appears almost certain Engineering will be disappointed once again to discover that all your data was left in GDB and so GDB was doing all the data storage. Considering the well-known proclivity of GDB to crash under stress it seems more likely this case is GDB and not Radian. Luckily, it is very easy to remove GDB as a possibility by eliminating the use of GDB as a data store.

That will also help your objective of measuring Radian performance and not inadvertently measuring ESRI performance.

StanNWT
196 post(s)
#12-Oct-17 16:21

Hi Dimitri,

I wanted to reply after testing both on my home laptop and work computers. Both had the same result which is a crash "manifold 9.0 has stopped working".

The dataset that I am using is the Can_Vec 1:50,000 rivers stored in a single file geodatabase. I have all the feature classes from the Can_Vec data.

ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/vector/canvec/archive/canvec_archive_20130515/canada_fgdb

This data set is of all of Canada at 1:50,000.

The data is in separate zip files, each zip file is for a set feature class types. Within each type there are points, polylines and polygons indicated by a (0, 1 or 2) as the last character in the feature class name. The feature class that I'm using is HD_1470009_1.

I suppose you could just download the ftp://ftp.geogratis.gc.ca/pub/nrcan_rncan/vector/canvec/archive/canvec_archive_20130515/canada_fgdb\canvec_gdb_CA_HD.zip file and import from the file geodatabase just for the hydrography, HD_1470009_1, then run the test yourselves?

The newest version of the Can_Vec data is located here:

ftp://ftp.geogratis.gc.ca//pub/nrcan_rncan/vector/canvec/fgdb

The newest can_vec data is stored separately in folders named for the feature class types: Admin, Elevation, Hydro, Land, ManMade, Rest_MGT, Toponymy, Transport. However, the newest version isn't the version I am having issue with. Yes I can do the test with the newest version of the rivers, feature class to try and rule out some error with the layer itself. I can also try a check geometry or repair geometry within ArcGIS to see if there's any issues as far as ArcGIS can determine with the feature class.

The data on the geogratis ftp site is stored separately for each file geodatabase feature class, likely due to size issues. In my file geodatabase I have organized the data into feature data sets but they load successfully into Manifold / Radian Studio.

As you suggested I first copied the feature class, which is comprised of the table and drawing elements in Manifold, I used Manifold Future 9.0.163.5 for the test on both machines, into the project and deleted the source file geodatabase reference from the project, and saved the project before trying the decompose to coordinates transform. I launched Manifold 9.0.163.5 to run the test at 2:40 PM local time and the crash happened at 5:59 PM.

The Windows Event Viewer log info is below:

Faulting application name: manifold.exe, version: 9.0.163.5, time stamp: 0x59d504f7

Faulting module name: ext.dll, version: 9.0.163.5, time stamp: 0x59d504db

Exception code: 0x40000015

Fault offset: 0x00000000022c7536

Faulting process id: 0x2144

Faulting application start time: 0x01d342d0bd17fcfc

Faulting application path: \Downloaded_Software\Manifold\manifold-future-9.0.163.5-x64\bin64\manifold.exe

Faulting module path: \Downloaded_Software\Manifold\manifold-future-9.0.163.5-x64\bin64\ext.dll

Report Id: 482389d4-aee0-11e7-97c9-3417ebbe3cad

The detailed version of the Event Viewer log is below:

- <Eventxmlns="http://schemas.microsoft.com/win/2004/08/events/event">

- <System>

<Provider Name="Application Error"/>

<EventIDQualifiers="0">1000</EventID>

<Level>2</Level>

<Task>100</Task>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2017-10-11T23:59:57.000000000Z"/>

<EventRecordID>78020</EventRecordID>

<Channel>Application</Channel>

<Computer>IK63-SEM0120.corp.ds.gov.nt.ca</Computer>

<Security />

</System>

- <EventData>

<Data>manifold.exe</Data>

<Data>9.0.163.5</Data>

<Data>59d504f7</Data>

<Data>ext.dll</Data>

<Data>9.0.163.5</Data>

<Data>59d504db</Data>

<Data>40000015</Data>

<Data>00000000022c7536</Data>

<Data>2144</Data>

<Data>01d342d0bd17fcfc</Data>

<Data>\Downloaded_Software\Manifold\manifold-future-9.0.163.5-x64\bin64\manifold.exe</Data>

<Data>\Downloaded_Software\Manifold\manifold-future-9.0.163.5-x64\bin64\ext.dll</Data>

<Data>482389d4-aee0-11e7-97c9-3417ebbe3cad</Data>

</EventData>

</Event>

I had over 300 million points when I scan through the details, but it fails when it starts to insert the records.

My workstation at work is as configured as follows:

Dell Precision T3610, 96 GB DDR3 1866 MHz RAM, 1 TB 7200 RPM OS drive, OCZ Revodrive 960, which is where my pagefile, TEMP, TMP folders are stored. I have an Nvidia Quadro M4000 and P4000 graphics cards installed with driver version 385.69 installed. My CPU is an Intel Xeon E5-1620 v2 @ 3.70 GHz. I'm running Windows 7 Enterprise 64-bit. Patches and service packs well that is controlled by the corporate IT environment. I know it's not possible to duplicate my system, but wanted to give some system specs.

As for the super wide post I made earlier in the thread, that is a result of me copying and pasting directly from the expression tab within the Transform dialog.

Dimitri


7,413 post(s)
#12-Oct-17 20:13

I know you're trying but we have to take this step by step. No jumping ahead. Also, please don't volunteer info from event log and stuff like that. Please just take it step by step.

Here's what I mean:

The data on the geogratis ftp site is stored separately for each file geodatabase feature class, likely due to size issues. In my file geodatabase I have organized the data into feature data sets but they load successfully into Manifold / Radian Studio.

Based on the above I do not know exactly what you have in your project and there is no way to download something from anywhere and get an exact match to what you have in your project. So we have to pin that down.

An ideal situation is to just look at the .map file. If you save the .map file how big is it? If you then zip it, how big is the zip file? Do you have a server somewhere you can put that file for download? Is the query you are using in that file?

Some more questions...

As you suggested I first copied the feature class, which is comprised of the table and drawing elements in Manifold, I used Manifold Future 9.0.163.5 for the test on both machines, into the project and deleted the source file geodatabase reference from the project, and saved the project before trying the decompose to coordinates transform. I launched Manifold 9.0.163.5 to run the test at 2:40 PM local time and the crash happened at 5:59 PM.

OK. It becomes muchmore interesting if there is no connection to GDB. Maybe Engineering will get lucky! :-) But still, when you say "run the test" I don't know the exact test you are running and how that relates to what, exactly, are the tables, schemas, etc. in the project. The best way to deal with this is to take a look at the actual project you are running and the actual query. We need that level of detail to make progress.

Dell Precision T3610, 96 GB DDR3 1866 MHz RAM, 1 TB 7200 RPM OS drive, OCZ Revodrive 960, which is where my pagefile, TEMP, TMP folders are stored.

I presume an OCZ Revodrive 960 is some sort of solid state disk, but I have no idea how big it is. You haven't mentioned how much free space you have on the various disks. All that is important to know.

When you get a crash by the way, don't do anything else until you first create a dump file. That's the important thing.

StanNWT
196 post(s)
#16-Oct-17 19:03

Hi Dimitri,

The .map file is 12,613, 248 KB, using 7-Zip with ultra compression, it's 3,240,485 KB.

There is only one drawing and related attribute table in the .map file. I do not have a server at work where I can make available a directory for file upload. My organization uses secure file transfer where a single recipient's email would have to be on the other end and I have a 2 GB file limit per day. I can break the file up into equal parts so it's under 2 GB, and using 7-zip you you download each part? But that requires a recipient email address. I don't personally have a drop box account to host the file for download. I can if you like download that GeoDatabase with the single feature class from NRCan, the one referenced in the ftp site, and connect that as a source, then copy the drawing and table in, and try and redo it so that it's as reproducible without me sending you my map file, if that might help? But I know you want my map file. I have done this with both my work computer and my laptop at home, with very different hardware, OS and installed software environments.

My work computer has:

An OCZ RevoDrive, is a PCI-Express SSD, it's 960 GB in size, it's really 894 GB, currently there is 682 GB free. My page file is 147,372 MB in size, recommended automatically by Windows on the RevoDrive. I have 96 GB of RAM installed. My storage RAID where I have the .map file, is a Drobo 5D with a total capacity of 21.6 TB and 18.76 TB free space. The original GeoDatabase is stored on a different Drobo 5D with a capacity of 10.49 TB and 7.76 TB free space. My C drive is a 1 TB 7200 RPM seagate HD with 663 GB free space.

My laptop at home has an mSATA cache drive of 32GB used by the Dell UEFI as an acceleration drive, and a 1 TB 5400 RPM hard drive, 16 GB RAM, a 2GB page file on either the C drive or that mSATA cache, I'm not sure. There's about 700 GB free on the hard drive of the laptop.

Both computers, have Manifold 9.0.163.5 crash after trying to insert records. I haven't been able to record where in the number of records that crash is, because I wasn't able to be in front of the computer looking at it for 3 hours straight on both machines, 3 hours that is after the scanning records phase. I believe on my laptop the scanning phase went upwards of 350 million records. Seems odd, that this operation has so small of a limit, compared to the forum post I read regarding reading 1.72 billion lidar points. But yes a totally different operation.

How would you suggest I proceed with sending the 7-zip file or portions at a time?

StanNWT
196 post(s)
#20-Oct-17 15:33

Dimitri,

Can I post drop box links in here so that you can download the .7z file? I've not used drop box as of yet in my personal life, as I've not needed to transfer large quantities of data, and I'm not really a fan of cloud storage for my personal info. In work environments organizations I've worked for never use drop box. I've used iOmega cloud storage but that was 6+ years ago.

Dimitri


7,413 post(s)
#20-Oct-17 19:07

Sorry for disappearing on you... there's a huge new build coming out tomorrow and I've been reviewing some of the internal builds working up to that. I just haven't had the opportunity to pursue this as I would like.

Here's an idea: 3 GB is not a lot to FTP. Can you connect to an FTP site? If so, drop a note to tech support saying "hi, I can reproduce a crash using this .map file and the following step by step procedure [insert here the step by step procedure]... can I FTP the example file to you?"

They'll issue you an FTP account with instructions, so you could upload it to them. They really should be doing this anyway and not us trying to do it cart before the horse style here in the forum.

StanNWT
196 post(s)
#21-Nov-17 23:46

Hi Dimitri,

I finally sent in a request using a support token this afternoon. Hopefully they'll get back to me with a possible ftp upload not withstanding my IT group filtering out the reply. I was on annual leave plus had a great deal of work to do before going on leave, so my reply to this thread has been tardy, my apologies.

StanNWT
196 post(s)
#22-Nov-17 15:29

Hi Dimitri,

I received a reply from support, they said they'd investigate, but without the (.map) file, how can they test my observations / issue with decomposing to coordinates that particular vector layer? I'm only mentioning it here because of your request.

Dimitri


7,413 post(s)
#22-Nov-17 15:35

I cannot comment on communications you are having with tech support except to remark that this does not sound right... could it be you misunderstood?

General advice when launching a support incident:

1. Read and apply http://manifold.net/tech/contacting_tech.shtml

2. If something tech support instructs is unclear, write back to them and ask.

StanNWT
196 post(s)
#23-Nov-17 16:13

Hi Dimitri,

Sorry for the troublesome posting on this issue. Just an fyi, I've tried decomposing to points with future 9.0.163.9 and it still crashes when inserting records after roughly 330 million records and that's around 53 + GB of data according to the "running command" window. It takes around 65 minutes on my computer to run through scanning data portion of the command, and 65 minutes to do the insertion of the records but it crashes before finishing, I'm not sure how far along, and the log window is not accessible because of the crash window. Yes this should all be handled by support...

Dimitri


7,413 post(s)
#23-Nov-17 17:45

Yes this should all be handled by support..

and

I received a reply from support,

Sounds like you have an incident going with tech support, but were puzzled by their reply. Did you write back to them for clarification? Did they reply? What did they say?

I've never known tech support to fail when engaged in an incident. They sink their teeth into it and don't give up. Do what they tell you to do and if anything they tell you is unclear, ask, and they'll find the root of any issue.

Dimitri


7,413 post(s)
#24-Nov-17 09:14

Any word?

I cannot emphasize enough the forum is loose and casual, which is good, but that makes it the wrong place to solve/report things like this quickly.

Given you first posted on 17 October I'm a bit concerned that here it is 24 November with a thread still running that could have been handled by support on 18 October. :-)

StanNWT
196 post(s)
#24-Nov-17 16:46

I just sent off a reply to support. Hopefully they will provide an ftp link, but they're being difficult about some things, as I see it anyways. Whether it's a bug report or a support incident / token usage, I don't care, I just need to be able to send the file, so that someone can validate that it's happening, a developer. Without having the map file, or downloading the layer from the File GeoDatabase from NRCan, it can't be assessed.

Dimitri


7,413 post(s)
#25-Nov-17 15:24

Hopefully they will provide an ftp link, but they're being difficult about some things, as I see it anyways. Whether it's a bug report or a support incident / token usage, I don't care, I just need to be able to send the file, so that someone can validate that it's happening, a developer.

That seems to be taking the wrong approach. You've hired some experts to solve a problem for you, so focus on what they ask you to do, not on why you think they are taking the wrong approach. At some point you may be on the same page with them, but until that happy day, please don't argue with what the experts ask of you. This stuff is complicated enough as it is.

Once you launch a support incident, as you mentioned you did, what you should do is promptly answer any questions tech support asks and do what they tell you to do. You've spent a token to put experts to work for you so let them do their jobs. If they need to see a file, they'll tell you.

If they ask you any questions, answer them promptly. If they ask you to do something, do it. If they ask you to send a file, send it. But don't slow things down by making them ask you the same thing three times before they get the info they need to move to the next step.

In the "for what it's worth" department I've noticed tech support usually asks people a few sensible questions about what they are doing before they instruct them to upload a zillion gigabytes worth of files. I hope you agree that is a very good thing to do, since it avoids wasting time in case some missing info can solve a problem right away.

Anyway, if you feel they are "being difficult" because they are taking a different approach than you think they should, please don't worry about it and don't argue with them. Just answer their questions, do what they ask and let them solve the problem for you. Have faith that if it is a bug, they'll find it and fix it and if there are any errors in workflow involved they'll find those as well. Let them take the lead and all will be well.

StanNWT
196 post(s)
#27-Nov-17 16:29

Hi Dimitri,

Support confirmed the error, on the second machine they tested it on. They're going to try and put in a fix into a future build release and I'll test it.

I'm also going to work on testing it against the full Canada wide hydrological data set. That data set is much more complex and is larger than the topographic river network data set I tested it against.

on my machine at work that ran the decompose to coordinates that crashed, before I closed the dialog box for application crash window I noticed my physical memory usage was 95%, and I have 96 GB of RAM on my workstation.

I'll let you know when I've tested it against the new build after I've let support know. I appreciate your feedback and patience with this thread.

Dimitri


7,413 post(s)
#27-Nov-17 17:31

Support confirmed the error, on the second machine they tested it on. They're going to try and put in a fix into a future build release and I'll test it.

...

I'll let you know when I've tested it against the new build after I've let support know. I appreciate your feedback and patience with this thread.

No, in contrast we all appreciate your feedback and patience! It's really great you've spent the time on this and it's great to hear that thanks to your effort this will get fixed quickly. We all owe you thanks!

StanNWT
196 post(s)
#30-Nov-17 21:39

I just saw the posting of Future 9.0.163.11. I presume by his comment about tables created before, i.e. tables / drawings created in Manifold Future prior to .11, will reject past the 2 billion limit. In the case of this thread, I presume I will need to delete the river drawing and table, re-import from the source Esri GeoDatabase, or even better create a brand new map file, import from the Esri GeoDatabase and try the same operation and see if it now works?

KlausDE

6,410 post(s)
#30-Nov-17 22:31

First test a simple SELECT INTO in MF ..11


Do you really want to ruin economy only to save the planet?

KlausDE

6,410 post(s)
#01-Dec-17 07:46

Bad advice. See Dimitri's explanation here .


Do you really want to ruin economy only to save the planet?

adamw


10,447 post(s)
#01-Dec-17 07:10

Strictly speaking, you don't have to reimport anything from the GDB (reimporting won't hurt, but you don't have to do it). You can simply run Decompose to Coordinates like you were doing before (unsuccessfully), since it creates a new table. New tables created with 9.0.163.11 are not subject to the limit the operation was running into.

Thanks a lot for the report and example data.

StanNWT
196 post(s)
#01-Dec-17 17:29

Thank all of the developers and support staff for working on this.

I did start from a new empty (.map) file. Re-importing the river layer from the File GeoDatabase, to be sure no inherited issues would show up from the previous (.map) file.

Indeed I can confirm that Manifold Future 9.0.163.11 does not crash after running the "decompose to coordinates" transform.

However, the result is the following:

RAM usage is now at 94 GB, I have 96 GB installed. Manifold itself doesn't use more than ~4 GB of RAM during the times that I observed it running and during the save operation which I did this morning. RAM was not released until I closed Manifold after saving, however, in my task manager's process tab I couldn't find any application using all that extra RAM, it was likely cache that wasn't showing up in the processes.

It took 15801 seconds to run the "decompose to coordinates transform.

It took 2550 seconds to save the file.

File size before saving was 11.6 GB

File size after saving was 166 GB.

Number of records in the resulting and drawing objects as well as table for coordinates created: 50,000.

Odd that the file size went up 14.3 times when there so few records visible in the table.

I can upload screen grabs to illustrate my results.

In any case, I only have 50,000 records. The approximate number of records that were scanned and possibly attempted to be added into the table would have been 350 million. Something went wrong somewhere, or I exceeded the limits but it didn't crash, which is was Dimitri's comments refer to here.

StanNWT
196 post(s)
#01-Dec-17 17:57

Perhaps, if the number of records being created given the storage size of each record is such that there's a limit being encountered, I wonder if there's a way to break the resulting tables / drawing layers created into multiple output drawings and their associated tables, to avoid a limit, presuming one is being encountered?

How that limit is defined, I don't presume to suggest to the developers, other than to suggest maybe when passing 80% of the known limit of an operation's result is passed split the process into a subsequent data set. Or perhaps when scanning before inserting the records, calculate an estimate of the output size or complexity and if it violates even approximately a known limit then break the process / output drawing layers and tables into smaller subsets (tiles)?

Dimitri


7,413 post(s)
#01-Dec-17 19:27

In any case, I only have 50,000 records.

No, you have all of the records. The table is only showing you the first 50,000. See, for example, "Table Fill Strategies" in the Tables topic. To see how many records in the table, open the table and then in the Contents pane click on Component. That will report the number of records.

The only limit to tables within a .map project is the table has to have less than 2 billion records. It can have thousands of fields, etc., but the table must have less than 2 billion records if it is stored in the .map project. If you store the table in a data source such as Oracle or whatever, I don't know what the limit is for that, but it is far beyond 2 billion records.

There have been several threads on this, by the way...

StanNWT
196 post(s)
#01-Dec-17 20:51

Here's one of the screen grabs... The extent of the points (HD_1470009_1 Decompose to Coordaintes Drawing) created can clearly be shown as is the extent of the rivers layer. The rivers (HD_1470009_1 Drawing) show up as black on top of the Provinces, territories and US states areas, i.e. the AC_1M_BoundaryPolygons, but you can still see the Polygon boundary lines underneath. This is now a view prior to Manifold finishing rendering the scene. This is after it has finished rendering all layers.

Attachments:
Manifold_Future_9_0_163_11_Map_Window_Extent_All_Rivers_v2_Dec1_2017.jpg

tjhb
10,094 post(s)
#01-Dec-17 21:22

What does this show? Can you spell it out?

How does it relate to Dimitri's (obvious and correct) point?

StanNWT
196 post(s)
#01-Dec-17 21:45

If all the vertices from all the river vector polyline segments were converted to coordinate points then the entire map space of Canada should have points across the map space, and as you can see only a very small subset of all the polylines seem to have had their vertices converted to coordinate points. The estimated number of records created from the 15.3 million polyline segments was 350 million + according to the scan that happens when you run "decompose to coordinates" on this data set.

The point to what I was trying to do is create vertices for every polyline segment for the river layer of the entire 1:50,000 national vector basemap for Canada. Yes I know I'm pushing Manifold / Radian hard. That was the exercise in one way. I wanted to see what I can do with it with freely available data that anyone, especially Manifold support could acccess the same data if there was a need to verify an issue. There's no sense pushing it with proprietary data that I can't give out to support investigate the issue / bug.

I do appreciate support and the forum users providing feedback and attempting to address issues, as I'm sure the entire community does.

tjhb
10,094 post(s)
#01-Dec-17 23:31

Thanks. That is very helpful. Let’s try to reproduce it.

If what you’re seeing is reproducible then, obviously, it’s not OK.

StanNWT
196 post(s)
#01-Dec-17 23:51

I'm zipping the (.map) file now. When I'm back in the office I'll upload it to support. It will likely be a 16 GB (.7z) file. The (.map) file is 158 GB on disk now. I can send it if a comparison between your process and results vs. mine is warranted?

Dimitri


7,413 post(s)
#02-Dec-17 02:11

This is now a view prior to Manifold finishing rendering the scene. This is after it has finished rendering all layers.

I do not understand the above. You say it was prior to finishing rendering but also it was after it was finished rendering. Was it finished rendering or not finished rendering?

StanNWT
196 post(s)
#02-Dec-17 06:05

Sorry Dimitri. Predictive text on my phone messed it up. I'm was trying to say that it's not a view prior to finishing rendering but after all the points are rendered

adamw


10,447 post(s)
#02-Dec-17 09:59

The screen shows that the points have not finished rendering.

Your map shows three layers. These layers render in parallel. The chevrons on tabs at the bottom indicate that the topmost layer (the result of the Decompose to Coordinates transform, the tab displays a blue chevron) is still rendering while the other two layers (the original lines and the background layer of areas, the tabs display no chevrons) have finished rendering.

Zoom into one of the regions which does not currently show points. I think you will see the points come up. They will come up on the zoomed to fit view as well, just slower.

Also, check the number of records in the table in the Component pane like Dimitri suggests. Or do a simple SELECT Count(*) FROM [HD_... Decompose to Coordinates].

Dimitri


7,413 post(s)
#02-Dec-17 02:21

only a very small subset of all the polylines seem to have had their vertices converted to coordinate points.

Forgot to mention... You can't see all points using symbols that take, say, 10 x 10 pixels per little round/filledin circle because at the level of zoom a single point symbol occupies many miles and thus tens, if not hundreds, of thousands of other points. The symbols cover each other up.

Rather than waste time drawing points over and over each other, Future shows a representative sample (a bit like clipping overlapping lables) without attempting to put 100,000 symbols onto the same pixel.

So you can't count the symbols you see on screen and assume that's all there is. What you can do is see how they are distributed, so that if all Canada should be covered you can tell if it is or is not. What your screenshot appears to illustrate, if all points have finished rendering, that only the far west has been converted.

StanNWT
196 post(s)
#02-Dec-17 06:11

I understand the efficiency of not drawing every single point. However, if all the coordinate points are indeed there, would it not be better to render a distribution of points sampled throughout the spatial extent of all the records in the table? Meaning should not see points across the whole map? When I get in the office I'll confirm that song the file was successful as well as a more deep inspection throughout all geographic areas of the river network throughout Canada where coordinate points were to be created from the transform.

Dimitri


7,413 post(s)
#02-Dec-17 06:40

However, if all the coordinate points are indeed there, would it not be better to render a distribution of points sampled throughout the spatial extent of all the records in the table? Meaning should not see points across the whole map?

Not what I meant. If there are a zillion points dispersed across Canada it will show that.

Consider the analogy I made to how overlapping labels are clipped. If some labels are overlapping only those labels are clipped. It's not like if some labels are overlapping in, say, Quebec, that no labels at all would be shown in, say, Alberta.

If you start with a table that contains hydrology lines for all of Canada and you ask the Transform template to create a point at each coordinate that is a vertex of a line, then it should do that throughout all of Canada.

If you don't get a set of points throughout all Canada, I guess that would mean one or more of three things:

1. A bug.

2. You hit the limit of 2 billion records per table. This is easy to check: how many records are in the result table? If it is exactly 2 billion, well, then you maxed out.

3. The table fed to the template did not contain all hydrology lines for all of Canada.

An example of how 3. might happen: Publishers often provide data sets by province, instead of for the entire country. For example, suppose you are collecting roads from OpenStreetMap (OSM) for all of Europe. You might have to get those extractions one country at a time. So, OK, you download France, Germany, Italy and Spain as four different downloads.

You import all four into Manifold as four drawings, France, Germany, Italy and Spain, and show them all together in a map. They all line up perfectly, of course, and look like one big data set. And here comes the mistake... with the France layer tab the active layer you launch a Transform template to create points at the vertex coordinates of every road and then when you drop that layer of points into your map you're surprised to see the points only covering France.

Is Future only showing part of the points? Nope. It is showing all the points that were created. But because only roads in France (the active layer) were fed to the transform template only points in France were created. C'est la vie! :-)

You can't exclude factors like 3. without a careful look at workflow, but my bet in this case is on 1 or 2.

adamw


10,447 post(s)
#02-Dec-17 10:04

An important point:

If a particular operation hits the limit on the number of records in a table, it does so loudly: there is an error message and the operation is stopped after that error message.

If the transform completed without displaying any errors, it completed in full.

There are exceptions to everything, but that's the general case (and it applies here).

StanNWT
196 post(s)
#04-Dec-17 16:25

Hi Dimitri,

I came in this morning and checked the number of records in the component pane for the table of newly created points form the "decompose to coordinates transform", there are 318,119,966 records.

I zoomed into Nova Scotia, a place in the country that didn't have any points showing up last Friday when I tried to view the newly created points layer. The point layer displays points covering the areas where there should be points.

One thing I've observer is that in the windows task manager the CPU process percentage is between 0% and 2% even when rendering the layers of information wherever I am in the map. Is this because the GPU is handling this and little CPU activity is taking place? Traditional software would see at least 12% CPU usage. I see enough of a distribution of points across the map to know that there are points everywhere though.

Attachments:
Manifold_Future_9_0_163_11_Map_Window_Extent_Points_Dec4_2017.jpg

adamw


10,447 post(s)
#04-Dec-17 16:36

That's likely because the rendering thread for the layer with 300+ million points spends most of the time swapping between disk and memory, handling this number of records - that time gets assigned to the system, not to the application.

adamw


10,447 post(s)
#02-Dec-17 10:22

RAM usage is now at 94 GB, I have 96 GB installed. Manifold itself doesn't use more than ~4 GB of RAM during the times that I observed it running and during the save operation which I did this morning. RAM was not released until I closed Manifold after saving, however, in my task manager's process tab I couldn't find any application using all that extra RAM, it was likely cache that wasn't showing up in the processes.

A note on this:

This is normal. You still benefit from all your RAM. We do use way more space than 4 GB. Since you say that the saved file was 166 GB, we used at least that much. It is just that we use that memory through frequently accessed temporary files which the operating system can unload from memory at any point if it wants to and so it does not count that towards memory used by the process.

The 4 GB is the immediate cache. There are scenarios that can benefit from a larger size, so we will allow adjusting the size, but these scenarios are truly humongous (terabytes), the dependency is non-linear for most operations.

StanNWT
196 post(s)
#04-Dec-17 16:47

Hi Adam,

The file turns out to be 156 GB = 166,072,182 KB... The 7-zipped file is 15,775,611 KB, a nice compression ratio. I might try and see if Manifold Future will get that kind of compression.

The next thing I'll try is the entire National Hydro Network for Canada, which has the stream order information. I have yet to assemble the data into a single Esri File GeoDatabase, as it's all in 8,159 separate map sheet GDB tiles, likely at 1:50,000. The National Hydro Network (NHN) data can be found at teh GeoBase website. I can provide some of the relevant information from the metadata if you're interested?

adamw


10,447 post(s)
#05-Dec-17 06:38

I am interested, absolutely! Any numbers or experiences you want to share, please share.

Dimitri


7,413 post(s)
#05-Dec-17 07:50

This is a very interesting thread... some comments in no particular order:

windows task manager

Windows Task Manager is not a good tool to understand what complex software is doing, let alone complex parallel software. As you can see from this thread, Task Manager will often lead you astray, so don't bet on it.

I have yet to assemble the data into a single Esri File GeoDatabase, as it's all in 8,159 separate map sheet GDB tiles, likely at 1:50,000.

Well, that's interesting. Given the zillions of files in each GDB folder, 8000+ separate file geodatabases is probably aroundone million files. Heck of an architecture. :-)

I'd be very curious in hearing your reports how you go about assembling 8000+ separate file GDBs into one file GDB, how long that takes and what the performance is, say, of ArcGIS Pro with that one GDB. How long does it take ArcGIS Pro to open that one GDB, to pan and zoom in that GDB? How long does it take ArcGIS Pro to do the equivalent operation you'll ask Future to do?

ESRI writes that file GDB scales to 1 TB, so you should be able to assemble those 8000+ GDBs into one GDB.

On the plus side, once you have that one GDB you can turn it into a .map or .mxb in one operation.

I'd like to say "in one click" but it is really three clicks: a right click to call up a context menu, a second click to choose Export from that menu, and a third click on the Save button. Might be a fourth click required to choose .map or .mxb from the pull down list. :-)

I might try and see if Manifold Future will get that kind of compression.

MXB tends to beat zip compression in most cases of larger vector data sets. For smaller data sets, a zipped GDB could be slightly smaller than the equivalent MXB. See the comparisons at the ends of these two help topics:

Example: Convert an ESRI File Geodatabase into a .map Project

Example: Convert an ESRI Personal Geodatabase into a .map Project

For larger raster data sets, a completely unscientific impression based on a few weeks of random experiences is that MXB provides significantly better compression than zipping anything ESRI.

I put the focus on larger data sets, since, of course, nobody cares whether two different compression approaches are slightly better or worse than each other for small files.

StanNWT
196 post(s)
#05-Dec-17 18:10

Hi Dimitri,

Actually the maximum File GeoDatabase size is 256 TB, but the user has to select this option in the environment settings in ArcMap or ArcCatalog. This is even supported in ArcGIS Desktop and has been since possibly before ArcGIS 10.x. I will likely adapt and existing python script which "walks" through a directory and all sub directories and will import all feature classes in a File GeoDatabase into a destination one. There will likely need to be various "merge" or "append" commands to reduce the total number of feature classes in the destination GeoDatabase to just the total number of distinct feature classes that exist, not by map sheet but by theme.

Natural Resources Canada, uses Oracle SDE Enterprise GeoDatabases, and these individual File GeoDatabases are created simply to provide open public access to freely available map sheet tiles essentially. If a member of the public wants to merge any number of sheets together, they'll have to do that themselves.

I've attached HTML and XML copies of the metadata for a single 1:50,000 National Hydronetwork tile.

I just wish I could access raster (Arc binary grids) stored in File GeoDatabases in Manifold Future / Radian.

I have a 1 arc-second ~30 m DEM from the USGS NED data that has spatial extents of upper left (83N, 180W) and lower right (51N, 88W). The uncompressed DEM size is 146 GB, a single hill shaded relief layer with a specified vertical exaggeration is 35 GB. The File GeoDatabase has two variants of the DEM, one with and without pyramids, as does the hill shading for a specified vertical exaggeration. I was able to mosaic all this together using ArcGIS Desktop and the previously mentioned python code walking through 1294 directories of tiled TIFFs for 1 arc-second data and a second operation of 2 arc-second data in a directory tree of 503 directories of tiled TIFFs. The 2 arc-second data was mosaiced to 1 arc-second then the two mosaics, mosaiced together to get a single raster.

I use the the DEM with the 1:50,000 vector data, of which the river layer I was able to process thanks to the update to Manifold Future, is one layer of the many layers that comprise the Can Vec data set.

Attachments:
nhn_rhn_01aa000_1_0_fgdc_en.html
nhn_rhn_01aa000_1_0_fgdc_en.xml
USGS_NED_DEM_30m_Coverage_Oct3_2017.jpg

Dimitri


7,413 post(s)
#06-Dec-17 07:39

the maximum File GeoDatabase size is 256 TB,

I should have been clearer as I meant not the entire folder but a drawing. I assumed you intend to blend it all into a single dataset. As ESRI writes here regarding the limits on size...

One TB for each dataset. Each file geodatabase can hold many datasets. The 1 TB limit can be raised to 256 TB for extremely large image datasets.

Not being all that familiar with ESRI limits, I suppose that means any single vector data set, the equivalent to a Manifold table, is limited to 1 TB unless it contains images, in which case it can be made larger. Is that it?

This is a really fascinating discussion. If you don't mind, I cannot resist asking a bit more about how arcstuff works with these things and how that might relate to your use of Future...

Natural Resources Canada, uses Oracle SDE Enterprise GeoDatabases,

Why not just connect with Future directly to the Enterprise Geodatabase in Oracle?

mosaiced together to get a single raster.

That is really interesting. To make sure I understand, let me repeat back what I think is the case: you started with 146 GB worth of DEM that was distributed through 1294 directories, and following your process you created a single file that contained all that data mosaiced together into a single raster. Is that right?

If so, some questions: How big is that single raster file? What format is it in? How long did the entire process take in ArcGIS Desktop? From a cold start, launching ArcGIS Desktop, how long does it take to open that file in ArcGIS Desktop? How long to display it? How is the performance panning and zooming throughout the entire file? How long does it take to open that file in ArcGIS Pro? How long to display it in Pro? How is the performance panning and zooming in Pro?

You mentioned NED, which is a DEM, that is, terrain elevation raster data. That's not vector data like I assume is the National Hydro Network. Is that correct, that the National Hydro Network is vector data?

If so, how do you propose to assemble the 8000+ files of the NHN into a single data set?

Can you connect to the master Oracle data set using Future?

StanNWT
196 post(s)
#11-Dec-17 22:22

Hi Dimitri,

If the size of the polyline layer gets too big in ArcGIS, I can split it by province and territory. However, my goal is to push things as hard and far as I can, both in ArcGIS as well as Radian / Manifold Future. I'll use a python script to append the feature classes from each separate File GeoDatabase which are in separate directories per 1:50,000 map sheet tile, into a single feature class in a master GeoDatabase. Then if it's successful, import that into Manifold Future from the File GeoDatabase as a source, then copying and pasting it into Manifold's (.map) file. It might just crash because of the size of the data being copied and pasted, but I was successful with the 15.3 million record 15,000 Can Vec base map data for the rivers, so it should work?

adamw


10,447 post(s)
#12-Dec-17 06:09

As long as the number of records does not exceed 2 billion (2*10^9) and there is enough space in the temp folder, this should work. If the number of records exceeds that, the operation should fail with an error, but there should be no crashes.

adamw


10,447 post(s)
#06-Dec-17 09:23

I just wish I could access raster (Arc binary grids) stored in File GeoDatabases in Manifold Future / Radian.

We cannot access raster data in file geodatabases because ESRI's SDK does not provide means to access it (can only access vectors / tables).

If / when ESRI will add such means, we will support rasters in file geodatabases, no questions.

StanNWT
196 post(s)
#11-Dec-17 22:17

Hi Adam,

Safe Software's FME Desktop and Server licences have the ability to read and write to the File GeoDatabase both raster and vector. I've asked a support person at Safe to inquire about Manifold support.

adamw


10,447 post(s)
#12-Dec-17 06:29

As I understand it, they require one of the ESRI products to be installed which they are then using to access geodatabases. Accessing geodatabases using the ESRI SDK, like we do, is better in that there is no requirement to have any other products installed (that's what the SDK is for).

We might allow accessing raster data in geodatabases as long as the user already has an ESRI product that can do so, if this type of feature fails to appear in the SDK for long enough.

tjhb
10,094 post(s)
#12-Dec-17 08:55

Please don’t!

That would be a retrograde step for data openness.

Only support open formats (including ESRI’s).

And where possible, lobby for government agencies not to use closed formats.

Make MXB open as well, of course.

StanNWT
196 post(s)
#12-Dec-17 20:56

I can't believe I'm saying this but not to get off topic for the thread but if a group of users had raster days within Esri File GeoDatabases and wants to utilize them within Manifold going forward then it's a feature. Having to export every thing first to some other format then import or link into Manifold can consume vast amounts of disk space even in transition and of course time. Given the ubiquitous nature of Esri thrift the GIS industry having rater file geodatabase support is a feature. Stomping the open standards foot isn't going to engraciate oneself necessarily to the larger geospatial practitioner community. But that's my two cents worth. I've been talking Manifolds capabilities in 64-bit GIS and CUDA since 2008 and now do the same with Radian Studio and Manifold Future but the Esri fan boy base is massive.

volker

1,086 post(s)
#08-Feb-18 13:26

Are Raster Datas in an ESRI File GeoDataBase still not supported ?

I try to link a gdb-File Rasterdata in MFD9 (9.0.164.0) and all what i see are the

mfd_meta and the mfd_root tables...


http://www.thegisservicesector.de

Dimitri


7,413 post(s)
#08-Feb-18 14:13

Are Raster Datas in an ESRI File GeoDataBase still not supported

No, a well-known limitation of ESRI's File Geodatabase API is that it does not support rasters. You can see from ESRI's download page they are at version 1.4 of the API, still no raster support. I don't think ESRI is in any hurry to add raster support to their SDK.

I can see Tim's argument against leveraging an installed ESRI license, but I can also see Stan's side of it too.

I guess I look at it like this: if you have raster data in an ESRI geodatabase you almost certainly have an ESRI license. If you also have Manifold and you want operations to go faster, well, why not make it easy for such clients to do that? The decision where to keep their data was already made: it's stored in a GDB. The only question is whether it is useful to make it easy to get the data in some other format as well.

If you like the idea of keeping data in open formats, may as well take a step in that direction by making it easy to get raster data out of GDB with Manifold, where you have a really versatile tool for saving it into other formats, such as generic DBMS.

I don't see this as a top priority but perhaps if volker is unable to convince ESRI to get on task and add raster to their SDK (that is a weak attempt at humor...), or if ESRI for their own reasons decides to keep raster out of the SDK, or if there emerges a lot of use of GDB to store rasters, well, then maybe it will be time to allow people who do have an ESRI license to leverage that to extract rasters from GDB and into some more open format.

lionel

995 post(s)
#08-Feb-18 14:51

1) there is no way to automate esri to export raster ( inside gdb ) to raster on file system and then use manifold API to load them all in map file ??

2) what is the common language esri and manifold have both in common ?

python ?

---manifold support only active state

---arc support really well python and the real python system, github , jupyter ....

3) manifod 8 support mappoint and outlook . mfd9 should support powerBI or ESRI gui if many users ask for them !!

4) see https://www.youtube.com/watch?v=owrqMPWHjf8

1)+2)+3) = third gis software that use arcgis and python


Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

adamw


10,447 post(s)
#10-Feb-18 13:42

You can write a script in ArcGIS that would export data (for example, all rasters in a geodatabase, that's the reason for writing a script - to avoid exporting them one by one manually).

You can write a script in 9 to import that.

The two scripts do not have to be in the same language.

lionel

995 post(s)
#11-Feb-18 11:11

i was thinking esri and manifold API could be use call in the same script to avoid writing physical file on OS file system.


Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.