Subscribe to this thread
Home - General / All posts - New video: Rendering Shootout - Manifold / PostgreSQL / Q
Dimitri


6,511 post(s)
#24-Feb-21 06:23

There's a new video posted on the videos page: Rendering Shootout - Manifold / PostgreSQL / Q

This shootout compares Manifold running both standalone and as a PostgreSQL client to Q running as a client to an enterprise-class PostgreSQL database server. The PostgreSQL server is on the same machine, so there are no network delays.

The video was prompted by requests to compare Manifold running with storage in a .map file to Q running with storage in an PostgreSQL/PostGIS database server. That's obviously not an apples-to-apples comparison, but it's interesting to try, to see what the different rendering speeds are like.

dchall8
847 post(s)
#24-Feb-21 19:09

It would be nice in your videos if you could link to the place where you got the data used in the video. If not in the YouTube discussion notes, maybe you could put links here in your announcements.

Dimitri


6,511 post(s)
#25-Feb-21 05:09

It's the same data set that was used in the previous rendering shootout video. There's a link to that .map file in the prior announcement. To replicate this video you have to load up a PostgreSQL/PostGIS server with the one, big, 22 million roads layer and also with the 49 separate states layers that all together total to the same 22 million roads.

dchall8
847 post(s)
#25-Feb-21 22:39

At 0:40 seconds into this video you mentioned that the roads layer was from the "so called GDT Dynamap 1000 database." I searched for that database and found mostly university libraries with the data for only one state. If there a one place to find the data for the 50 states? Certainly a .map file is nice, but sometimes you find other valuable information when you are at the original source.

hphillips31 post(s)
#26-Feb-21 00:38

I believe the GDT detailed roads data for the individual states were made available on data set CDs called 'ESRI Data and Maps' that came with ArcView or ArcGIS in the period around the year 2000. These covered Canada, Europe, Mexico, US, and the World in about 120 themes. According to the licensing those data were not freely redistributable; that is probably why you can't find them to download. If the roads are in an existing .map file then you are fortunate.

The old ESRI Data and Maps on CD has apparently morphed into the current online version, ArcGIS Data and Maps https://www.arcgis.com/home/group.html?id=24838c2d95e14dd18c25e9bad55a7f82#overview - it looks like that data is free to download but those roads data available there may not be equivalent to the GDT data in the .map file.

Dimitri


6,511 post(s)
#26-Feb-21 07:55

The easiest way to get the data used is to download the .map project. Anybody who wants to create shapefiles, or GDB, or GPKG can easily do that by exporting whatever they want rom the .map project. If they don't have Manifold, well, then they can find the original data and hassle with that.

I searched for that database and found mostly university libraries with the data for only one state.

GDT put Dynamap 1000 for 1999 into the public domain, which is why Manifold uses it for examples. It's still out there on the web, but as with just about all data of this vintage, the original was published as individual state files.

Download those and merge them if you want all the lower 48 + DC in one file. It's handy to have them as different states, by the way, because then you can import 49 of them at a time to get the 49 "by state" layers used in the example.

sometimes you find other valuable information when you are at the original source.

There's nothing special about GDT's roads. It's just a dataset of over 22 million roads for CONUS that's a handy data set to use to compare rendering performance using a subject (roads) that people intuitively understand. It's not going to change the performance comparisons using a different set of roads, or roads for 2010 instead of 1999. Manifold will still be way faster than Arc or Q.

If anybody wants to do do other types of analytics, say, using attributes, there are many roads data sets besides GDT that are available, such as the roads extractions from OpenStreetMap at osmdata.xyz. Those are big, all in one files, so you have to do some work to extract by state if you want to compare an all in one layer with the same data as many layers. OSM extractions from geofabrik.de are also great data sets for comparisons. For the US, you can also use roads from the US Census Bureau's TIGER data sets - those are completely in the public domain.

lionel

700 post(s)
#24-Feb-21 20:45

1) Do we must understand that data store in manifold storage for example filename.map ( with drawing that have the prefix native in video ) is more efficient (in rendering time) using manifold itself that use data store in external *.map file like postgresql Server ?

So data store inside manifold filename.map is more efficient than Qgis that don' t implement their own driver/file-system ( fileproject extension with File OS system ) OR manifold when they use both the same Microsoft PostgresQL Driver .

2) I think when software use ( have to use ..mandatory ) third component (dll : here postgresql driver ) , then there is no difference beetween software that use the same components when focus on rendering here ( explicit in the vidéo) . it is a theory/logic statment for me here but not test in reality .... to know if this really describe the reality !

3) what ll be interesting is to know which implementation of "embedding driver" ( my way to explain this) is the more efficient beetween Manifold and Arcgis /Mapinfo about rendering Speed if Arcgis/mapInfo project file is able to store data we call drawing in manifold !

regard's

I really like the gang of four ( not about mao but design pattern) but it is about compute data not rendering data ( move coordinate text to Screen System coordinate ...direct X / graphic card ?) !!


union

Dimitri


6,511 post(s)
#25-Feb-21 05:18

1) Do we must understand that data store in manifold storage for example filename.map ( with drawing that have the prefix native in video ) is more efficient (in rendering time) using manifold itself that use data store in external *.map file like postgresql Server ?

Yes. For rendering displays that must reckon many objects, with only 22 to 44 million of them Manifold .map is a faster data store than a PostgreSQL database server, all other things (running on the same machine, etc.) being equal.

If you get up to really huge numbers of objects, then I'd expect PostgreSQL would at some point be faster. That's mainly a function of how Manifold is currently tuned in terms of optimizing data access. It's smarter today to tune for best performance in the one to tens of GB range, given that's what almost everybody is actually working with, instead of 100's of GB.

As data sizes continue to increase, at some point Manifold will invest a couple of months into the Map 2.0 initiative. Between that and implementing distributed data store in Map 2.0 (which PostgreSQL can do today), it would be reasonable to expect that Manifold's speed advantage with native store would still be there, with quite likely a visible edge over PostgreSQL even with terabyte data.

So data store inside manifold filename.map is more efficient than Qgis that don' t implement their own driver/file-system ( fileproject extension with File OS system ) OR manifold when they use both the same Microsoft PostgresQL Driver .

Yes. Manifold using .map is much faster than QGIS running any native store, or QGIS running PostgreSQL, or Manifold running PostgreSQL.

2) I think when software use ( have to use ..mandatory ) third component (dll : here postgresql driver ) , then there is no difference beetween software that use the same components when focus on rendering here ( explicit in the vidéo) .

No, that is incorrect for two reasons. First, there are efficient and inefficient ways of using the same PostgreSQL dll to query a PostgreSQL database. Second, if there are bottlenecks within the client application an inefficient client will not be able to use the data PostgreSQL provides, to do things like generating a display that will fit on screen from 44 million objects, as fast as an efficient client that has few or no bottlenecks.

QGIS isn't a parallel application, so it has many bottlenecks that Manifold does not have, and it also may not have the repertoire of methods Manifold can bring to bear on the most efficient possible use of the PostgreSQL drivers. The combination of the two is probably why Manifold usually can pull and utilize data out of PostgreSQL about twice as fast (or even faster) than Q.

You see less of that effect with small data, or when zoomed in so that only relatively few objects need to be pulled from PostgreSQL. Q also does some tricks that are cheating, for example, it caches pre-rendered displays so if you go back to that it shows them instantly. But that's cheating with a multiuser DBMS because Q doesn't know if data has changed as a result of some other user or other process changing what is in the database. It could be showing objects, many of them, that aren't there anymore. It's one of those classic "not quite right but looks OK" things that you see in Q.

But even with all that, and even considering QGIS was originally created to display data from PostgreSQL, Manifold is still visibly faster at displaying data from PostgreSQL.

lionel

700 post(s)
#25-Feb-21 23:49

thank's for the informations


union

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.