georeference.org
Subscribe to this thread
Home - General / All posts - Radian Studio works with ESRI geodatabases
artlembo


2,691 post(s)
online
#01-Mar-17 14:36

Since I come out of the traditional GIS world, I thought this was a cool feature of Radian: the ability to directly work with ESRI geodatabases. The video is only 4 minutes long, but you might want to show it to your GIS friends - it really opens up many possibilities. Check out the video here.

on another note, I don't mind if the forum administrators choose to remove this last request, but I've also tweeted this out @artlembo. This stuff spreads like wildfire if the right people retweet things, so if anyone is inclined, feel free to retweet it, putting whatever spin you want on it.

Dimitri


3,771 post(s)
#01-Mar-17 17:32

Art, no worries posting videos and suggestions to pass them on. Your videos are a great learning tool and a big help getting people started.


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

dchall8
266 post(s)
#01-Mar-17 17:48

Was this part of your big picture plan to take over the GIS world? Does this mean that ESRI will work 1,000x faster with bazillions of graphic processors? I have people I need to tell.

Okay I've watched the video now. I think I understand now that Radian is the only part of this that works 1,000x faster than anything else. Art demonstrated using a .gdb file inside Radian (which is an incredible and looong sought feature). But he was not using ESRI and Radian in an integrated sense to make the ESRI software faster. Is that correct?

artlembo


2,691 post(s)
online
#01-Mar-17 18:10

you are correct. In this case, we can imagine that ESRI is part of the stack. You might have a Department with SQLServer, one with Oracle, and one with PostgreSQL. In my original video, you can see how Radian can work among them seamlessly.

So now, imagine that you have ESRI inside of that stack. Here we had a geodatabase (a true file geodatabase, not a personal geodatabase that uses a .mdb file). You can then use Radian to read the data and perform processing on the file - using Radian.

Imagine you have a function you need to run over, and over, and over again (i.e. find all the different soil types that make up a parcel). In this case, you can use the SQL in Radian to dump that off. Your organization can still be an ESRI shop, making use of all the ArcGIS goodness you like. But, with Radian, you can also tap into that database.

And here is the cool part - like my first video, you can grab data out of a geodatabase and put it in Oracle, SQLServer, PostgreSQL, SQLite, etc. So, Radian becomes a nice bridge to all these databases, like a true ETL package. I think that there will likely be a large number of people who might just want Radian as an ETL tool.

I have a project that I am about to start up with a student where we are going to use Radian in this manner for a local municipality with a heterogeneous IT system. The goal is for Radian to grab stuff from everywhere and just get answers to questions that the municipality wants. It will be the first time they can tie their disparate infrastructure together. I'm looking forward to seeing how it goes, and I'll blog about it when we get started.

*p.s. in this case, Radian does not work 1000x faster than ArcGIS. I haven't done the timings. I will do some multicore stuff later today and report back. The main goal was to work with the architecture of the thing. I'll monkey around with some other processes later (both multicore and GPU) and see what happens. I don't anticipate HUGE differences for vector stuff, but with raster, who knows.

artlembo


2,691 post(s)
online
#01-Mar-17 18:46

one update: regardless of whether I run a query or transform from the data directly inside of the geodatabase, or copy the geodatabase directly into a Radian project, the time to perform the work is the same.

hugh109 post(s)
online
#01-Mar-17 19:22

though once in the Radian Manifold world it is so much more direct to do things, like figure what fields are what in census data and compute/map new fields. And for US census GDBs projection seems to work fine

Attachments:
FlBlockGroupsGDB.jpg

csb114 post(s)
#03-Mar-17 05:18

This would be great, but i do not have a GDB option in my datasource list

Any idea why?

using version 9.0.159.0

Attachments:
noGDB.JPG

tjhb
7,048 post(s)
#03-Mar-17 06:13

GDB support was added in Cutting Edge build 9.0.159.1, and updated in 9.0.159.2. You should use 9.0.159.2 to test GDB connectivity.

GDB support is not present in the current release build, 9.0.159.0. That's why you can't see it.

Cutting Edge builds are not available for installation, but as portable executable packages. In other words, you download the*.zip package, extract the folder it contains (e.g. manifold-9.0.158.2-x64), navigate to the \bin or \bin64 subfolder (for 32 or 64-bit executables respectively), then double-click manifold.exe to launch.

Bear in mind that the first publicly reported crash with Radian Studio beta or public builds only occurred after the ESRI GDB libraries had been linked in. Probably just coincidence!

ColinD


1,759 post(s)
#03-Mar-17 09:41

I connected to a GDB with no crash using 159.2


Aussie Nature Shots

adamw


6,606 post(s)
online
#03-Mar-17 10:25

Well, of course. We connect and work without crashes to GDB all the time. It is just that there was a report of a crash and maybe GDB was somehow involved - either our code or ESRI's - or maybe it wasn't. If and when we have a way to reproduce the crash or see it ourselves through a dump file, we will say for sure.

csb114 post(s)
#03-Mar-17 15:00

That is awesome.

Worked great.

Thanks.

tonyw
359 post(s)
#04-Mar-17 01:50

Thanks Tim,

I probably missed the instructions but I didn't realize cutting edge builds were meant to be run directly and couldn't be installed. When I couldn't install them I didn't pursue the new builds and missed out trying the enhanced features. I'm keen on using M8 to access data in an RS9 .map file plus getting into GDB files.

adamw


6,606 post(s)
online
#04-Mar-17 05:58

Be sure to read the instructions in the first post in the Cutting Edge thread for 9.0.159.x.

There is a bit of installing in order to test the ODBC driver from Manifold 8 - (a) you have to temporarily uninstall the official build of Radian and register the ODBC driver in the portable install with the system, and (b) you might want to switch to a working update build for Manifold 8 (that's optional).

We will remove the requirement to temporarily uninstall the official build of Radian in the future.

antoniocarlos

431 post(s)
#08-Mar-17 12:38

I have an older GDB file from Navtech that is not read by the current Radian version (.159.4). I dont think I can share it but is there a way to help Manifold decode it to be able to read it? QGIS 2.18.3 can read it.

ACGT


How soon?

artlembo


2,691 post(s)
online
#08-Mar-17 14:29

is it a .gdb, or a .mdb?

antoniocarlos

431 post(s)
#08-Mar-17 14:35

It is a .gdb. I think it is pre version 10. Global Mapper does not read it either but lets me know that I have to update it to a 10 version. I could send it to you it you want. I cant share it with Manifold because of their stated policies on their web site.

ACGT


How soon?

artlembo


2,691 post(s)
online
#08-Mar-17 15:28

I won't have time to look at it. Do you have ArcGIS where you could do the upgrade?

antoniocarlos

431 post(s)
#08-Mar-17 16:32

No worries.

I can access the data using QGIS. It is just that I would like to use Radian. It is an older version of the file geodatabase that is probably very rare to find in the wild. BTW Art I will contact you later in the week privately.

ACGT


How soon?

adamw


6,606 post(s)
online
#09-Mar-17 07:42

We will log the error message returned by ESRI's code if it fails to open the data set - that way we will at least know whether it's the ESRI SDK that can not open the data set (in which case this is kind of the end of it, we might try using an earlier version of the SDK in addition to the current version, but that has its own drawbacks) or if it's the Radian code (the best scenario, because then we can probably fix it).

I understand you can not send the exact data in question, but just in case - if you create an example data set using the ESRI tools you have, does it not exhibit the problem? If it does, perhaps you can send that.

adamw


6,606 post(s)
online
#09-Mar-17 14:02

A follow-up: we checked and indeed if the geodatabase is old (made by ArcGIS 9.2 or so), the ESRI SDK refuses to open it. We are going to report the failure in the log and fail for now. Perhaps QGIS uses an older version of the SDK and can open older geodatabases at the cost of not being able to do something else.

Dimitri


3,771 post(s)
#10-Mar-17 13:09

It is a .gdb. I think it is pre version 10. Global Mapper does not read it either but lets me know that I have to update it to a 10 version.

Two comments. The first is everybody might be barking up the wrong tree on old version of the SDK versus new version. The key question is...

Can you open any other .gdb?

The problem might not be old version / new version but just that you cannot open any .gdb because you did not install the 2013 C++ redistributable that ESRI's SDK requires. If you can open some .gdb geodatabases but not this particular one, then of course that is not the issue.

I cant share it with Manifold because of their stated policies on their web site.

What policy would that be? I've just re-read the Contacting Technical Support page and I don't see any Manifold policy that would prevent you from responding to a request from tech support to send a file. Tech support gets files from customers all the time, a routine part of the job.


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

antoniocarlos

431 post(s)
#10-Mar-17 14:48

Hi

I can open all the US Census GDBs that I have downloaded. I will look at others from the US Forest Service and the EPA soon.

You just gave me an idea. Perhaps some of these are 9.x old.

I am more than happy to send the file to tech. But isn't there a note somewhere a note that warns about sending over something is not in the public domain. In my case the data are provided by Navtech and has use restrictions (even though it is old and dated).

ACGT


How soon?

Dimitri


3,771 post(s)
#10-Mar-17 20:00

that warns about sending over something is not in the public domain.

I think if you mean if you just send an unsolicited letter out of the blue to sales or whatever.

In general, if you send a note to tech support, either to start a tech support incident or to make a bug report, you should not attach files since a) they get stripped out, and b) email is not reliable for big files anyway.

If tech asks for a file to work a support incident or to investigate a bug report they'll provide instructions for FTP, including encryption if confidentiality is required. Manifold gets all sorts of highly confidential things all of the time so it is a very routine thing to guard data against disclosure.

Like any reputable company Manifold can't accept files from people who do not have a right to provide them, but that usually is not an issue for things like support work for a legitimate licensee of a data set.

Quite often when data vendors license data they do so under licenses that allow the licensee to utilize contractors and vendors working on behalf of the licensee. The use of subcontractors, outsourcing, remote support and so on is very widespread so licenses are not normally designed to conflict with such common practice. It all depends on the particular license you have but I can't recall ever seeing license that would block your support contractor working to resolve a problem with your legitimately licensed use of a data set. There may be some, I just can't ever recall seeing something like that.

After all, the data vendors are in the business of selling data so they normally are very happy somebody besides them is spending money and effort to increase the market for their data. In addition, they usually are very happy to trade free samples of data and we send them some engineering licenses for their people to use. Once the right people are in contact with each other that sort of thing is routine. :-)


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

antoniocarlos

431 post(s)
#12-Mar-17 12:22

Thanks Dimitri. Got it.


How soon?

antoniocarlos

431 post(s)
#13-Mar-17 14:53

Hi all

Here is a link to a useful database for creating world maps at small scale. It is compressed as a zip file but contains the mdx and Qgis project files to read directly using ESRI and QGIS. Some of the datasets that are embedded within the zip file are file gdbs and can not be read with the latest version of Radian.

http://naciscdn.org/naturalearth/packages/Natural_Earth_quick_start.zip

I have already sent the report to Manifold with the "bug report" about the other dataset that can not be read by Radian mentioned above.

Cheers.


How soon?

Dimitri


3,771 post(s)
#13-Mar-17 15:22

and can not be read with the latest version of Radian.

Do you know if they are old or new? Can they be read by the latest version of ESRI ArcGIS?


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

adamw


6,606 post(s)
online
#13-Mar-17 15:56

Unfortunately, ESRI's SDK refuses to open the GDB in the archive with "This release of the GeoDatabase is either invalid or out of date." Looks like the GDB was created by an older version of ArcGIS.

antoniocarlos

431 post(s)
#13-Mar-17 18:01

About the file in the link above.

I do not have access to ARcGIS 10 right now so I cant say if it can be read by ESRI. I am trying to quit ESRI but they always pull me back in! :-) QGIS 2.18.3 has no problems reading all the files, and Global Mapper reads everything (including the color palettes) included in the zip file except the gdb files/directories. Global Mapper 18 give me the same error Adam reports regarding the gdb files in the directory.

ACGT


How soon?

jkelly


1,230 post(s)
#14-Mar-17 02:04

The Natural_Earth_quick_start_for_ArcMap.mxd opens up fine in ArcMap 10.4.1

I opened up an individual gdb as well (ne_10m_admin_1_states_provinces_geodb.gdb), it opened no dramas as well in 10.4.1


James Kelly

http://www.locationsolve.com

Dimitri


3,771 post(s)
#14-Mar-17 07:32

To add to Adam's comments...

ArcMap 10 reads both old and new versions of GDB most likely because it includes a copy of ESRI's original legacy code, as well as whatever new code they have created upon which (presumably) the SDK is based.

ESRI's SDK, by the way, is not thread safe. Using it from within a fully-parallel application like Radian involves some jumping through hoops, and also means that the SDK limits performance of gdb as compared to what it could be if the SDK were thread safe.

Given that ArcGIS is not parallel and that no other application besides Radian in GIS is parallel, I don't expect ESRI will put a priority on making the SDK thread safe. But perhaps if their user base asks for it that will happen.

It's clear from moves like last week's ESRI upgrade from requiring the 2013 C++ redistributable to the 2015 C++ redistributable (which Manifold incorporated, and which is why using GDB within Radian no longer requires installation of the 2013 redistributable in addition to the 2015 redistributable used by Radian) that ESRI is maintaining the SDK. It could be that sometime in the future it will become thread safe and thus allow increased performance of file geodatabases with parallel applications.

But as it stands, it's great that ESRI released the SDK and also made the terms of use reasonable. Kudos to ESRI.


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

adamw


6,606 post(s)
online
#14-Mar-17 06:47

QGIS reads file geodatabases through GDAL / OGR. GDAL has two drivers for geodatabases - a driver based on ESRI's SDK and a driver based on reverse-engineering. The first driver allows writes, can use spatial indexes and can read some things the latter driver can not, but it does not read geodatabases created by ArcGIS 9, only 10. The latter driver is read-only, builds its own spatial index as it reads data, and can not read everything the SDK can, but it can read geodatabases created by ArcGIS 9.

Our dataport uses ESRI's SDK, like the first driver in GDAL. We might try supporting geodatabases created by ArcGIS 9 if there's a lot of demand, but, frankly, whether we do this or not, the best thing to do with such data sets is convert them to a newer format.

Dimitri


3,771 post(s)
#15-Mar-17 16:54

For those interested in file geodatabases, in the Radian user manual the new examples in the Interoperability section of connecting through Radian with ODBC to a third party use ESRI GDB geodatabases. It's pretty cool that something like LibreOffice or Release 8 that has no ability to connect to GDB can use ODBC to reach that GDB through Radian.


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

tjhb
7,048 post(s)
#16-Mar-17 01:14

This batch of topics is so good. For me they explain a lot of things I only half knew (or less), filling in lots of holes in my knowledge simply and with no fuss. The step-by-step guidance is terrific, perfect.

Manifold User Community Use Agreement Copyright (C) 2007-2011 Manifold Software Limited. All rights reserved.