georeference.org
Subscribe to this thread
Home - General / All posts - Support for TopoJSON / GeoJSON?
jockeryl
155 post(s)
#22-Feb-17 03:24

I can't find a list of supported formats for Radian.

http://manifold.net/doc/radian/index.htm#

The index for R9 help looks incomplete compared to Manifold v8.

Do/will you support GeoJSON & TopoJSON import/export?

https://github.com/topojson/topojson-specification

Now I have to use OGR and QGIS for that step in Manifold 8.

No support there yet:

http://www.manifold.net/info/formats.shtml

mechalas

838 post(s)
#22-Feb-17 05:06

Radian supports GeoJSON. It can import JavaScript (*.js) files, and there are a number of SQL functions for working with the format, such as:

  • GeomJsonGeo - Takes a geom and an optional coordinate system (pass an empty string for no optional coordinate system) and returns GeoJSON with the geom.

  • StringJsonGeoCoordSystem - Takes GeoJSON and parses it into a coordinate system.

  • StringJsonGeoGeom - Takes GeoJSON and parses it into a geom.

mdsumner


4,151 post(s)
#22-Feb-17 05:40

http://mapstarter.com/

Dimitri


3,771 post(s)
#23-Feb-17 19:21

How does mapstarter convert GeoJSON for use in Manifold 8? The page noted above doesn't seem to do that.


Словом - иностранец. «Мастер и Маргарита» (М.Булгаков)

mdsumner


4,151 post(s)
#23-Feb-17 20:36

No such claim was made

tjhb
7,048 post(s)
#23-Feb-17 20:46

So what did you mean?

mdsumner


4,151 post(s)
#23-Feb-17 21:23

Nothing. Link for the OP, to use or ignore. Feel free to unload/unpack whatever's bothering you but probably there's a better place.

tjhb
7,048 post(s)
#23-Feb-17 21:31

Nothing bothering me, just didn't know what you meant.

mdsumner


4,151 post(s)
#24-Feb-17 08:05

Sorry, grumpy. I read too much into your bold emphasis there.

adamw


6,606 post(s)
online
#22-Feb-17 07:26

As other users said, Radian supports GeoJSON (and since TopoJSON is an extension of GeoJSON, you can read TopoJSON as well, Radian will ignore topology data, but read the coordinates).

Is it that you have a text file with GeoJSON / TopoJSON object and want it imported? We don't have the conversion to / from GeoJSON exposed as an import, it looked more like something you'd do in the middle of some other import (like the conversion to / from WKB - we don't have specialized import for "WKB", we are converting to / from WKB when handling other formats, ie, when reading or writing GeoPackage), so you currently have to copy / paste the contents of the text file into a table cell and perform the conversion using a query - but we may add an import as well, if it is desired. How are you getting the file, what is the process?

joebocop
258 post(s)
#23-Feb-17 17:55

Import of GeoJSON/TopoJSON would be useful for us as the web services we consume via REST endpoints all return objects in JSON format. A means of linking a Drawing from a parameterized URL that returns a JSON documents would be very handy indeed.

tjhb
7,048 post(s)
#23-Feb-17 22:25

the web services we consume via REST endpoints all return objects in JSON format

Would it help if this was already built in? (File > Create... > New Data Source... File: Web Server: arcgisrest. See attachment.)

Or am I barking up the wrong tree? I often do that.

Attachments:
ArcGIS REST.png

jockeryl
155 post(s)
#24-Feb-17 01:34

Thanks you for the confirmation about GeoJSON import, does it support export too? Do I understand that the API and UI Import/Export yet have to fully support the import/export?

SQL only feature now?

GeoJSON is now a IETF RFC: https://tools.ietf.org/html/rfc7946.

We are using it mainly as a part of a workflow where GeoJSON is both imported and exported.

We use Leaflet as a UI frontend and SQL Spatial 2016, Azure DocumentDB and MongoDB (AWS) as backend, all which can store and index JSON readily.

We also use OSRM for routing, which is producing millions of routes daily, OSRM only communicates via JSON REST services.

The whole Hadoop ecosystem also support JSON very well, we will grow into that moving forward when we get into millions/billions of individual JSON objects.

GeoJSON pretty much dominate the web mapping and have taken over from KML as the 'standard' interchange format as it's much more versatile and extensible.

The core benefit is that it can readily be processed by Javascript in the browser.

TopoJSON is not a direct extension of GeoJSON, if you drop the topology information, you have a hard time to restore the actual data. If I understand it correctly the TopoJSON actually translate the geographical coordinates to edge/arc references + local transformation and scaling. I don't think that if you treat is as GeoJSON the result will come out as intended, it requires quite alot of logic to reconstitute the original geographical shapes. https://github.com/topojson/topojson/wiki/Introduction

TopoJSON is to GeoJSON what E00 is to SHP.

It is also a very good data transfer format as it's around 4x smaller as each edge/node is only saved once and referenced by any polygon sharing the edge/node.

As Tjhb said, import/ingest a webservice would be a great feature where GeoJSON could be supported, for ESRI, Mapbox and OSM/QGIS/GDAL users like us. Making it easy for us to expose as a webservice would be excellent, maybe that's a feature for Manifold9 IMS...

Not sure if Radian support streamed import and export, that would be an excellent future capability, when I can use Radian as a processing engine in my workflows, to act like as CEP or ETL: Informatica/FME/SSIS/Flume/Spark is used today. The world of data is moving more to streaming data, with IOT and mobile devices driving the data collection. Most of our project data is from such sources, and by saving and importing/exporting from disk based files we lose agility and performance.

I would prioritize GeoJSON before TopoJSON as it's more widely used, but the benefits of TopoJSON is undeniable.

For now I use OGR/QGIS for all import/export of GeoJSON/TopoJSON files, but as I think they will be even more important as key formats in the Big Data world, I would hope Manifold add them both as first class import as well as export formats in UI and API.

tjhb
7,048 post(s)
#24-Feb-17 03:56

In case you didn't understand me, or didn't look at the image...

As Tjhb said, import/ingest a webservice would be a great feature where GeoJSON could be supported, for ESRI, Mapbox and OSM/QGIS/GDAL users like us. Making it easy for us to expose as a webservice would be excellent, maybe that's a feature for Manifold9 IMS...

That is already built in. Try it out.

adamw


6,606 post(s)
online
#24-Feb-17 07:46

Regarding TopoJSON vs GeoJSON - we looked into the format closer and yes, you are right indeed when you say that ignoring topology information won't get us far. TopoJSON not an extension to GeoJSON in that sense, sometimes you get new values in addition to those used by GeoJSON, but sometimes the new values (arcs) replace the GeoJSON ones (coordinates), the reader has to interpret the new values to generate geometry. We added a wishlist item to support TopoJSON.

Regarding GeoJSON - we support it already. We have functions to both parse and generate it, we both expose them for direct use and use them internally as necessary. However, GeoJSON is like WKB - it is a service format, not a protocol. You say:

A means of linking a Drawing from a parameterized URL that returns a JSON documents would be very handy indeed.

and my immediate question is - what is the protocol? Ie, there's ArcGIS REST API, they use JSON (not GeoJSON as far as I know) in places, and we support that, but we don't have a dataport for "JSON", we have a dataport for ArcGIS REST which has some conventions beyond JSON. Similarly, there's WFS, they encode geometry using GML, and we have a dataport for WFS and not for "geometry encoded as GML". Because just saying "here is an URL, it is going to return GML (or JSON)" is usually not enough, there are usually rules as to how to form that URL, there are some ways to retrieve metadata regarding the returned geometry, etc - a protocol.

Now, it is possible that you really just have plain URLs that return GeoJSON / TopoJSON, there is no metadata and this is not part of a bigger protocol with a spec (ie, because that's an internal line of business service), and you really just want us to fire away a web request, get the data, decode it and put it into a table + drawing. If this is the case, OK, we will put that into a wishlist, why not. But if what you have is part of a bigger protocol, we should support that protocol and not get into the middle of it trying to scavenge JSON out of requests made according to the rules we are oblivious to.

joebocop
258 post(s)
#24-Feb-17 08:09

Thanks adamw.

I understand that it makes no sense to assume that Radian would be able to make use of any old URL that returns JSON documents. We have been using Microsoft's Power BI to do analytics, and are interested to see how Radian can improve our processing. Power BI presents the json document to the user upon first connection and allows you to define the schema of your data objects.

Perhaps a means of defining a schema for these arbitrary JSON documents?

adamw


6,606 post(s)
online
#24-Feb-17 08:14

You said what platform this is coming from below (fulcrum). From the first look I think it makes sense to both add support for generic JSON / GeoJSON connections, as well as for fulcrum in particular (they have queries, why not allow linking to a fulcrum URL, then opening a command window on the resulting dataport and allow running queries - sending the text to the server and displaying the returned results - it's natural). We'll think about how best to do it.

joebocop
258 post(s)
#15-Mar-17 20:13

Following up, to have Fulcrum supported out-of-the-box by Radian would obviously be dreamy for me, but I understand it's not likely to be a high priority for many other Radian customers.

Should I be turning my desire for Fulcrum support into a "Suggestion", or are wheels already turning toward this end?

Thank you.

adamw


6,606 post(s)
online
#16-Mar-17 07:06

We already added that to the wishlist, but obviously if they add some new service you'd like supported (that is not there now and so we did not see it when composing the wishlist item) or if there's some specifics you want us to pay attention to, please submit a suggestion, it always helps!

joebocop
258 post(s)
#24-Feb-17 07:57

Thanks tjhb. The endpoint I am after is provided by a platform called fulcrum (fulcrumapp.com). They expose two API editions, one conforms to REST (http://developer.fulcrumapp.com/endpoints/records/), the other simply returns GeoJSON as text (http://developer.fulcrumapp.com/query-api/reference/).

Attempting to connect to the latter in Radian returns 403 Forbidden, when the same url pasted into a web browser returns the expected GeoJSON string.

Attempting to connect to the former is apparently successful (test connection result is "Connection Established"), but I get an empty datasource component in the project pane. Data returned is not GeoJSON but rather just JSON.

Likely whatever fulcrum is doing does not conform to the "arcgisrest" standard expected by Radian, or at least that's my presumption.

Looks like the right tree to be barking up, though. Thanks.

Manifold User Community Use Agreement Copyright (C) 2007-2011 Manifold Software Limited. All rights reserved.