Subscribe to this thread
Home - General / All posts - GeoJson dataport size limit increase
dprellwitz59 post(s)
#20-Jun-20 20:00

I've been struggling with bldg footprints correctly registering with satellite images to extract ground use info (concrete, bldg's, grass, shrubs, etc) and match that with parcel information for a small HOA in CA. Looks like the Microsoft Bldg Ft Prnt GeoJSON files in GITHUB are ideal; however, import into M9 is problematic for the California dataset due to file size (2.5Gb). I understand that a fix is due shortly for the portable M9; should i just wait or continue to "fight" with ARCGis people (i can't seem to get a login)?

BTW, all of you on Georef site and Manifold support staff are fabulous! have been for all of the years (12?) I've been using Manifold! Thank You all!

dprellwitz59 post(s)
#20-Jun-20 23:18

Attached is a screenshot of what i'm trying to "improve"... Google Sat base layer using "data-source-favorite"; Parcel outlines in Blue, bldg outlines in red - both from Koordinates website downloaded and imported at TAB files. Edited to remove unnecessary elements not associated with HOA.

Almost all of the satellite images, building footprints and street maps do NOT align with parcel maps. I'm hoping that using the Microsoft Building Footprint GeoJSON files with Bing satellite will give me the best and most accurate layers upon which to now extract front yard/street/hardscape information. I'll create (by hand, i suspect) each front yard's layout, how the existing irrigation system zones are implemented, where the city water is connected to each system, what type of irrigation is used at each zone, piping used, head-types used and controller/backflow/flowmeter element is located. But to do this, i need to have an accurate parcel map with imagery and building footprints... Am i barking up the wrong approach here? thanks for the help.

btw, i'm using Manifold 9 (with some Manifold 8 files, some editing), but i have both M8 and M9 loaded.

Attachments:
SCW-M9-screenshot.GIF

Dimitri

6,118 post(s)
#21-Jun-20 14:56

I'm hoping that using the Microsoft Building Footprint GeoJSON files with Bing satellite will give me the best and most accurate layers upon which to now extract front yard/street/hardscape information

I don't think that's going to work, mainly because the Microsoft building footprints data is absolutely awful. See the Example: Import GeoJSON File topic, which uses the Microsoft data set. The building footprints, for the most part, aren't even close to being accurate.

The Microsoft stuff is interesting from a machine learning perspective, but from a GIS perspective it's more about the terrible quality of what the algorithms produced. It's what keeps manual digitizing shops in business.

If you're dead set on using the Microsoft data set, I was going to suggest using Manifold's GDAL dataport to import the Microsoft stuff, but GDAL doesn't seem to handle the Microsoft data set, so that's out. Using Manifold's native GeoJSON dataport, one thing you could do to get around the 2 GB text limit is to open the California data set in a text editor and then save it as three parts, repeating the very beginnings and very ends to correctly open and close the data portions. It's all human readable text in JSON format, so you can do that. You can then import the three parts and merge them.

I think a better approach to getting building footprints is either a) use OpenStreetMap (which often has collected footprint data), or b) get data for the area of interest from whatever county it is in.

For imagery, popular imagerservers like Google and Bing have a lot of issues in terms of georeferencing and orthorectification. State and local government jurisdictions often have better data based on aerial photos or drone overflights. Some locations (not sure about in California) also have NAIP photography that is sub-meter accuracy.

Depending on budget, if it is just an HOA (home owner association), those tend not to be all that large, and it might be possible to get custom drone or aerial imagery shot and then mosaiced together for a reasonable price.

dprellwitz59 post(s)
#21-Jun-20 23:13

Dimitri:

as always, you're the best! I took the same approach - I found a local Sonoma County LiDAR project done in 2018 with tons of material. I was able to get a ARCgis REST link to their Ortho's (6") which i then used to overlay a Sonoma County Bldg Footprint geom which was close. I then used the Sonoma County's Parcel drawing to overlay the Ortho. I used M9's editing to move all the parcel layer's poly's to approximate ideal locations; altered bldg poly's to fit the Othro's picture to give me a good set of base drawings i now can use to do the rest of what i need (i hope).

However, maybe i'm kind of stupid, but, i have a Impervious-surfaces Geodatabase from the same LiDAR people that i can't for the life of me get to import, link, attach or become useful. here's the link to their description page: http://sonomaopenspace.maps.arcgis.com/home/item.html?id=72b8e24c47bf457083da9caf6de69801

I've downloaded the zip file to a "Import" folder, expanded the file, then tried to use import, then tried New data source, then link - none of which gives me anything in my M9 project folder to use... what's up? Obviously some form of operator error, but what???

{P.S., just saw your post on GDB files - i'll review that and try to grok where i failed...}

thanks!

/David

dprellwitz59 post(s)
#22-Jun-20 01:55

Dimitri, on the GDB load issue, i tried the approach that you and Adam had shown in the other post by Corentin: "import gdb file not working in M9 ?" I downloaded the gdb test files created by rk, extracted them to my M9-training drive/folder, and, using a fresh M9 load with and without new Map component, tried both methods of import/link - neither worked. Checked the Log files - no problems noted, but it did show the link request with 0 seconds to execute.

So i looked for the filegdbapi.dll - not found on C:\.

(I use a 550Gb SSD as my C:\ system drive, but load most programs onto a 750GB D: HD drive and share data on one of two 2TB HD drives).

I found the "filegdbapi.dll" files in D:, in the downloads section i use to start M9.

("D:\Daybee\Downloads\manifold-9.0.172.3-x64\manifold-9.0.172.3-x64\bin\filegdbapi.dll and in... \bin64...)

I thought that the portable install would setup the necessary system references for these? Looks like that might be the issue?

/David

adamw


9,299 post(s)
#22-Jun-20 08:42

It is puzzling to hear that importing a GDB doesn't seem to show any data in one more thread.

We did the same test as Dimitri above on a clean test system and everything worked flawlessly - there is a big drawing with many objects, etc. We understand though that you are getting no data.

We will try to find out what the issue might be, will perhaps add more reports to the log window.

One thing you can do to help is this: open Manifold, try linking the GDB using the steps in Dimitri's post (that's better than trying to import because the steps specifically spell out that the data source is going to be GDB, leaving less room for error), expand the data source, get the wrong result (no data inside), verify that the log window contains no complaints (make a screenshot), then open Task Manager, switch to the process list (Details), locate the MANIFOLD.EXE process, right-click it and select 'Create dump file'. Then move the dump file somewhere, contact tech support and tell them that you have the dump file to upload, they will provide you with FTP instructions to do so. From the dump file we will see what DLLs are loaded and will be able to verify their versions. If the versions are wrong - good, we found the source of the issue. If the versions are correct - also good, we can rule out configuration issues and should pay more attention to the code that gets invoked, maybe it contains a bug that only triggers sometimes, this does happen.

adamw


9,299 post(s)
#22-Jun-20 09:07

We might have an idea of what is wrong.

The machine might have an outdated version of the Visual C++ runtime libraries. You get the latest versions of these libraries when you install Manifold from an EXE / MSI, but when you install Manifold from a portable ZIP, you have to keep the runtime libraries up to date manually.

Visual C++ runtime libraries had a nasty issue in the last year where not only the compatibility was broken (the old versions were not good enough for modules expecting the newer ones), but the incompatibility did not result in a visible failure when the modules were being loaded and instead things were failing at runtime, in ways that made detecting and reporting the failure nearly impossible. This issue was seen previously, eg, see this thread (and this post). It is possible that you are stepping onto it as well.

Install the latest versions of the Visual C++ runtime libraries from:

https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads

If you are running 64-bit Windows, install both the 32-bit and 64-bit modules.

Then launch a new instance of Manifold and see if the GDB will now show the data.

Dimitri

6,118 post(s)
#22-Jun-20 08:30

Don't overthink this...

here's the link to their description page: http://sonomaopenspace.maps.arcgis.com/home/item.html?id=72b8e24c47bf457083da9caf6de69801

1. Downloaded and unzipped.

2. Launched 64bit 9.0.172.3 from the portable installation (the first download in the product downloads page). Did this exactly as recommended in the "To use a portable installation:" section of the Installations topic.

3. File - Create - New Data Source, of type gdb, navigated to where the gdb is in the unzipped folder, and double-clicked that. Did this just as shown in the Example: Connect to an ESRI GDB File Geodatabase topic.

Works perfectly.

There's no need to mess around looking for DLLS or any of that. It's so simple I'm puzzled what might be the problem you've encountered.

Some random guesses: not launching Manifold from the portable installation (see the topic), maybe have ESRI or Q installations that have a downrev or wrong copy of the ESRI gdb dll ahead of the Manifold path, so that's getting used instead of the right dll, not connecting to the gdb the way the topic says... ?

Here are my results, zoomed in to the Sonoma County Airport...

Attachments:
sonoma_gdb.jpg

dprellwitz59 post(s)
#22-Jun-20 19:28

Dimitri and Adam, that worked! Seems i had an old version of visual studio installed (2009, i think). Here's what i did:

1. I installed the newest version of Visual Studio (2019-free). Reboot.

2. Downloaded a fresh copy of M9 (172.3). Started 64bit M9 from portable folder.

3. Started new project.

4. File - create new data source - file type-gdb; navigate to impervious surface folder; open.

5. Double click image file; here's the result for the area i'm interested in:

Thank you!

You guys are the best I've worked with in years! I don't say that lightly as i was a developer from the 1960's through the 1980's and a IT manager for the last 40 years. Your patience and passion are tops.

And, NO ONE can say "RTFM" and have you diving for manuals like Dimitri!

Thank You!

/David

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.