Subscribe to this thread
Home - General / All posts - .MBtiles to ??? for downstream ESRI users
cycloxslug57 post(s)
#16-Oct-20 03:33

I am collecting sonar data in the field and using a program called ReefMaster to interpret the sidescan sonar data files into georectified image mosaics. It can export them as .MBtiles or, PNG with associated kml calibration file, or as a .kml super-overlay. MBtiles seem convenient and portable, and I have successfully imported them to Manifold 9. Reefmaster does not specify the associated projection, but it seems fine with the default WGS84/Pseudo-Mercator.

However, when MBtiles are first loaded into Manifold 9, they generate an error (little red stop sign on the map tab, message in the message console) - "Image cannot be rendered because it stores no data for intermediate levels". Reprojecting the images (into EPSG:2835, though I don't suppose it matters?) seems to clear this error on the newly created image.

Two questions:

1) Is there a better workflow for clearing the errors on the MBtiles?

2) I need to provide these files to colleagues and students who work exclusively in ArcGIS (which I do not have). It does not appear that ArcGIS can import MBtiles. So what file type would be best (assuming TIF?) and what would be the appropriate workflow in Manifold 9 to ensure the images have the appropriate projection information?

tjhb

9,452 post(s)
#16-Oct-20 04:02

(1) See the TileUpdatePyramids function.

(2) Prompt the colleagues and students to add Manifold to their toolset. Ignore them if they say no.

cycloxslug57 post(s)
#16-Oct-20 04:40

definitely not sure how to do 1:

running TileUpdatePyramids([Center-east Image])

in the Command window returns "Cannot parse query", which I assume means I've got the syntax for calling a function incorrect, but I am having a hard time understanding the documentation, and experience with MFD8 is no help at all.

2) Not an option... the people who currently need these files are not GIS people (and honestly, I'm not either...just enough to do some things now and then and view spatial data)

tjhb

9,452 post(s)
#16-Oct-20 04:53

Re 1, do a lazy forum search on TileUpdatePyramids to see how to use it.

Re 2, you immediately see the problem. If users are incompetent in GIS generally, what makes them safe or worthwhile users of ArcGIS?

Point 2 is an important question. GIS is not brain surgery, but it’s not high-school maths either. Training, study and practice are required.

tjhb

9,452 post(s)
#16-Oct-20 05:37

Re 1, do a lazy forum search on TileUpdatePyramids to see how to use it.

It's not quite that simple, because syntax has changed over time.

The current correct syntax is

--SQL9

TABLE CALL TileUpdatePyramids([Image]);

Where [Image] is the image name.

(There are also the TileUpdateFieldPyramids and TileUpdatePyramidsAll functions, recent additions which complicate things though only slightly.)

Point 2 sticks though. Learning is required. If the people who "need these files" are not at least learning GIS, you could suggest that they pay a specialist.

Dimitri


6,233 post(s)
#16-Oct-20 08:15

MBtiles seem convenient and portable, and I have successfully imported them to Manifold 9.

How did you import MBtiles into 9? That's not a format directly supported by Manifold. If you import .png, there's no need to build intermediate levels, so that might be a result of whatever process was used to import mbtiles. For example, GDAL might not build intermediate levels.

However, when MBtiles are first loaded into Manifold 9, they generate an error (little red stop sign on the map tab, message in the message console) - "Image cannot be rendered because it stores no data for intermediate levels"

That's not an error in the data, that's an advisory message the image import is not yet complete, in that the image is not ready for use. Native Manifold dataports do all that for you, but the presumption is if you use a third party import (GDAL) you don't want Manifold in any way altering what that third party you called upon does for you.

1) Is there a better workflow for clearing the errors on the MBtiles?

Yes, there is. The "Messages" part of the User Interface topic puts it better than I can, so I'll just copy it:

Messages

If we double-click open a component and a red message icon appears in the window's lower tab, we should choose View - Messages from the main menu to read the message. Blank drawings showing a message icon usually indicate lack of a spatial index, which Manifold will offer to build for us. Blank images showing a message icon usually indicate a need to build or update an index or intermediate levels, which Manifold will offer to do for us.

F7 is a keyboard shortcut for View - Messages (as the View menu tells us).

Once you build intermediate levels, you never have to re-do it, as then the image is fully provisioned and ready to go. Reprojecting the image will, as a side effect of reprojection, also build intermediate levels. But it's not the right way if all you want to do is provide intermediate levels. Much better to simply click View - Messages and then click the button to have Manifold build intermediate levels

Given that not everyone reads the Basics topics, and also given that for those who do it's easy to skip over various details, I think it would be good if one could click on the icon and see the message without needing to call up View - Messages. I get it that doing that complicates with extra code where the icon can go (interfering with normal click commands on tabs, for example) but it would be nice to have. I've sent in a suggestion to that effect.

The other way to have better workflow, of course, is to import as .png. That's way more portable than MBtiles.

In terms of sharing, share projected images using TIFF format, which Manifold writes as GeoTIFF. GeoTIFF embeds projection info, and I believe that Arc can read GeoTIFF, perhaps even without requiring the user to spend a few thousand on an extension product. :-)

As to how to ensure your images have the correct projection, that depends on whatever information is provided from the original source of the images. If it provides enough information to display the image with correct georeferencing in Google Earth, no doubt there's a way of extracting that information in a way that it can be used in GIS.

If you're lucky, it could be you could use the kml overlay file provided, as shown in the Example: Import KML with GroundOverlay Image example topic.

Dimitri


6,233 post(s)
#16-Oct-20 08:33

Ah.. forgot to ask. What format, exactly, are those mbtiles delivered in? If it is genuinely mbtiles, it should be in an sqlite file dbms. What happens when you import them using the sqlite dataport in Manifold? In that case, it should capture full projection info (which is required to be Spherical Mercator).

Edited: Just checked... won't help. It reads raster tilesets in as individual varbinary records.

joebocop
426 post(s)
#16-Oct-20 17:39

I work with mbtiles in Manifold. Perhaps this workflow will help.

First, mbtiles are normally in EPSG:3857. Do you know if yours are? I find GDAL to be extremely useful in these scenarios, where the output of 'gdalinfo your_mbtile_file.mbtiles' will show you everything you need to know about what ReefMaster has created for you.

The easiest way to view mbtiles in Manifold is to File --> Link to a file type ". Files (GDAL/OGR) (*.*)", unchecking "Save Cache". This is super slick, so long as you've got GDAL configured correctly to work with Release 9. The datasource component is created instantly, and you can pop it open immediately, with no need to build pyramids or await any building of anything else.

Manifold then needs to be "told" (Assign Initial Coordinate System) which coordinate system the linked component is using. ReefMaster ought to let you know what that is, otherwise the gdalinfo output on that mbtile will be your best source of truth.

With the component's coordinate system confirmed in Manifold, if you need to export to TIFF or any other Release 9 supported export format, it's as simple as right-click on the image component and Export. If you need to work with the data within Manifold, just copy and paste the linked gdal datasource's compoennts into your Manifold project and you'll be set to work with the local components.

One final consideration, any exported tif is going to be bigger than the compressed mbtiles, and there are many different flavours of TIF that your recipients may/may not be able to handle. TIFF exported by Release 9 is usable in ArcMap and ArcGIS Pro, in my experience.

That is 100% of what I know about working with mbtiles in Manifold 9.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.