Subscribe to this thread
Home - General / All posts - Mosaicing large amounts of raster data to GeoPackage
Matsamentet1 post(s)
#18-Mar-19 14:28

Hi!

I do a lot of work creating GeoPackages from large amounts of raster maps and imagery. The source data can be several thousand files, with a total size of up to several hundred gigabytes. The format is usually GeoTIFF.

To create these GeoPackage mosaics I have been using GDAL. However, as demands for higher resolution data are rising, I find that GDAL's single core performance is no longer enough. I discovered that Manifold makes great claims to multiprocessing, so my question is, can Manifold build raster GeoPackages and fully leverage multi-core processing?

To compare to another software, the functionality I need is similar to that of GeoCompressor from Hexagon, with the difference that I need to create GeoPackages instead of ECW.

tjhb

8,657 post(s)
#18-Mar-19 14:45

If I understand the task, it is not a case where multiple cores will help significantly, since the issue is not heavy calculation (unless you are also reprojecting) but data transport.

Besides using multiple cores (wherever it is helpful), Manifold 9 absolutely excels at data transport.

And it can read and write both geometry and images to/from GPKG format, yes.

The general method (not tested right now) is: Link all of your source images (no need to use Import), add them to a Map, use Edit > Merge, Export the result to GPKG.

(There is a further clever optimization, but it may currently have a glitch.)

You can test everything except the export using the free Viewer.

Dimitri


5,359 post(s)
#20-Mar-19 04:50

the functionality I need is similar to that of GeoCompressor from Hexagon, with the difference that I need to create GeoPackages instead of ECW.

It might be that you are not so much running into the limitations of GDAL as you are running into the differences between ECW and GPKG.

ECW is a very different technology than GPKG. Part of the objective of ECW is allowing extraction of subsets of imagery (zoomed in views, zoomed out views) for viewing in a very fast way, albeit at the cost of (usually) lossy compression and slow performance for writing/analytics. GPKG, in contrast, is more of a general purpose DBMS format, albeit a slower database than, say, PostgreSQL.

Saving to GPKG, therefore, will not get you the same functionality you get with ECW. It will be different functionality, with a different balance of advantages and disadvantages.

The only format I know that combines the benefits of both, fast access and viewing with larger data, plus fast DBMS for general purpose, parallel, analytics and writing, is Manifold's native .map project format. Save the data to .map with Release 9 and then anyone can use the free Viewer to view it.

Can Manifold very rapidly merge several hundred gigabytes of different images into a single, seamless image? Sure. Can Manifold save that into GPKG? Sure. But Manifold cannot give GPKG the multi-resolution/wavelet tricks of ECW, nor can Manifold change GPKG from being a slower database technology into a faster one like PostgreSQL, or even faster, for GIS purposes, like Manifold .map.

Manifold, by the way, can also save to ECW for free up to the limits of the free ECW code built into Manifold. If you have a Hexagon license, Manifold can save to unlimited size ECW.

Dimitri


5,359 post(s)
#20-Mar-19 09:16

If you have a Hexagon license, Manifold can save to unlimited size ECW.

Whoops... my mistake. I should have read the ECW topic. :-)

Manifold uses the ER Mapper SDK to write Version 2 ECW (which all recent ECW SDKs from Hexagon can read) and the Hexagon SDK to read all ECWs (providing read capability for version 2 and version 3). But, Manifold won't automatically switch to using a licensed Hexagon SDK to write version 3.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.