Subscribe to this thread
Home - General / All posts - GDAL 2.2 /vsicurl
joebocop
358 post(s)
#02-Feb-19 23:51

Two clients I have worked with this year are storing drone imagery in AWS S3 buckets in "cloud optimized geotiff" format (https://www.cogeo.org/). This is essentially a tiled GeoTiff with built-in overviews, and with the header data moved to the beginning of the file for HTTP GET-RANGE reading.

GDAL 2.2, usable with Release 9 (http://www.manifold.net/doc/mfd9/index.htm#gdal___ogr.htm), can read these files over http using vsicurl, and even directly from non-public S3 buckets using vsis3, without having to first download the entire file.

To read the metadata of a cloud-optimized geotiff using gdalinfo, for example:

gdalinfo /vsicurl/https://www.server.com/my_co_tiff.tif

A few other desktop GIS applications support these datasets:

  • QGIS 3.4.4 (via GDAL 2.4),
  • AcrGIS Desktop 10.6 (via the OptimzeRasters toolset's "Raster Proxies": https://github.com/Esri/OptimizeRasters/blob/master/Documentation/OptimizeRasters_UserDoc.pdf) and
  • CADCorp SIS Desktop 9 (via GDAL 2.3.1)
In Release 9.0.168.8, I create a gdal data source, or attempt to import a gdal file by specifying the URL to a cloud-optimized GeoTiff with /vsicurl/ pre-pended, but the data source just writes "Cannot open file" to the log pane.

Has anyone else seen success in loading a cloud-optimized geotiff from a GDAL data source over http in Release 9?

mdsumner


4,208 post(s)
#05-Feb-19 12:59

Try

gdal_translate /vsicurl/https://www.server.com/my_co_tiff.tif  vsi.vrt -of VRT

and then see if 9.0 will open the vsi.vrt - this is a virtual GDAL format, it just wraps the specific source into a local file.


https://github.com/mdsumner

joebocop
358 post(s)
#06-Feb-19 17:37

Oh man, classy, thank you for that tip!

That technique extends the power of those GDAL virtual file systems (vsizip, vsis3, vsicurl, etc) into Arc and MapInfo as well.

Thank you very much.

Attachments:
vrt_cogtiff.PNG

joebocop
358 post(s)
#07-Feb-19 00:05

I haven't been able to "link" (Create Datasource) successfully (using the Release 9 built-in File:TIFF data port) to a tiled TIFF file on disk.

The Data Source "expands" without error, and I can open the image component listed therein, but what is displayed is just an empty white background. Zoom to fit, or any attempt to style do not show me the data, so perhaps the Manifold TIFF dataport does not recognize tile tiffs, only non-tiled?

mdsumner


4,208 post(s)
#07-Feb-19 05:54

Try a subset so it's faster to stream down, ullr arg to gdal_translate. Got a link I could try?


https://github.com/mdsumner

joebocop
358 post(s)
#14-Feb-19 19:19

Ok, here's an example:

https://gte-rexcfnagudxgnuuhpqphnpuq.s3-us-west-2.amazonaws.com/cc_cogt_jpeg-ycbcr_mask.tif

Two interesting aspects of this tiff file are that

  • it was created from an "original" un-tiled RGBA tiff, using gdal 2.4, by running these two commands:

gdal_translate -mask 4 -b 1 -b 2 -b 3 -co PHOTOMETRIC=YCBCR -co TILED=YES -co COMPRESS=JPEG -co JPEG_QUALITY=50 --config GDAL_TIFF_INTERNAL_MASK YES cc_ortho.tif cc_cogt_jpeg-ycbcr_mask.tif

gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL -r average cc_cogt_jpeg-ycbcr_mask.tif

  • it is, therefore, a 3-channel YCBCR colourspace raster, having a 1-bit "mask band". It's not a 4th band, or an alpha band; it is a "mask", created in that first command from the original 4th band alpha channel (I understand this might be complicating matters, but the space savings are significant)

The Radiant.Earth marblecutter correctly detects the mask, as you can see the on-the-fly tiling preview here:

http://tiles.rdnt.io/preview?url=https%3A%2F%2Fgte-rexcfnagudxgnuuhpqphnpuq.s3-us-west-2.amazonaws.com%2Fcc_cogt_jpeg-ycbcr_mask.tif&rgb=1%2C2%2C3&nodata=&resample=average#15/64.8425/-139.8759

Thank you for having a look. Would be interested to know your experience with it.

Dimitri


5,249 post(s)
#15-Feb-19 11:40

so perhaps the Manifold TIFF dataport does not recognize tile tiffs, only non-tiled?

The Manifold TIFF dataport works fine with tiled TIFFs. Here are examples of the tiled test files from libTIFF, which all import perfectly:

cramps-tile.tif256x256 tiled version of cramps.tif (no compression)

quad-tile.tif512x384 tiled version of quad-lzw.tif (lzw)

zackthecat.tif234x213 8-bit YCbCr (OLD jpeg) tiled "ZackTheCat"

A quick way to see what a dataport can do is to find a known-good example of a given format (Google "examples of tiled TIFF" for instance) and try importing it. If it works, that's possibly something to cross of the list of what needs to be troubleshooted.

If you open a TIFF and see all white, launch the Style panel, select all three channels and apply medium Autocontrast and then remember to press the Update Style button.

Attachments:
cramps-tile.tif
quad-tile.tif
zackthecat.tif

joebocop
358 post(s)
#15-Feb-19 17:25

First, thanks for the autocontrast style tip.

Next, import of my example tiff, or any of your examples, works "as expected", no problems there.

I haven't been successful "linking" (old term, sorry) rather than importing, though. Here is the log window output when I expand the "linked" tiff datasource of my example tiff file:

2019-02-15 08:57:43 -- Create: (New Project) (0.005 sec)

2019-02-15 08:57:58 ** Unknown field with tag 42112 (0xa480) encountered

2019-02-15 08:57:58 ** Unknown field with tag 42112 (0xa480) encountered

2019-02-15 08:57:58 ** Unknown field with tag 42112 (0xa480) encountered

2019-02-15 08:57:58 *** Can not handle image with PhotometricInterpretation=4

Similarly, when I create a datasource from your "zackthecat.tif" file (which is also YCbCr) I get the same issue as with the YCbCr tiff file I shared, albeit without the log pane warning. Opening the datasource's image component shows an all-white screen.

If I copy the data source's image component and paste it into the project, the resulting image components behave as follows.

In the case of zackthecat.tif, with all three channels selected in the style pane, I can right-click autocontrast (med), and the value fields appear to be sensibly populated with float values ranging from -55 to 171. When I click "Update Style", however, the image component window remains all-white.

In the case of my example tiff, with all three channels selected in the style pane, I can right-click autocontrast (med), but none of the value fields' values change in any way. The range remains populated with 0-255. Clicking Update Style, of course, yields no change in the all-white display of the open image component window.

I have a bad feeling that I'm doing something fundamentally incorrect here, so please accept my apologies in advance if the solution is an RTFM'r.

Dimitri


5,249 post(s)
#15-Feb-19 17:37

I'm baffled what it could be. Here's what I did:

1. Clicked on the zackthecat.tif link in my prior post to download it from the forum.

2. Launched Manifold from bin64 (64-bit), using Build 9.0.168.8 (the current cutting edge build).

3. File - Create New Data Source.

4. Chose file: tiff as the Type. Uncheck save cached data between sessions. Navigate to the downloaded .tif and choose that. Press Create Data Source.

5. That creates the data source as you see it in the screenshot below. Open the image and it displays, no need to do anything.

Just for the heck of it, I did File - Link, navigated to the .tif and linked it. Here it is as a second "linked" (same thing) data source:

Attachments:
zack.png
zack2.png

joebocop
358 post(s)
#15-Feb-19 17:52

What the.

These instructions are so basic even I can follow them.

I had not tried File --> Link. My earlier attempts had all been of the Right-click --> Create --> New Datasource (file:tiff) variety.

If instead of that I do File --> Link, both my example image as well as zackthecat.tiff display correctly.

In the case of my example image, there is modest period of a dialog showing "copying" when I open the image component, but it does display. This note still displays in the log:

2019-02-15 09:48:06 *** Can not handle image with PhotometricInterpretation=4

Is there a difference between File-->Link and

CREATE DATASOURCE [Data Source] (

  PROPERTY 'Source' '{ "Source": "C:\\\\Users\\\\joe\\\\Desktop\\\\cc_cogt_jpeg-ycbcr_mask.tif", "SourceCacheExternal": false }',

  PROPERTY 'Type' 'tiff'

);

Attachments:
ztc_wtf.png

joebocop
358 post(s)
#15-Feb-19 20:21

Ok, so I had been caching no data like

Under those circumstances, I get a white-only component.

If I instead enable cache, like...

...then it "works", while still misunderstanding the "mask band".

Does that make sense, and does that behaviour equate to what is expected? The absence of cache will preclude the data source from displaying anything?

Thanks again, learning here.

Attachments:
ds_cache.png
ds_no_cache.png
ds_works.png

Dimitri


5,249 post(s)
#16-Feb-19 06:01

Yes, that's correct. As noted in the File - Link topic, "The File - Linkcommand is basically a simplified form of the File - Create - New Data Sourcecommand. " It checks Save Cache by default. Creating a data source explicitly saying not to use cache is not the same. Leaving the Save Cache box checked in the File - Link dialog, and leaving the Cache Data box checked when using Create Data Source, lets the system get around some of the limitations of read-only linked data. You can assign initial coordinate systems and the system can do (usually) some style manipulations as well that otherwise wouldn't be possible.

Try an experiment with the zackthecat.tif file: in Windows set the file to be read-only. Now, you can see what happens when you try to create a data source on it with the default option to Save Cache unchecked.

It can be trickier to do integration with the results of a GDAL stream because that's something of a black box, but the one thing you do know is that it is not bi-directional. It's a one-way, read-only thing.

A mild surprise is this

2019-02-15 09:48:06 *** Can not handle image with PhotometricInterpretation=4

That's a surprise on the GDAL front and on the Manifold front. GDAL apparently has different ways of expressing itself when it announces a four-band image. Import NAIP through GDAL and you get a separate RGB image and a mask image. Do it this way and you get a PhotometricInterpretation=4. OK.

Manifold should be tweaked to take that and consume it the way it does native NAIP fourband import, as an RGBA so that people can manipulate that fourth band as they like.

Dimitri


5,249 post(s)
#17-Feb-19 05:39

Forgot to mention: I sent in a suggestion to do as discussed in the last sentence above.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.