Subscribe to this thread
Home - General / All posts - GDAL 2.2 /vsicurl
joebocop
384 post(s)
#02-Feb-19 23:51

Two clients I have worked with this year are storing drone imagery in AWS S3 buckets in "cloud optimized geotiff" format (https://www.cogeo.org/). This is essentially a tiled GeoTiff with built-in overviews, and with the header data moved to the beginning of the file for HTTP GET-RANGE reading.

GDAL 2.2, usable with Release 9 (http://www.manifold.net/doc/mfd9/index.htm#gdal___ogr.htm), can read these files over http using vsicurl, and even directly from non-public S3 buckets using vsis3, without having to first download the entire file.

To read the metadata of a cloud-optimized geotiff using gdalinfo, for example:

gdalinfo /vsicurl/https://www.server.com/my_co_tiff.tif

A few other desktop GIS applications support these datasets:

  • QGIS 3.4.4 (via GDAL 2.4),
  • AcrGIS Desktop 10.6 (via the OptimzeRasters toolset's "Raster Proxies": https://github.com/Esri/OptimizeRasters/blob/master/Documentation/OptimizeRasters_UserDoc.pdf) and
  • CADCorp SIS Desktop 9 (via GDAL 2.3.1)
In Release 9.0.168.8, I create a gdal data source, or attempt to import a gdal file by specifying the URL to a cloud-optimized GeoTiff with /vsicurl/ pre-pended, but the data source just writes "Cannot open file" to the log pane.

Has anyone else seen success in loading a cloud-optimized geotiff from a GDAL data source over http in Release 9?

mdsumner


4,212 post(s)
#05-Feb-19 12:59

Try

gdal_translate /vsicurl/https://www.server.com/my_co_tiff.tif  vsi.vrt -of VRT

and then see if 9.0 will open the vsi.vrt - this is a virtual GDAL format, it just wraps the specific source into a local file.


https://github.com/mdsumner

joebocop
384 post(s)
#06-Feb-19 17:37

Oh man, classy, thank you for that tip!

That technique extends the power of those GDAL virtual file systems (vsizip, vsis3, vsicurl, etc) into Arc and MapInfo as well.

Thank you very much.

Attachments:
vrt_cogtiff.PNG

joebocop
384 post(s)
#07-Feb-19 00:05

I haven't been able to "link" (Create Datasource) successfully (using the Release 9 built-in File:TIFF data port) to a tiled TIFF file on disk.

The Data Source "expands" without error, and I can open the image component listed therein, but what is displayed is just an empty white background. Zoom to fit, or any attempt to style do not show me the data, so perhaps the Manifold TIFF dataport does not recognize tile tiffs, only non-tiled?

mdsumner


4,212 post(s)
#07-Feb-19 05:54

Try a subset so it's faster to stream down, ullr arg to gdal_translate. Got a link I could try?


https://github.com/mdsumner

joebocop
384 post(s)
#14-Feb-19 19:19

Ok, here's an example:

https://gte-rexcfnagudxgnuuhpqphnpuq.s3-us-west-2.amazonaws.com/cc_cogt_jpeg-ycbcr_mask.tif

Two interesting aspects of this tiff file are that

  • it was created from an "original" un-tiled RGBA tiff, using gdal 2.4, by running these two commands:

gdal_translate -mask 4 -b 1 -b 2 -b 3 -co PHOTOMETRIC=YCBCR -co TILED=YES -co COMPRESS=JPEG -co JPEG_QUALITY=50 --config GDAL_TIFF_INTERNAL_MASK YES cc_ortho.tif cc_cogt_jpeg-ycbcr_mask.tif

gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL -r average cc_cogt_jpeg-ycbcr_mask.tif

  • it is, therefore, a 3-channel YCBCR colourspace raster, having a 1-bit "mask band". It's not a 4th band, or an alpha band; it is a "mask", created in that first command from the original 4th band alpha channel (I understand this might be complicating matters, but the space savings are significant)

The Radiant.Earth marblecutter correctly detects the mask, as you can see the on-the-fly tiling preview here:

http://tiles.rdnt.io/preview?url=https%3A%2F%2Fgte-rexcfnagudxgnuuhpqphnpuq.s3-us-west-2.amazonaws.com%2Fcc_cogt_jpeg-ycbcr_mask.tif&rgb=1%2C2%2C3&nodata=&resample=average#15/64.8425/-139.8759

Thank you for having a look. Would be interested to know your experience with it.

Dimitri


5,491 post(s)
#15-Feb-19 11:40

so perhaps the Manifold TIFF dataport does not recognize tile tiffs, only non-tiled?

The Manifold TIFF dataport works fine with tiled TIFFs. Here are examples of the tiled test files from libTIFF, which all import perfectly:

cramps-tile.tif256x256 tiled version of cramps.tif (no compression)

quad-tile.tif512x384 tiled version of quad-lzw.tif (lzw)

zackthecat.tif234x213 8-bit YCbCr (OLD jpeg) tiled "ZackTheCat"

A quick way to see what a dataport can do is to find a known-good example of a given format (Google "examples of tiled TIFF" for instance) and try importing it. If it works, that's possibly something to cross of the list of what needs to be troubleshooted.

If you open a TIFF and see all white, launch the Style panel, select all three channels and apply medium Autocontrast and then remember to press the Update Style button.

Attachments:
cramps-tile.tif
quad-tile.tif
zackthecat.tif

joebocop
384 post(s)
#15-Feb-19 17:25

First, thanks for the autocontrast style tip.

Next, import of my example tiff, or any of your examples, works "as expected", no problems there.

I haven't been successful "linking" (old term, sorry) rather than importing, though. Here is the log window output when I expand the "linked" tiff datasource of my example tiff file:

2019-02-15 08:57:43 -- Create: (New Project) (0.005 sec)

2019-02-15 08:57:58 ** Unknown field with tag 42112 (0xa480) encountered

2019-02-15 08:57:58 ** Unknown field with tag 42112 (0xa480) encountered

2019-02-15 08:57:58 ** Unknown field with tag 42112 (0xa480) encountered

2019-02-15 08:57:58 *** Can not handle image with PhotometricInterpretation=4

Similarly, when I create a datasource from your "zackthecat.tif" file (which is also YCbCr) I get the same issue as with the YCbCr tiff file I shared, albeit without the log pane warning. Opening the datasource's image component shows an all-white screen.

If I copy the data source's image component and paste it into the project, the resulting image components behave as follows.

In the case of zackthecat.tif, with all three channels selected in the style pane, I can right-click autocontrast (med), and the value fields appear to be sensibly populated with float values ranging from -55 to 171. When I click "Update Style", however, the image component window remains all-white.

In the case of my example tiff, with all three channels selected in the style pane, I can right-click autocontrast (med), but none of the value fields' values change in any way. The range remains populated with 0-255. Clicking Update Style, of course, yields no change in the all-white display of the open image component window.

I have a bad feeling that I'm doing something fundamentally incorrect here, so please accept my apologies in advance if the solution is an RTFM'r.

Dimitri


5,491 post(s)
#15-Feb-19 17:37

I'm baffled what it could be. Here's what I did:

1. Clicked on the zackthecat.tif link in my prior post to download it from the forum.

2. Launched Manifold from bin64 (64-bit), using Build 9.0.168.8 (the current cutting edge build).

3. File - Create New Data Source.

4. Chose file: tiff as the Type. Uncheck save cached data between sessions. Navigate to the downloaded .tif and choose that. Press Create Data Source.

5. That creates the data source as you see it in the screenshot below. Open the image and it displays, no need to do anything.

Just for the heck of it, I did File - Link, navigated to the .tif and linked it. Here it is as a second "linked" (same thing) data source:

Attachments:
zack.png
zack2.png

joebocop
384 post(s)
#15-Feb-19 17:52

What the.

These instructions are so basic even I can follow them.

I had not tried File --> Link. My earlier attempts had all been of the Right-click --> Create --> New Datasource (file:tiff) variety.

If instead of that I do File --> Link, both my example image as well as zackthecat.tiff display correctly.

In the case of my example image, there is modest period of a dialog showing "copying" when I open the image component, but it does display. This note still displays in the log:

2019-02-15 09:48:06 *** Can not handle image with PhotometricInterpretation=4

Is there a difference between File-->Link and

CREATE DATASOURCE [Data Source] (

  PROPERTY 'Source' '{ "Source": "C:\\\\Users\\\\joe\\\\Desktop\\\\cc_cogt_jpeg-ycbcr_mask.tif", "SourceCacheExternal": false }',

  PROPERTY 'Type' 'tiff'

);

Attachments:
ztc_wtf.png

joebocop
384 post(s)
#15-Feb-19 20:21

Ok, so I had been caching no data like

Under those circumstances, I get a white-only component.

If I instead enable cache, like...

...then it "works", while still misunderstanding the "mask band".

Does that make sense, and does that behaviour equate to what is expected? The absence of cache will preclude the data source from displaying anything?

Thanks again, learning here.

Attachments:
ds_cache.png
ds_no_cache.png
ds_works.png

Dimitri


5,491 post(s)
#16-Feb-19 06:01

Yes, that's correct. As noted in the File - Link topic, "The File - Linkcommand is basically a simplified form of the File - Create - New Data Sourcecommand. " It checks Save Cache by default. Creating a data source explicitly saying not to use cache is not the same. Leaving the Save Cache box checked in the File - Link dialog, and leaving the Cache Data box checked when using Create Data Source, lets the system get around some of the limitations of read-only linked data. You can assign initial coordinate systems and the system can do (usually) some style manipulations as well that otherwise wouldn't be possible.

Try an experiment with the zackthecat.tif file: in Windows set the file to be read-only. Now, you can see what happens when you try to create a data source on it with the default option to Save Cache unchecked.

It can be trickier to do integration with the results of a GDAL stream because that's something of a black box, but the one thing you do know is that it is not bi-directional. It's a one-way, read-only thing.

A mild surprise is this

2019-02-15 09:48:06 *** Can not handle image with PhotometricInterpretation=4

That's a surprise on the GDAL front and on the Manifold front. GDAL apparently has different ways of expressing itself when it announces a four-band image. Import NAIP through GDAL and you get a separate RGB image and a mask image. Do it this way and you get a PhotometricInterpretation=4. OK.

Manifold should be tweaked to take that and consume it the way it does native NAIP fourband import, as an RGBA so that people can manipulate that fourth band as they like.

Dimitri


5,491 post(s)
#17-Feb-19 05:39

Forgot to mention: I sent in a suggestion to do as discussed in the last sentence above.

adamw


8,634 post(s)
#18-Feb-19 13:11

Yes, this is expected. If you do ... "SourceCacheExternal": false ... = uncheck 'Cache data' when creating the data source, the system will not let you change the style, because it has nowhere to put the changes, the data set is treated as read-only and that includes things like properties that define the style. Consequently, the style remains hard-set to the default 0..255 and that makes the image white.

For now, either link with cache or copy / paste the image to the writable parent data source and work with that image.

For the future, we are considering changing the default style for images to use the range of available values automatically. That way when linking without cache, you still won't be able to change the style (of the image inside the data source, will still be able to change the style if you copy and paste the image into a writable parent data source), but the image will at least make some sense. (Although even then, we'd suggest to save the cache. Without the cache, we have to recompute intermediate levels over and over, for example. If there are specific reasons for not wanting the cache, let's rather talk about them and try to solve them.)

For the mask band, could you reupload the image linked above somewhere else? I get "The specified key does not exist" when I try to download it.

joebocop
384 post(s)
#18-Feb-19 16:13

Thank you Adamw.

Here is the file:

http://gte-rexcfnagudxgnuuhpqphnpuq.s3-us-west-2.amazonaws.com/or_test/cc_cogt_jpeg-ycbcr_mask.tif.

adamw


8,634 post(s)
#19-Feb-19 07:07

Thanks, we reproduced the complaints about photometric interpretation, will take a look into what's going on.

adamw


8,634 post(s)
#19-Feb-19 11:52

A follow up:

We investigated and the file contains two series of images (same data at different levels): the "normal" one and the "mask" one. We are importing the "normal" image fine, but are ignoring the "mask" image. We will adjust the code to import the "mask" image as well (by making unmasked pixels in the "normal" image invisible).

Thanks for the file.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.