Subscribe to this thread
Home - General / All posts - TIFF regressions in 9.0.168.x?
tjhb

8,760 post(s)
#10-Apr-19 05:01

I am not sure whether these issues arose with 9.0.168.11 or with an earlier build (earlier beta builds no longer launch). They are not present in 9.0.168.0.

The import and export issues may be unrelated.

Has anyone else seen them?

(1)

The first is that I am seeing no data, after importing any image from a particular TIFF dataset in EPSG:2193 (New Zealand Transverse Mercator / NZGD2000), BGRA, uint8x4. The images are GeoTIFF. (I have removed the unnecessary worldfiles and auxiliary files; putting them back does not make a difference.)

The projection looks fine to me. Tile size in Manifold is 256 x 256.

I have tried all permutations in the Style panel, including RGB, BGR, RGBA, BGRA, and assigning each channel on its own. The result is always blank white.

Import of the same images is straightforward in 9.0.168.0. Projection parameters match those assigned by 9.0.168.11, tile size is again 256 x 256. Style is automatically assigned as BGRA. (I usually give the alpha channel a fixed value of 0, since it is empty.)

Most importantly, perhaps, if the imagery is imported in 9.0.168.0 and the project is saved, then 9.0.168.11 has no trouble opening it, displaying or manipulating the native images--except (so far) for large exports to TIFF, as in item 2 below.

Importing images from a second TIFF dataset succeeds in both builds. There are some differences. The second set is TIFF (not GeoTIFF) with .tfw, so projection needs to be assigned manually. Manifold tile size for these images is 128 x 128. Another difference is that local offsets are always integer, while for the first set the Y offset is always fractional (x.4).

(2)

A second issue with 9.0.168.11 is that exporting large images (e.g. 48000 x 60000 px) to TIFF format is failing at about 100%, with error "Cannot open file". The files that appear in the export location include .aux and .ige, besides .tiff and .tiff.mapmeta. Repeated several times, with two different images.

Exporting a smaller image (8000 x 12000) to TIFF from this build succeeds. It produces .aux, .tiff and .tiff.mapmeta files. No .ige.

From 9.0.168.0, exporting the large images to TIFF succeeds, from the same project, on the same machine, and produces only a .tiff file and a .tiff.mapmeta file.

I will send in a report in due course, with some sample data. If anyone else has seen something similar it would be good to know.

Dimitri


5,452 post(s)
#10-Apr-19 06:10

I have tried all permutations in the Style panel, including RGB, BGR, RGBA, BGRA, and assigning each channel on its own. The result is always blank white.

Have you tried manipulating the ranges, such as by setting autocontrast, full range, etc?

tjhb

8,760 post(s)
#10-Apr-19 07:42

Thanks Dimitri. I hadn't, but I have tried all those now. No change. All pixels stubbornly blank.

Now I have directly compared

(a) one image imported with 168.0, saved to a project, then reopened into 168.11

with

(b) the same image imported directly into 168.11.

I may have missed something, but here is the only difference I can find so far.

If I run

SELECT [X] AS [tileX][Y] AS [tileY],

SPLIT CALL TileToValuesX4([Tile])

FROM <image table>;

then for (a), I get values like [ 145, 143, 118, 0 ] for all non-NULL pixels; while for (b) I get values like [ 145, 143, 118, 255 ]. The BGR values are just examples. It is the "opposite" Alpha values that I'm wondering about.

That difference should not matter, since I'm assigning the A output to a fixed value of 0.

But it does seem to stand stand out. I'm not sure that we would expect Alpha data values to be "opposite" in the two cases--or rather only their interpretation.

tjhb

8,760 post(s)
#10-Apr-19 09:08

OK, let's say that's the only difference. That, when imported into 9.0.168.11, these RGBA images show channel 3 to be filled with value 255, whereas when imported into 9.0.168.0, channel 3 shows as filled with 0.

What if we fix that one thing in 168.11, i.e. fill channel 3 with 0 instead of 255. Do we get an image then?

Answer: yes.

UPDATE <image table>

SET [Tile] = 

    TileChannelsConcat(

        -- keep existing channels 0, 1, 2

        TileChannelsConcat(

            TileChannelsConcat(

                TileChannel([Tile], 0),

                TileChannel([Tile], 1)

                ),

            TileChannel([Tile], 2)

            ),

        -- replace values in channel 3 with zeroes

        TileFill(

            TileChannel([Tile], 3),

            0

            )

        )

;

Now when I open the Image, rebuild intermediate levels, switch to the Style panel and select either the RGBA or RGB preset, I get normal colour. Yay.

Tentative inferences:

(1) It may or may not be intentional for images which have an "empty" Alpha channel to have that channel filled with 255 on import. I don't know.

(2) More importantly, there seems to be a bug in the way images which do have value 255 in channel 3 are interpreted. This suppresses the other channels, regardless of whether channel 3 is used in the Style panel. Setting Alpha to a fixed value of 0, and mapping the other three channels to RGB, should mean that channel 3 has no effect. But instead it completely masks the other three channels. (By the way, the same applies if Alpha is set to a fixed value of 255.)

Last question. What if we do the opposite: instead of fixing an image imported by 9.0.168.11, try to break an image imported by 9.0.168.0, by filling its channel 3 (normally 0) with value 255?

Yes, that breaks it. The image becomes invisible, and we can't adjust the Style settings to bring it back (not even by ignoring channel 3 entirely).

(I haven't tested but imagine we can "break" any RGBA or BGRA image in the same way.)

adamw


8,579 post(s)
#24-Apr-19 10:14

By the way, when replacing the alpha channel, instead of splitting the tile into channels and concatenating them all in the order you want, you could have used TileChannels and specified which channels you want and in what order.

Eg:

--SQL9

? TileJson(

    TileChannelsConcat(

      TileChannels(

        StringJsonTile('[ 10, 20, 30, 40 ]', 1, 1, 4, false),

        VectorMakeX3(0, 1, 2)),

      TileMakeNew(1, 1, 0)

    ))

-- result: [ 10, 20, 30, 0 ]

This looks scarier than it is because of the calls to StringJsonTile / TileJson which I use to make the example self-describing.

What happens is this: We construct a 1x1 tile with 4 channels, initialized to 10, 20, 30, 40 using StringJsonTile. Then we take this tile and produce tile of the same size, but with channel values reshuffled - we take three channels: 0, 1, 2, and return a 1x1 tile with 3 channels. (So, this just throws out the last channel. We could have reordered channels: 2, 1, 0. Or we could have used the same channel more than once: 2, 1, 2. Etc.) Then we append the 4th channel filled with zeroes (could have filled with anything, obviously).

tjhb

8,760 post(s)
#24-Apr-19 13:20

Very nice (and useful). I had hoped there might be a better way! Thanks.

Dimitri


5,452 post(s)
#10-Apr-19 14:21

That arises from a set of changes made in 168.10, which are the same in 168.11:

Reading TIFF no longer reorders channels in color images to BGR and no longer inverts A from 0 = fully transparent to 0 = fully opaque. Instead, all data necessary to properly display the image is put into its style.

As to how that plays out, see the "Opacity or Transparency?" note at the end of this Example.

Another possibility is that maybe you invoked automatic inversion.

Specifying style for image channels allows inverting data by setting the channel minimum value to be higher than the channel maximum value. For example, switching the bounds for an improperly interpreted alpha channel from min = 0, max = 255 to min = 255, max = 0 will invert the interpretation without doing anything to the stored pixel values. See the example in the Style: Images topic.

See the "Example: Automatic Inversion" part of Style: Images topic.

See also adamw's discussion here, with some unified discussion based on that in the TIF topic.

tjhb

8,760 post(s)
#10-Apr-19 23:31

Many thanks Dimitri, that is a very useful summary for revision and extension. It was a while since I had read the Style: Images topic in full, and I had never read Example: Set Image Transparency using Alpha (should have). Both are great writing and teaching.

Adam's notes about ambiguous cases and the corresponding notes in the manual are also very useful.

I think you have cleared up my first "tentative inference" (actually not an inference, but a lack of one):

(1) It may or may not be intentional for images which have an "empty" Alpha channel to have that channel filled with 255 on import. I don't know.

The basic answer is yes, but it wasn't really the right question. The notes about automatic inversion are indeed the right place to start there. Thanks.

My next step on that score is to inspect the GeoTIFF tags, to see what raw material Manifold is dealing with.

On my second inference [reworded for clarity]:

(2) More importantly, there seems to be a bug in the way images which do have value 255 in channel 3 are interpreted. ... Setting Alpha to a fixed value of 0, and mapping [channels 0, 1, 2] to RGB, should mean that channel 3 has no effect. But instead [if channel 3 contains a value of 255, the effect is to mask channels 0, 1, 2, even if channel 3 is not used for Style at all].

I still tentatively think that is right. The images I am having trouble with may show a corner case where the particular arrangement/content of channels means that the Style options can't be used to make any pixels visible.

I'll post an example image, or a link to one, in case you are anyone else is interested in checking. If it seems warranted after more checks, I'll send in a bug report.

tjhb

8,760 post(s)
#11-Apr-19 00:15

I have attached gdalinfo output for one of the images. The image is posted on Dropbox for anyone interested to test. It is 264 MB (LZW compressed at source).

Alpha seems to be correctly tagged, and shows opacity, as Manifold expects for a TIFF image.

All pixels have channel 4 (Alpha) = 255, meaning 100% opaque (0% transparent).

But I get no opacity at all, instead 100% transparency. Moreover, I can't persuade Manifold either to respect the Alpha channel, or to ignore it, and show the RGB data.

Attachments:
BQ34_5K_0701.txt

Dimitri


5,452 post(s)
#11-Apr-19 06:06

I don't think it's the alpha channel: I think there's a problem in the TIFF import that results in no data being imported into the channels. I've reported it as a possible bug and have uploaded the sample to tech.

When you import it using .11 right clicking on the ranges in style leaves them at 0 to 255. Create a data source to the TIF using GDAL and the image appears immediately with range values such as 22 to 241.

Manifold imports it with what, at first glance, is the correct projection:

NZGD2000 / New Zealand Transverse Mercator 2000 (EPSG:2193)

GDAL does not import the projection.

Looking at the GDAL metadata, which Manifold reports but GDAL does not, we see a strange metadata tag:

<Item name="Corder">PCRL</Item>

That's a weird thing to find in a TIFF, since PCRL progression is something associated with a JPEG2000 file, not a TIFF file. Corder is a default setting of the Kanaku library apparently used within the OpenJPEG library that is used within GDAL.

My guess is that this image came from JPEG2000 format and was converted into geoTIFF using GDAL. Something about it now, some sort of GDALism, makes sense to GDAL but not to 9. Could be an issue in GDAL, or 9, or both, or it just might be one of those gray areas. In any event, if GDAL can read it but 9 does not, that should be fixed.

Regarding projections:

1. Create a data source for the TIF using GDAL. You get a visible image, but no projection.

2. Import the same TIF using 9. You get a projection, but no visible image.

3. Save the projection used by the imported TIF as a favorite coordinate system.

4. Delete the imported TIF.

6. Copy the linked TIF from the data source and paste into the .map project.

7. Assign coordinate system using the previously saved favorite.

8. Compare what you get in a map against a Bing satellite background layer.

The georegistration is not bad, but it is offset to the South by about 3600 meters. If we use Repair Coordinate System to create an "alt" version of the coordinate system with false northing set to 9996400, instead of to 10000000, it lines up perfectly.

That's just a quick hack, of course, with no explanation why it seems to work ("seems" because I didn't precisely look at pixel alignments in case of wrong pixel size, etc..). But it's a start.

PS: Sure wish we had a tag editor built in... be great to have that as a list, but not where it would bloat up the properties dialog, perhaps in a sidecar "tags" table. :-)

tjhb

8,760 post(s)
#11-Apr-19 07:10

I've reported it as a possible bug and have uploaded the sample to tech.

Thanks Dimitri, that's great.

You've made good points which I want to work through and follow properly. E.g. I had no idea what the Corder tag or PCRL value meant, but to my discredit I just ignored them.

However regarding your first point (I'm going backwards):

I don't think it's the alpha channel: I think there's a problem in the TIFF import that results in no data being imported into the channels.

That was my feeling at first, until I did this (above), that is, until I manually flipped the Alpha value back from 255 to 0. Doing that restored normal operation (after rebuilding levels).

Doesn't that prove that the import did (mainly) work, and that the problem is exactly in the interpretation of Alpha? If the RGB data can be retrieved just by remapping Alpha 255 to 0, then the RGB channels must have been imported correctly. It feels to me as if one small piece of code was not updated for the changes in 168.10, where Manifold introduced automatic (~heuristic) channel inversion for some source types.

Dimitri


5,452 post(s)
#11-Apr-19 09:11

Maybe alpha, but it seems there could be something else going on. Do this:

1. Import the .tif using .11

2. Open the image and zoom to fit. Nothing.

3. Scroll to zoom in. At around 1:2400 the image appears. Here's a location:

{ "Center": [ 175.4541789595971, -41.35739256963128 ], "Entity": "[BQ34_5K_0701]", "Scale": 1246.6903496042835 }

Put the above into a location and go there, and the image appears. So... it doesn't seem to be just the alpha interpretation if that is it, as however alpha is interpreted should be the same regardless of zoom.

The issue has been reported, tech has the data, so I'm sure this will get solved and a fix issued. Strange problem! :-)

tjhb

8,760 post(s)
#12-Apr-19 22:10

That was very useful. Thanks.

On this laptop monitor, the threshold linear scale is 1:1603 (visible at or below that). Native scale (1:1 pixel) corresponds to linear scale 1:1134. (So Zoom to native also works, of course.)

I tried recreating pyramids, and removing and restoring all indexes and properties, but neither of those things cured the issue (neitehr in .11, nor in .0 after import into .11 and saving).

This is just for interest, since as you say Tech will solve it.

Rakau75 post(s)
#10-Apr-19 18:00

I am also having some issues.

I use DroneDeploy a lot. Have had issues with their Tiff files going way back (http://www.georeference.org/forum/t143446.6#143447)

Last flight has 30 tiles. When importing into M9 and merging, there is a feint out line around edge, as attached.

Playing with the various settings as suggested by Dimitri can turn edge into solid black

No issues in M8 with same files

Attachments:
TiffOutLine.PNG

Dimitri


5,452 post(s)
#11-Apr-19 04:59

Post a sample. What build are you using?

Rakau75 post(s)
#11-Apr-19 07:07

Build: 9.0.168.11

I7 -3930k CPU @3.2Ghz

Cuda (GForce GTX 660 (3.0))

Ram 32 Gb

Win 8

The results.png shows a diagonal line top right to bottom left. If I merge all 28 images, I see what looks like an outline of where I would expect the merged image to show.

Attachments:
PaulApril19_Orthomosaic-5-0_export_MonApr08230125.905222.tfw
PaulApril19_Orthomosaic-5-0_export_MonApr08230125.905222.tif

tjhb

8,760 post(s)
#11-Apr-19 07:25

Obviously you won't want to post all 28 source images. But for anyone else to understand the problem, you will probably have to post more than just this.

Maybe some exact steps, or sequential screenshots, to describe your workflow? Possibly along with a picture of what you do expect as a result, since that depends on actual data, which no one else here has.

You've also said there is no problem using Manifold 8--but doing exactly what? Again exact steps or screenshots would be best.

Rakau75 post(s)
#11-Apr-19 08:31

Files are all about 30 MB

If anyone wants to pm me I canmaybec share the DD and they can be downloaded directly.

Will put together a step by step later

Rakau75 post(s)
#11-Apr-19 20:35

Have created basic workflow for both M9 and M8.

Playing around with the various options, I managed to get an image using RGB channel, however the results are far from satisfactory.

When zoomed into a similar area with the two versions, the quality of the M9 merged image is extremely course.

The import from Drone Deploy was 2.6cm resolution

Is there a dropbox address we can post large files to?

There are 84 files total of 574 Mb for the entire group

Attachments:
workflow.pdf

Dimitri


5,452 post(s)
#12-Apr-19 05:10

Is there a dropbox address we can post large files to?

A dropbox account is something you get for your use. Get an account and then you can share links to what you want to share for people you want to have access. Since dropbox has a bandwidth limit for their free service, put up a file for a few days so people interested in a thread can download it, and then remove the file so the link doesn't stay live for years.

Other popular choices are google drive and imgur. You can post .tif on imgur and zip together any accessory files, like a "world" file, that go with the .tif and post the .zip here since it will be under the forum limit.

Don't post all files. Post just one or two.

Forest
580 post(s)
#11-Apr-19 08:21

I have seen the transparent tiff issue too. Was planning on submitting a bug report. Note that ArcGIS correctly displays the same geotiffs so they are valid. Issue was experienced with Build: 9.0.168.11

vincent

1,774 post(s)
#11-Apr-19 13:22

I have the reverse situation here. 168.11 is the first version in which I see the content of tiff without any action. They where always white before (but still valid).

Dimitri


5,452 post(s)
#11-Apr-19 13:42

Post a sample. :-) All tifs are different.

vincent

1,774 post(s)
#12-Apr-19 13:08

ftp://transfert.mffp.gouv.qc.ca/Public/Diffusion/DonneeGratuite/Foret/IMAGERIE/Produits_derives_LiDAR/22D/22D09NE/MNT_22D09NE.tif

Dimitri


5,452 post(s)
#12-Apr-19 15:58

Umm.. I meant Forest, in response to "I have seen the transparent tiff issue too." :-) If you see the content of a tiff without any action and it all works OK, no need to post a sample.

Vincent, that's a 1.5 GB file. Is it OK ?

vincent

1,774 post(s)
#12-Apr-19 18:35

Yes. A good part of Quebec is covered by LiDAR, freely available, and DEM file are arround 1.3 GB each. There is Slope, Shading and DSM as well (remove last segment of the URL to access other products).

adamw


8,579 post(s)
#19-Apr-19 16:51

A second issue with 9.0.168.11 is that exporting large images (e.g. 48000 x 60000 px) to TIFF format is failing at about 100%, with error "Cannot open file".

We found this, this is our bug. We will fix it.

(Looking at the first issue.)

adamw


8,579 post(s)
#23-Apr-19 13:13

We found and fixed the first issue as well. The issue was not related to TIFF, it was related to all 4-channel images.

Thanks for the detailed report and the data.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.