Subscribe to this thread
Home - General / All posts - Pb to import large tiff image
Corentin97 post(s)
#28-Jan-21 18:15

Hi,

I got from there a 9Go tiff image :

https://eogdata.mines.edu/nighttime_light/annual/v20/2019/VNL_npp_2019_global_average_vcmslcfg_c202010201200.tif.gz

if I import it in M9 I got a table, which is like that :

then I create an image from this table from this window :

Then what ever I put in the "bound" field, I always get an ugly image like :

I am supposed to see nightlight of earth...

What is wrong?

Dimitri


6,511 post(s)
#28-Jan-21 18:37

Unless the image is 128 pixels by 128 pixels, the bounds are wrong.

If it is a tiff image, importing a tiff image creates an image and a table. There should be no need to create an image manually.

What does the .gz decompress into? How do you import that?

Just for fun, I'm downloading the image now. It's a slow server, so looks like it will take a few hours.

Corentin97 post(s)
#28-Jan-21 18:41

The .gz decompress as a tif. I simply did "file/import" in M9. I was expecting Manifold to create automaticaly the image. But I got only the table.

Corentin97 post(s)
#28-Jan-21 18:43

I did it twice, got twice only a table. The second time the table has more records??

First time the table had ~73000, second time it had ~85000 records.

I opened the tif in QGIS without any problem

Dimitri


6,511 post(s)
#28-Jan-21 18:52

Could you be running out of temp space or disk space? I downloaded a 284 MB file from that same EOG folder, and it decompressed into a 11.3 GB TIF, so I'd expect a 9.3 .gz to decompress into what, about 370 GB if it is the same compression? Do you have enough free disk space in TEMP and so on?

The 11 GB TIFF opened just fine in 9. That "small" image has 177,788 records and is 86,401 x 33,601 pixels in size. The "big" image would be 32 times larger if the .gz compression ratio is the same, so you're not getting all the data. My guess is that you're running out of disk space.

Attachments:
night_lights.png

Corentin97 post(s)
#28-Jan-21 19:02

The problem is clearly there :

The tiff is 10.8Go. I only have 9Go of free space on my disk. When M9 is done importing the data, my free space is 0 octect!

I know I need to clean this disk ;)

I did it again and saw that the Manifold log says 'cannot write data'.

There is no way manifold tells me that he fails importing the data because I am running out of disk space? (a popup message maybe?)

Thank you for your help! I will clean my disk and try again!

Dimitri


6,511 post(s)
#29-Jan-21 07:20

OK, to follow up....

I downloaded the 9.3 GB .tif.gz file and it unzipped (un-gz'd, that is...) into an 11 GB .tif file. So, whoever is compressing these files is using different compression ratios for the different files, since the 284 MB file also decompressed into an 11 GB file.

The resulting .tif imports fine into Release 9 (image below).

As to Manifold checking your Windows setup, that can require significant work for applications that leverage Windows like Manifold, where you want all Windows settings to automatically apply and be honored. It's not just running out of disk in the folder where you want to extract or manually write something, it is running out of disk space in places like whatever is the TEMP folder or page/swap space.

It's a performance hit to always be double-checking Windows every time such operational memory is used. In the tradeoff between running slower and simply advising people "make sure you have plenty of free space," in an era where 4 TB disks cost $55, the latter is probably the way to go. :-)

That's not part of the world for limited, client apps like Q or Arc, which don't have the ability to import anything but can only link. They save space with that strategy, but they end up being much slower and also are constrained by whatever can be done in a 1980's format like TIF (speed, for example), aren't portable to different machines just by moving a project file, etc, etc. Link-only packages do simplify storage management, since whatever space the linked file occupies in the folder where it sits is pretty much what it takes.

By the way, it's interesting to see that this is one case where Manifold .map format results in less storage space required than TIF. Importing the 11 GB TIF file into Manifold and saving the project resulted in an only 3.3 GB project.

Attachments:
night_lights2.png

Dimitri


6,511 post(s)
#29-Jan-21 07:50

Importing the 11 GB TIF file into Manifold and saving the project resulted in an only 3.3 GB project.

One more thing... the 3.3 GB .map project was for importing the 11 GB TIF file that was in the 284 MB .gz container.

Importing the 11 GB TIF file that was in the 9.3 GB .gz container and then saving that project resulted in a 15.2 GB .map project file. That's a more expected result, where importing lots of TIF data into a project and saving the project usually will result in slightly larger .map storage size (.mxb tends to be much smaller, of course).

I guess that's the difference between the "median masked" data that was in the 284 MB .gz file and the "average" data in the 9.3 GB .gz file. The median masked data is way more amenable to being compressed, both by GnuZip (.gz) compression and by Manifold .map project storage.

Corentin97 post(s)
#29-Jan-21 10:10

Thank you very much for your time and your help

my c: drive is full but I will let my /temp on this disk because it is a SSD. I have plenty of space on my other drive which is not ssd...

This drive is full because of a box drive (a dropbox like app) that cannot be set to locate its file elsewhere than on C:/

Thank you for your answers that are alway very usefull and informative. I always learn something reading you :)

Corentin97 post(s)
#03-Feb-21 12:36

After cleaning the disk the image import easily!

Thank you for your help

oeaulong

309 post(s)
#28-Jan-21 18:40

Are there any clues in the metadata? Projection? Datum? pixel size?

9+GB is not trivial for me to register & download to test. sorry.

Dimitri


6,511 post(s)
#28-Jan-21 19:00

Yes, and the 9+GB is just the .gz. Decompress it and it gets much bigger. :-) I have a drive with a few terabytes free, but it's a really slow drive. I'll see how it goes.

oeaulong

309 post(s)
#28-Jan-21 19:02

Thank you D. The necessary overhead space makes sense.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.