Subscribe to this thread
Home - General / All posts - Importing large amounts of imagery
Mike Pelletier


1,575 post(s)
#18-Feb-19 16:09

We have roughly 1.4 TB worth of 4-band 16 bit tiff images and I'm struggling to get them into 9. Primarily I used them as a simple RGB image in the form of an ECW file. Mfd 8 and 9 both do not understand the projection created by Globalmapper which is what I used to create the file. 8 let's me reassign the projection while 9 does not. That is a large bummer and even more so because trying to import the ECW fails with the Windows message "the program needs to close".

Trying to import the original tiff files into 9 fails about a third of the way through and I get the same unhelpful error message and crash. I've reported this to tech.

Question for the forum. Have others successfully imported this much info and did you have to break up the import into small chunks?

adamw


8,447 post(s)
#19-Feb-19 06:42

What are the dimensions of the image you are trying to import, how much space do you have on the temp drive and how much space do you have on the target drive? Import is uncompressing the image, so you might simply be running out of space. In 8, importing a ECW / JPEG2K was embedding it into the MAP file, this no longer happens.

We will allow changing the projection of a linked ECW / JPEG2K image (that's our issue, yes - we currently don't let the ECW / JPEG2K dataports use .MAPCACHE and we have to so that they have a place to put the adjusted projection to).

Mike Pelletier


1,575 post(s)
#19-Feb-19 20:55

Ah, it's likely a temp file space issue. I have a dedicated 2 TB hard drive for the project file but from reading the manual sounds like I need at least 3 times the project size for the temp files. Wow.

Recommendations for the pagefile size?

Once I merge the images together does 9 store blank areas (white or black) efficiently like .ecw files or does it scale up as if the full rectangle contains all useful pixels?

So by the term embed do you mean 9 uses more temp file space compared to 8? Both ultimately seem to embed all their data in the MAP file unless using link.

Mike Pelletier


1,575 post(s)
#20-Feb-19 00:00

To be clear, my import job is 1400 tiffs each about 1 GB in size. Attached is my computer specs.

When I import a single tiff and save the project it is 1 GB in size, so perhaps this means the tiffs are not compressed.

The 9 documentation says the temp files can be up to 3 times the project file. But hopefully 9 just needs for temp files 3 times the size of the largest import file (3 GB) and not three times 1.4 TB ultimate project size.

Attachments:
Capture.JPG

adamw


8,447 post(s)
#20-Feb-19 07:14

Yes, we need the amount of space proportional to the changes - so, three times the size of the largest import between the saves. That is, when you have a 500 GB MAP file and you are adding 100 GB to it, we need 3*100=300 GB of free space. If the drive with the 500 GB MAP file is the same drive that hosts the temp folder, we need 300 GB on that drive. If the drive with the MAP file is different from the drive with the temp folder, the 300 GB is split between them: roughly speaking, 200 GB for the drive with the MAP file and 100 GB for the drive with the temp folder.

When you import an image into 9, it gets uncompressed from whatever format it was previously in. We don't compress images stored in 9 - we have a branch where we experiment with that and our general opinion that compression for images has to be lossy, otherwise the gains are too small to bother with, and everything lossy needs users consent - that's why we aren't generally compressing tiles right now. I am saying "generally", because we do compress tiles in several useful corner cases, but that's about it.

adamw


8,447 post(s)
#20-Feb-19 07:35

One more thing: if the original data for the image = level 0 is 1.6 TB, intermediate levels on top of level 0 will take about a third of that = ~0.6 TB more. We have been talking about dropping the very first intermediate level via an option to reduce storage requirements - this will reduce the required storage to something like ~0.2 TB at the cost of slower rendering at the zoom ranges which engage level 1 (slightly above the native scale). Or maybe we will allow specifying the step between intermediate levels - setting it to 3 will reduce the storage requirements to something like ~0.2 TB as well (slightly higher).

Dimitri


5,359 post(s)
online
#20-Feb-19 12:10

But hopefully 9 just needs for temp files 3 times the size of the largest import file (3 GB)

From the Performance Tips topic:

Temp files can expand to three or more times the size of the actual project.

If you have a 1.4 TB project, think in terms of a 4.2 TB temp space.

Look, a 6 TB disk can be had on newegg.com for around $115, less on sale. 12 TB is more, around $400. That's cheap compared to the cost of manually plowing through so many relatively big images (1 or 2 GB isn't big, but a thousand of them adds up...), and it's cheap compared to the time you save using 9 instead of something else that takes a long time to work with a 1 GB image.

Given how cheap disk space has become, I think whatever Manifold can do to increase speed at the cost of storage size, including retaining the very first intermediate level of an image, is a good trade off. You forget the cost of an extra $100 or even $400 in disk cost very quickly, but slow response with bigger images is something that annoys you every time you work with that image. :-)

Dimitri


5,359 post(s)
online
#20-Feb-19 13:29

Should clarify the above: the "three times" bit is a safe, really safe, buffer if you will do big things with the data like creating copies, merging and such. If all you are doing is importing and just looking at the data then basically all you are looking at is the size of the data plus intermediate levels like adamw as listed. Apologies for automatically jumping into "three times" mode.

Mike Pelletier


1,575 post(s)
#20-Feb-19 14:47

Thanks Adam and Dimitri. This clarifies things up nicely. It would be nice to add this detail to the documentation. Yes, it's time to invest in more storage.

Started the import again yesterday with an external 1 TB drive being used for the temp file. It's at 15 hrs and appears to be 3/4 of the way through. Good!

Mike Pelletier


1,575 post(s)
#20-Feb-19 15:36

... and the process failed as expected based on 1 TB temp drive being to small. New drivers are on order.

adamw


8,447 post(s)
#20-Feb-19 16:55

When you start over, consider doing the import piecemeal even though you will have bigger drives. It's just more robust in case something goes wrong for reasons unrelated to the amount of space - eg, if some of the files refuse to import.

After the import is done, you'll probably want to merge everything together into a single component - that will be a big and bulky step and if you start running low on space even with new drives, consider merging into a separate MAP file (put the MAP file with the imported images onto drive 1, create a new MAP file on drive 2, link the MAP file with the imported images, create a map in the new MAP file, bring all images from the linked MAP file, merge into a new image in the new MAP file, wait, save).

For the record, based on your report of 3/4 out of 1.4 TB done in 15 hours, it was going at about 20 MB/sec - pretty good, that's good sustained speed for a big data set that overflows all caches it hits (plus there's fragmentation, decoding / repackaging in the middle, plus that's read + write... the speed was good).

Mike Pelletier


1,575 post(s)
#20-Feb-19 18:27

Thanks for the methodology advice!

After merging the images, the result will have a bounding box that includes large amounts of no data. Will this balloon the project size? If so I can chunk up the merged areas to keep that to a minimum.

Yup, 20 MB/sec is what I remember the status bar reporting. Nice to hear my system is running well.

adamw


8,447 post(s)
#21-Feb-19 07:20

Empty space should not use much storage (merging does not even create records for tiles that are completely empty, indexes handle this well too).

One more thing regarding merging:

Merging requires putting images into the same map first. We are currently rendering maps with tons of layers way too aggressively, so if you just create a map with all images and open it zoomed to fit, it might clog the UI for a long time. Here's a better method: create a map - drop a single image - zoom to it and zoom a little more into it so that it covers the whole screen and other images are outside of the screen - then drop remaining images in packs of, say, 50-100, and keep turning them off in the layers pane (so, drop a pack of images, turn them off, drop another pack, turn them off, etc). You don't have to remember which images you already dropped vs which ones you didn't drop yet, you can just drop them all into the map again at the very end and duplicates are going to be ignored. After you drop all images and have them all or nearly all turned off, save the MAP file (will go fast), then do the merge.

We'll absolutely adjust our code so that you don't have to keep any of this in mind and can just drop all layers at once and zoom to fit and do whatever, but right now it is better to pay attention.

Mike Pelletier


1,575 post(s)
#08-Mar-19 17:33

Using the latest build, 168.10, with the project describe above, I run into trouble with intermediate levels. Upon import of a sample of five images, I get the notice to build intermediate levels. I built them on two images, then zoomed in on one image as suggested in a map, added other images, and then ran the merge tool. Within the merged image, the 3 images without intermediate levels show up black and the 2 with intermediate levels look good. There is no message offering to build intermediate levels.

tjhb

8,657 post(s)
#08-Mar-19 22:21

What happens if you build intermediate levels for all images?

Easy to do (minimal SQL).

adamw


8,447 post(s)
#09-Mar-19 08:48

We'll check what's going on.

dchall8
578 post(s)
#08-Mar-19 18:12

What is your importing process?

Back in 2017 I was playing around with a TB or so of Lidar files. Importing them was tedious to say the least. The forum came up with a solution that worked great.

Mike Pelletier


1,575 post(s)
#08-Mar-19 19:11

Thanks for posting! That might be just the ticket once the image display issues are figured out.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.