And now, let's be explicit about GIS:
just tested this with the latest Manifold Edge. Same issue
Arc is a quality product that makes a great point of reference. This is definitely a topic worth exploring with precision, to help guide the way forward, and to avoid making Farmer Andy mistakes.
I assume you mean the image in the URL at the beginning of this thread. That unzips into a 16 gigabyte .ige image data file with a 27 MB .img header and some other files.
Like many things in GIS, the best way depends on what you need to do. To answer that you have to ask yourself two key questions:
1. Is this a one-time thing?
This is the key question. People don't usually just glance once at 16 gigabytes of data and then throw it away forever. They usually look at it more than once on more than one day. They pan and zoom. They use it in layers with other images or vectors. They often do something with it, like processing or re-projection. They usually need to save intermediate versions of projects that use the data.
2. Do you care about viewing or processing?
Processing is where you can get really killed on time. It's nice to improve first-time viewing speed, but that's a one-time saving of seconds or minutes at most. A far bigger win is reducing processing time from eight hours or eight days to one hour or one day, especially if you have to do that processing more than once using that same data as a base.
Is the Farmer Andy approach right for you? If you are convinced that for *all* cases you will *only* take a single look at a big image and then never, ever look at it or do anything with it, well, then sure, you should choose a package which does that fastest even if everything else it does is slower.
Most of us will prefer the Farmer Mike approach: in the real world we know we tend to work with the same image more than once, we need to look at it more than once, we need to do stuff with it and we need to save intermediate steps to have fallbacks for when Windows 10 decides to reboot for an update in the middle of a big job. We are willing to take a bit more time just once so that after that everything is always much faster.
The Farmer Andy approach is to leave the data in .img format, be happy it opens quickly, and learn to live with slower display and much slower processing.
The Farmer Mike approach is to just once convert to .map format, and then be happy that every time after that you use the image it opens instantly, displays instantly, pans and zooms and visualizes faster and usually processes much faster, often from four to twenty times faster, and saves any changes instantly.
Are you sure you want to be Farmer Andy?
Let's consider the numbers. I downloaded the file, unzipped it and then, using a slow machine with an external disk drive, I brought it into a Manifold project twice:
First, I used File - Link to link it. It linked instantly, opening the file instantly with the image and table appearing instantly in the project from the ensemble of files.
--> That is important in the real world, because when you open ensembles of files that can contain many images it is good to have the entire list appear instantly. You can then decide what you want to see.
Manifold defers first-time display computation of images, to avoid spending time computing images you don't care about. In this case, when you display the image the first time that took 555 seconds (9 minutes). After that initial hit, it always displays the image instantly and it pans and zooms instantly.
Saving the project with a link, the first time it took 94 seconds. Call it 11 minutes total time to link the .img, open it the first time, and then save the project. That's the part Farmer Andy would complain about.
But then, always after that, whenever you open that .map it opens instantly, it displays the image instantly, and it pans and zooms within the image instantly. It also saves changes instantly. That's the part Farmer Mike loves.
All that happens because Manifold works magic when linking such files so that after the initial computation they can display and be worked with instantly. Some of that magic helps with processing, but not as much as importing into a .map file would help processing.
So let's look at the import case. We'll do the "full Farmer Mike" thing.
I used File - Import to import it. It took 1269 seconds (21 minutes) to import the data, It then, of course, opens the image instantly and pans and zooms instantly. Saving the project took an additional 360 seconds (6.5 minutes), which comes to a total of, say, 28 minutes to migrate the 16 GB image into a .map project for routine, instant use with Manifold.
But that is a one-time overhead. Once it is in .map it is super fast always. It opens instantly, pans and zooms instantly, and quite likely will be significantly faster for processing than Arc.
Processing is the obvious comparison. Does the Farmer Mike approach do in one hour what the Farmer Andy approach does in eight hours? I guess that depends on what you do.
So far, it's not been easy to find something that Arc will do as fast as 9 or faster than 9. Usually 9 is three to ten times faster, depending on how many cores your machine has and what you are doing. So, if something took you six hours in Arc but even counting the conversion time to .map it would take you only two hours in Manifold, that's clearly a big gain, even for just the one-time case. But that's just a guess without a direct apples to apples comparison.
Until you get all the raster tools you want, you can make comparisons with those that are there already (various tile functions). If you do something involving the entire image after importing it into a .map, how does that compare to the time it takes to do exactly the same thing in Q or Arc against the .img/.ige ensemble? Is it faster or slower? How does changing the number of cores used in 9 affect the speed? How do the GPU-enabled functions compare?
Like I say, all the cases we've tried 9 is faster (or, we've fixed 9 so it is...). If you find a case where Arc is faster, let us know and we'll make sure whatever is holding 9 back goes away so 9 is faster in that case as well. :-)
What's important about using .map, by the way, is not just processing. Display is faster also, as you can see for yourself right now, comparing Arc using the .img to Manifold using the same image in a .map project:
See a) how quickly Arc can open a saved project, b) how quickly Arc generates a first display of that project, c) how quickly Arc can pan and zoom to arbitrary locations within that image, d) how quickly Arc can re-project on the fly in real-world use, and e) how quickly Arc can save versions of the project. Is it as fast at all that as 9? Where is it faster and where is it slower?
for example, launch the .map with the 16 GB image in it, open the particular sample image in 9 an image window. Zoom to Fit. Draw a very small zoom box over Seattle. Zoom to fit. Next, in turn draw very small zoom boxes into Miami, Boston, San Diego and Kansas City, zooming to fit in between. It takes a tenth of a second to redisplay any of those with 9. How long with Arc? Can you really zoom in anywhere and absolutely instantaneously pan and zoom however you like? I expect ESRI products to be fast at display, but some, including some of their latest, are far slower than 9 at such display.
Next, create a map and drop a Bing streets layer into it. Zoom into the view showing the US in your map. Drag and drop the 16 GB image into it. Next, create a Google streets transparent layer and drag and drop that into the map above the 16 GB image. You now have three layers.
The 16 GB image is in Albers conic. To show it in a map that uses Pseudo-Mercator requires a re-projection on the fly with every pan and zoom.
With Manifold you can pan and zoom instantaneously in that ensemble, even considering that Manifold is re-projecting on the fly that 16 GB image from Albers conic into Pseudo Mercator. Is the display of all three layers and the re-projection of the 16 GB image likewise instantaneous in Arc? Can you pan and zoom with only 1/10th of a second to redraw any display? Try it in ArcGIS Pro.
Create some locations so you have some extra stuff in the project. Add a vector layer to overlay the other layers. Display all that, panning and zoom box to various locations to give all the data a real workout. Save the project. How long to save? How long does it take the equivalent Arc workflow to add the vector layer, reproject on the fly to display, and to save the Arc project? Where is it faster or slower?
Those are all very important real world questions, to avoid making Farmer Andy mistakes. My experience is that 9 is dramatically faster than, say, ArcGIS Pro in the above, but I'd like to hear reports from others. I'd bet that anybody who actually tries such comparisons isn't going to praise the Farmer Andy lifestyle.
Let me close by emphasizing that if you really do want a one-time glance at big data I agree it is convenient to have a Farmer Andy machine. Quick hookup is not a mistake for a one-time user, it is the mission. Sometimes you don't know what the big data is and you need to look at several big data sets knowing full well most won't be right and won't ever be used.
I agree that in such cases it is wonderful to have a quick viewing capability, and that whatever can be done in Manifold to provide such a quick look should be done, so long as that does not end up with lesser performance in other cases, as currently happens with Arc. Until that happens, there's nothing wrong with using some other tool for a quick look but using Manifold for day to day superfast operations. It's good to have that option.