Subscribe to this thread
Home - General / All posts - How can I import a large IMG file and have it work properly in Manifold?
Ajayk3 post(s)
#22-Sep-17 14:21

How can I import a large IMG file and have it work properly in Manifold?

I've searched pretty hard in the Help files and this forum for hours and got nowhere so far.

Manifold System 8.0 Professional Edition Build 8.0.30.0

CPU Intel Core i7-6650U CPU @2.20Ghz

RAM 16309 Mbytes

OS Windows 8 (b9200)

Running in 64 bit mode

Microsoft Surface Pro 4

Running in 64 bit mode

Windows Update says up to date as of today.

Edition Windows 10 Pro

Version 1703

OS Build 15063.608

64- bit operating system

Steps

I downloaded the USA National Landcover database in zipped format and unzipped it.

https://www.mrlc.gov/nlcd11_data.php > 'NLCD 2011 Land cover 1.1Gb'

The unzipped folder contains these files

18/09/2017 06:29 PM 34,234 appendix1_nlcd2006_scene_list_by_path_row .txt

18/09/2017 06:29 PM 46,375 nlcd_2006_landcover_2011_edition_2014_10_10.faq.html

18/09/2017 06:29 PM 55,036 nlcd_2006_landcover_2011_edition_2014_10_10.html

18/09/2017 06:31 PM 16,839,202,917 nlcd_2006_landcover_2011_edition_2014_10_10.ige

18/09/2017 06:31 PM 27,753 nlcd_2006_landcover_2011_edition_2014_10_10.img

18/09/2017 06:31 PM 591 nlcd_2006_landcover_2011_edition_2014_10_10.img.vat.dbf

18/09/2017 06:31 PM 1,409,090,606 nlcd_2006_landcover_2011_edition_2014_10_10.rrd

18/09/2017 06:31 PM 37,314 nlcd_2006_landcover_2011_edition_2014_10_10.sgml

18/09/2017 06:31 PM 42,474 nlcd_2006_landcover_2011_edition_2014_10_10.txt

18/09/2017 06:31 PM 41,303 nlcd_2006_landcover_2011_edition_2014_10_10.xml

18/09/2017 06:31 PM <DIR> spatial_metadata

11 File(s) 18,248,578,603 bytes

3 Dir(s) 101,796,048,896 bytes free

I opened Manifold and used File>Import>Image to import

nlcd_2006_landcover_2011_edition_2014_10_10.img

Upon opening, Manifold states image size is 161190 x 104424

This is confirmed in the folder FAQ file

"Dimensions 104424 x 161190 x 1, type Pixel "

I experienced the same problems that were reported on the GeoRef community board in June 2014

http://www.georeference.org/forum/t122083.14#122084

namely, that all I could see on screen was a black image.

The thread does not contain a resolution of the problem.

The black image appeared to be correctly projected and contain the right number of pixels.. eg moving the cursor showed expected Lat Longs, and I was able to add the image into a Map with a drawing of US state boundaries and it looked to be in the right place

I tried following some of the steps discussed in the GeoRef thread, viz copying and pasting as either image or surface.

The whole image took 5 hours to C&P (!!)

I also tried taking a small selection box (checked that it was a part of the map that should have data, north-central Kansas), and C&P that as image or surface, but got no usable result, still just a mini black image.

The box was very small, just 4 pixels, however it was very slow to select and later de-select. Ie more than 3 minutes processing time

I also C&P the small selection box as a Table, and found that the table had zero for all colour-related fields, eg RGB 0,0,0 for all pixels.

There appeared to be no non-intrinsic fields in the data, and 27 intrinsic data fields. Only the co-ordinate fields had non-zero values.

I expected a field titled "Value" because....

(from FAQ. Html file in unzipped folder)

Value

Land Cover Class Code Value. (Source: NLCD Legend Land Cover Class Descriptions)

Value Definition

11 Open Water - All areas of open water, generally with less than 25% cover or vegetation or soil

12 Perennial Ice/Snow - All areas characterized by a perennial cover of ice and/or .... & etc

The FAQ file also states the default colours assigned to the Values.

I checked the Manifold help file on the topic "Import Image>Importing Images"

And found this in the list of supported formats

ERDAS IMG

ERDAS files in .img format. Works for IMAGINE format .img files.

I have successfully imported .img files before in to Manifold.

The FAQ states the file type as

Data format: ERDAS (version Imagine 9.3) .img Size: 1392.69

"What hardware or software do I need in order to use the data set?

ESRI ArcMap Suite and/or Arc/Info software, and supporting operating systems.

Finally I checked to see if the files may have been corrupted during unzip.

I downloaded a trial version of Global Mapper v18.2, and imported the *.img file.

It opened in less than half a second, and I was able to zoom in and out with no noticeable lag. The colours on the map looked to correctly match those described in the metadata. This indicates that the IMG file is not corrupted, and that it can be read satisfactorily with my computer's RAM and CPU. The trial version of Global Mapper doesn’t allow me to export data, so I can't save and export in a different format.

Can anyone help me to open this image and make it usable in Manifold. ?

Alex J

Dimitri


4,941 post(s)
#22-Sep-17 15:12

and that it can be read satisfactorily with my computer's RAM and CPU

With Global Mapper. GM is more oriented to big images than Release 8. In contrast, Release 8 is much better for vector and data work.

Both GM and 8 are good products but each has a different emphasis and each has a different technology approach. 8, for example, expects that given RAM is nearly free you're going to have more than 16 gb if you intend to work with big images. Begin by reading the Performance Tips topic so you can operate 8 as intended.

If you don't want to deal with that, no problem. Manifold has a product for you that does huge drawings and images with zero effort: For big images use Radian (Manifold Future). Get a taste with the free Manifold Viewer (use the Future Viewer edition to see the very latest and most powerful).

Radian-based products also have the very latest dataports for a much wider selection of supported formats and variations of those formats. You can check out any data set of interest with Future Viewer to see how it works. See also the YouTube videos in the Gallery page. Radian can pop open a 110 GB image in 1/10 second... much quicker than Release 8.

Ajayk3 post(s)
#23-Sep-17 06:42

Thanks Dmitri,

the Performance tips link really had nothing to offer for my Win10 machine. I have plenty of RAM and fast processors.

I was able to open the very large IMG file with QGIS

Like GM, QGIS opened the file instantly.

QGIS is free, and it allowed me to convert and export/save as .bil .hdr surface format.It took about 30 minutes all up to convert and save.

Manifold 8 was then able to import and open this surface, but it was still extremely slow. ... more than 30 minutes to open the file, then more than 30 minutes again to open the image, and more than 30 minutes to save even with no changes . Also very slow to do any basic processing, eg to clip out one state also took more than 30 minutes.

I looked at the user manual for Radian Studio... it doesn't say it will open IMG files...

tjhb

8,168 post(s)
#23-Sep-17 08:22

So you haven’t tested Manifold Viewer/Manifold Future?

Why? That is nuts.

artlembo

3,074 post(s)
#26-Jul-18 14:55

just tested this with the latest Manifold Edge. Same issue. Takes extremely long to bring in the file - either as an import or a new data source. QGIS and ArcGIS open the file instantaneously.

As far as processing, there are about 16 billion pixels. I ran a cross tabulation command in ArcGIS with a second 16B pixel raster, and it completed it - I hadn't timed it, left for the day and when I got back, the command was completed. But, ArcGIS not only read both files in an instant, but was able to chug through the processing of a cross tabulation.

I know more raster functionality is coming with 9, so it will be interesting to see how the performance improves, and what additional raster functions are introduced (raster calculator, map algebra, cross tabulation, watershed creation, line-of-sight, etc.). This is part of the reason I am holding off on the How do I do that in Manifold 9 book - it is all ready for vector, just waiting for the additional raster tools.

Dimitri


4,941 post(s)
#27-Jul-18 07:08

I'll reply in two posts.

First, the parable of the two farmers, Farmer Andy and Farmer Mike...

Both are good farmers, and both have a cow from which they get milk to make cheese. Both make good cheese. Both farmers have a milking machine that also makes cheese.

Farmer Andy bought his machine when the salesman showed him how the machine instantly hooks up to the cow and instantly starts milking. Farmer Andy likes that so much he doesn't mind that the machine takes all day to make cheese.

Every day, Farmer Andy gets up early in the morning to milk his cow and make cheese. Every day he loves it that the machine hooks up instantly and starts milking right away. He's gotten used to waiting around all day for the cheese. He doesn't like it that he has to spend so much time waiting for the machine to make cheese, but he consoles himself with the idea that the process starts quickly in the morning.

Farmer Mike bought a different machine after he got tired of waiting all day for cheese. The first time Farmer Mike used his machine it took him an entire hour to set it up for his cow. But after that, it always hooks up instantly, it always starts milking instantly and it always makes cheese in just one hour.

Every day, Farmer Mike gets up early in the morning to milk his cow and make cheese. Every day he loves it that the machine hooks up instantly and starts milking right away. It only took a couple of days for him to forget that the very first time it took him a whole hour to configure the machine for the cow. Now it's always instant. Even better, Farmer Mike loves it that the machine makes cheese in just one hour. Every day Mike thinks "Wow... hard to believe I used to sit around all day waiting for the old machine to make cheese, and now I get done in just one hour!"

Farmer Mike loves all the extra time he gets. He spends more time with his family, he gets to enjoy life more, and he has plenty of extra time to make more money by being a better farmer. He pities his neighbor, Farmer Andy, who is stuck waiting all day for cheese.

Farmer Mike has tried to convince Farmer Andy that choosing his machine on the basis of a one-time startup was a very poor decision. But Farmer Andy won't hear of it. He keeps saying, "Oh, I tried your machine at the trade show and it took a whole hour to hook up to the cow!" He just won't listen to Farmer Mike when Farmer Mike says, "But that's only the first time: every time after that it is instant, and then it makes cheese in one hour while you spend all day, every day, waiting on a slow machine!"

Would you rather be Farmer Mike or Farmer Andy?

Dimitri


4,941 post(s)
#27-Jul-18 08:11

And now, let's be explicit about GIS:

just tested this with the latest Manifold Edge. Same issue

Arc is a quality product that makes a great point of reference. This is definitely a topic worth exploring with precision, to help guide the way forward, and to avoid making Farmer Andy mistakes.

I assume you mean the image in the URL at the beginning of this thread. That unzips into a 16 gigabyte .ige image data file with a 27 MB .img header and some other files.

Like many things in GIS, the best way depends on what you need to do. To answer that you have to ask yourself two key questions:

1. Is this a one-time thing?

This is the key question. People don't usually just glance once at 16 gigabytes of data and then throw it away forever. They usually look at it more than once on more than one day. They pan and zoom. They use it in layers with other images or vectors. They often do something with it, like processing or re-projection. They usually need to save intermediate versions of projects that use the data.

2. Do you care about viewing or processing?

Processing is where you can get really killed on time. It's nice to improve first-time viewing speed, but that's a one-time saving of seconds or minutes at most. A far bigger win is reducing processing time from eight hours or eight days to one hour or one day, especially if you have to do that processing more than once using that same data as a base.

Is the Farmer Andy approach right for you? If you are convinced that for *all* cases you will *only* take a single look at a big image and then never, ever look at it or do anything with it, well, then sure, you should choose a package which does that fastest even if everything else it does is slower.

Most of us will prefer the Farmer Mike approach: in the real world we know we tend to work with the same image more than once, we need to look at it more than once, we need to do stuff with it and we need to save intermediate steps to have fallbacks for when Windows 10 decides to reboot for an update in the middle of a big job. We are willing to take a bit more time just once so that after that everything is always much faster.

The Farmer Andy approach is to leave the data in .img format, be happy it opens quickly, and learn to live with slower display and much slower processing.

The Farmer Mike approach is to just once convert to .map format, and then be happy that every time after that you use the image it opens instantly, displays instantly, pans and zooms and visualizes faster and usually processes much faster, often from four to twenty times faster, and saves any changes instantly.

Are you sure you want to be Farmer Andy?

Let's consider the numbers. I downloaded the file, unzipped it and then, using a slow machine with an external disk drive, I brought it into a Manifold project twice:

First, I used File - Link to link it. It linked instantly, opening the file instantly with the image and table appearing instantly in the project from the ensemble of files.

--> That is important in the real world, because when you open ensembles of files that can contain many images it is good to have the entire list appear instantly. You can then decide what you want to see.

Manifold defers first-time display computation of images, to avoid spending time computing images you don't care about. In this case, when you display the image the first time that took 555 seconds (9 minutes). After that initial hit, it always displays the image instantly and it pans and zooms instantly.

Saving the project with a link, the first time it took 94 seconds. Call it 11 minutes total time to link the .img, open it the first time, and then save the project. That's the part Farmer Andy would complain about.

But then, always after that, whenever you open that .map it opens instantly, it displays the image instantly, and it pans and zooms within the image instantly. It also saves changes instantly. That's the part Farmer Mike loves.

All that happens because Manifold works magic when linking such files so that after the initial computation they can display and be worked with instantly. Some of that magic helps with processing, but not as much as importing into a .map file would help processing.

So let's look at the import case. We'll do the "full Farmer Mike" thing.

I used File - Import to import it. It took 1269 seconds (21 minutes) to import the data, It then, of course, opens the image instantly and pans and zooms instantly. Saving the project took an additional 360 seconds (6.5 minutes), which comes to a total of, say, 28 minutes to migrate the 16 GB image into a .map project for routine, instant use with Manifold.

But that is a one-time overhead. Once it is in .map it is super fast always. It opens instantly, pans and zooms instantly, and quite likely will be significantly faster for processing than Arc.

Processing is the obvious comparison. Does the Farmer Mike approach do in one hour what the Farmer Andy approach does in eight hours? I guess that depends on what you do.

So far, it's not been easy to find something that Arc will do as fast as 9 or faster than 9. Usually 9 is three to ten times faster, depending on how many cores your machine has and what you are doing. So, if something took you six hours in Arc but even counting the conversion time to .map it would take you only two hours in Manifold, that's clearly a big gain, even for just the one-time case. But that's just a guess without a direct apples to apples comparison.

Until you get all the raster tools you want, you can make comparisons with those that are there already (various tile functions). If you do something involving the entire image after importing it into a .map, how does that compare to the time it takes to do exactly the same thing in Q or Arc against the .img/.ige ensemble? Is it faster or slower? How does changing the number of cores used in 9 affect the speed? How do the GPU-enabled functions compare?

Like I say, all the cases we've tried 9 is faster (or, we've fixed 9 so it is...). If you find a case where Arc is faster, let us know and we'll make sure whatever is holding 9 back goes away so 9 is faster in that case as well. :-)

What's important about using .map, by the way, is not just processing. Display is faster also, as you can see for yourself right now, comparing Arc using the .img to Manifold using the same image in a .map project:

See a) how quickly Arc can open a saved project, b) how quickly Arc generates a first display of that project, c) how quickly Arc can pan and zoom to arbitrary locations within that image, d) how quickly Arc can re-project on the fly in real-world use, and e) how quickly Arc can save versions of the project. Is it as fast at all that as 9? Where is it faster and where is it slower?

for example, launch the .map with the 16 GB image in it, open the particular sample image in 9 an image window. Zoom to Fit. Draw a very small zoom box over Seattle. Zoom to fit. Next, in turn draw very small zoom boxes into Miami, Boston, San Diego and Kansas City, zooming to fit in between. It takes a tenth of a second to redisplay any of those with 9. How long with Arc? Can you really zoom in anywhere and absolutely instantaneously pan and zoom however you like? I expect ESRI products to be fast at display, but some, including some of their latest, are far slower than 9 at such display.

Next, create a map and drop a Bing streets layer into it. Zoom into the view showing the US in your map. Drag and drop the 16 GB image into it. Next, create a Google streets transparent layer and drag and drop that into the map above the 16 GB image. You now have three layers.

The 16 GB image is in Albers conic. To show it in a map that uses Pseudo-Mercator requires a re-projection on the fly with every pan and zoom.

With Manifold you can pan and zoom instantaneously in that ensemble, even considering that Manifold is re-projecting on the fly that 16 GB image from Albers conic into Pseudo Mercator. Is the display of all three layers and the re-projection of the 16 GB image likewise instantaneous in Arc? Can you pan and zoom with only 1/10th of a second to redraw any display? Try it in ArcGIS Pro.

Create some locations so you have some extra stuff in the project. Add a vector layer to overlay the other layers. Display all that, panning and zoom box to various locations to give all the data a real workout. Save the project. How long to save? How long does it take the equivalent Arc workflow to add the vector layer, reproject on the fly to display, and to save the Arc project? Where is it faster or slower?

Those are all very important real world questions, to avoid making Farmer Andy mistakes. My experience is that 9 is dramatically faster than, say, ArcGIS Pro in the above, but I'd like to hear reports from others. I'd bet that anybody who actually tries such comparisons isn't going to praise the Farmer Andy lifestyle.

Let me close by emphasizing that if you really do want a one-time glance at big data I agree it is convenient to have a Farmer Andy machine. Quick hookup is not a mistake for a one-time user, it is the mission. Sometimes you don't know what the big data is and you need to look at several big data sets knowing full well most won't be right and won't ever be used.

I agree that in such cases it is wonderful to have a quick viewing capability, and that whatever can be done in Manifold to provide such a quick look should be done, so long as that does not end up with lesser performance in other cases, as currently happens with Arc. Until that happens, there's nothing wrong with using some other tool for a quick look but using Manifold for day to day superfast operations. It's good to have that option.

adamw

8,037 post(s)
#27-Jul-18 09:37

As far as I can see, the issue is that ArcGIS and QGIS can render intermediate levels right from IMG (from the accompanying RRD) while 9 cannot do so and has to create them from the base level.

The performance of creating intermediate levels from the base level seems fine: processing 16 GB of original pixels in ~1200 sec is 13+ MB/sec, and given that the processing consists of reading from the original storage, writing to a new storage plus computing intermediate levels and writing them as well, this is quite workable. The issue is that there shouldn't perhaps be any of that processing if intermediate levels are already available in the file.

We'll look into supporting intermediate levels for IMG (ERDAS). Things like that are no big deal to add. If there's any other format that has intermediate levels which we do not currently support, let us know. In general, when we implement support for a particular format we tend to support everything that we can reasonably get out of it, but we might sometimes be lacking documentation for a specific format feature plus formats sometimes get new features later on plus we sometimes get new features which the dataport could use to better represent the data as well, so this is always a moving target. We are totally prepared to add things that we don't support but could.

Thanks for the report.

Dimitri


4,941 post(s)
#27-Jul-18 10:14

We'll look into supporting intermediate levels for IMG (ERDAS).

That would be a welcome advance for a quick, one-time look when linking the file. Should be done, of course, so when you want to take a quick look at something you can do that in Manifold.

But for maximum performance in processing you still want to import the image into .map and then thereafter use it only in .map.

If we have both, of course, then both Farmer Andy and Farmer Mike are happy. :-)

[Which reminds me... would be useful to have an unlink. Link to take a quick look at many big IMG files and then when you see the one you want you can command an unlink, that is, importing it.]

An edit: Ooops... I forgot we don't need an "unlink" since that's what Copy and Paste is for: Copy the linked thing and Paste it into the project and that is the same (after the usual chugging associated with any big import) as an import.

Dimitri


4,941 post(s)
#23-Sep-17 08:28

I have plenty of RAM

No, not unless what you reported in the post is inaccurate:

RAM 16309 Mbytes

That's not plenty of RAM for a system that is designed to exploit lots of cheap RAM. Besides the Performance Tips topic, the Using RAM and Other Machine Resources link recommended in Performance Tips discusses how Release 8 architecture exploits cheap RAM.

If you really want to work with images that are tens of GB in size you'll need to max out RAM and have very fast hard disks with Release 8. It still will be slow since 8 was not designed as a display engine for large images. 8 was designed for a more balanced GIS experience with a focus on vectors, data analytics and manipulation and what now are considered mid-sized, not large, images.

Like GM, QGIS opened the file instantly.

Of course. Q is a different system, originally designed as a viewer of large images so it does that well. Q was based on US government-developed code for a viewer, released into the public domain. That's the prime reason why it works well as a viewer of images. Nothing wrong with that.

But for all that, a design choice that makes for good internals for a viewer of rasters is also a design choice that trades off capabilities in other areas. Q cuts a lot of engineering corners when it comes to data and vectors, a good example being the use of shapefiles (!) as a native format. Shapefiles may have been a fine choice back in the day when GIS people could not spell "DBMS" even after saying it a few dozen times, but in modern times is its really goofy to use a format designed in 1973 as a native data store. DBF, the data store component of shapefiles, is indeed that old. Shapefile format is a terrible choice for modern times when the data aspect of GIS has become primary, perhaps the most valuable aspect of GIS, and only the unsophisticated think of GIS as a more automated set of crayons for producing visuals.

There are many good GIS products out there: Q is one of them, as is Arc, Global Mapper, GRASS, MapInfo, Release 8 and others. They all are a bit different and have different emphasis. Most have some things they do extraordinarily well, sort of a signature high point that other products do not do as well, and most have some things they do less well where other products will shine.

8 has genuinely exceptional balance. If you look at all of the "old-style", that is non-parallel, GIS packages you can't find another one that has better overall balance and the wonderful integration 8 has. What you find instead tends to be packages that are put together of different pieces that require considerably more effort to achieve what is easier to do in 8, and usually (but not always) without the incredible depth and sophistication of 8, and coupled with the sheer reliability and quality of 8.

Quite often, the performance of 8 is also exceptional in those regimes which it targets. So, no, if you want to look at images that are tens of gigabytes in size 8 is not the best choice. It was never intended for that since a choice of architectures that makes that go well is a choice of architectures that gives up the seamless balance for vectors, data, and rasters in sizes where most people work. But the choice of a balance point often means that for vector operations and certain types of analytics 8 will run dozens, or even hundreds of times faster than MapInfo, Arc, or Q. Like any tool, know your tool and know when it is the right tool to apply.

I looked at the user manual for Radian Studio... it doesn't say it will open IMG files...

Well, I have to say it warms the heart you see you referred to the user manual. So few people do that it is very pleasant to see a willingness to leverage documentation. But, as cool as that is I had hoped to spare you the need to read documentation.

I recommended that you use Viewer to save you the hassle of needing to read manuals. :-) If it is quicker to just "do it" without needing to think, I say "just do it."

Seriously, that is part of the mission of the new, parallel technology. A core part of user-friendliness is giving people the ability to just do what they want without having to figure out how to get around the limitations of software. For example, with 8 you get great balance in the range for which it was designed, but at the cost of having to think in advance and plan for work outside that range, such as with multi-gigabyte images. With Arc and Q you get endless trinkets, but at the cost of fear at encountering instability with bigger vectors.

When parallel Radian technology allows work with super-huge images and huge vectors, all those worries go away. You just open it and use it.

A good example is the Australian rivers data set seen in the YouTube video at https://www.youtube.com/watch?v=R1G11KtJ2EU

You read all sorts of threads on general forums like reddit's r/GIS about people saying things like "oh, I need to work with this data but it's so big it just crashes Arc or Q" or, in the case of the famous Australian rivers data set that enjoyed 15 minutes of worldwide fame outside of GIS when a guy started printing posters of it and selling them to the public, well, his comment reported in the tech press was that the biggest problem in doing the project was he was using Q and it kept crashing and locking up all the time on so much data.

The Radian approach to that is "Big data? No problem. No need to worry or plan way in advance. Just open it and run with it."

In all fairness, 8 would be slower with that data too. It wouldn't crash or freeze up, but it would not have the superb speed provided by Radian.

Onward...

A useful list of formats is on the Radian Data Sources web page. IMG is in there.

The Radian manual does not cover all the many formats since most are self-explanatory. Over time more and more will be added but virtually nobody reads that stuff since usually it is just boilerplate: "Ah, yeah... this is like the last hundred formats discussed... click File - Open and double-click on the file..."

To see all the formats, it is much easier in a world where nobody reads documentation but just prefers to download stuff and poke at buttons is to... Download Viewer and Poke at Viewer Buttons! :-) That's one reason Manifold issued Viewer. Not a main reason, but a legitimate one.

Viewer is free. Download the Future version and open the file.

Heck, if you love the way Q does it you can do that in Radian/Viewer as well: Radian includes a dataport for GDAL so if you prefer the GDAL importer (which is what Q uses), you can use that for Radian also.

Like I wrote before, if you want to work with big images Manifold has a solution for you. Viewing them and doing much work with them is even free, in the form of Viewer. Enjoy!

Ajayk3 post(s)
#25-Sep-17 06:14

Thanks again for you comments Dmitri, I appreciate your comments about the 8 system being optimized to for a wide range of tasks. Confession... I'm not an expert user, but you've probably guessed that. Just reading the link on using RAM etc, I have SSD so didn't think there would be much to gain speed-wise by adding RAM...

"At such low prices it is simply criminal not to have at least 4 GB of RAM. Manifold therefore expectsthat we will have lots of RAM in our machine. Occasionally, one reads criticism of Manifold that the system uses a lot of memory. If the memory is available, of course it does! It would be irresponsible not to do so. By using RAM whenever possible Manifold assures that it will run very fast. You must, of course, run 64-bit Windows to take advantage of lots of inexpensive memory."

Also Is that up to date ? 4GB ? I noticed in Task Mgr that Manifold was only using 8GB RAM while the image loading was happening, with very liitle else loaded other than Win10 OS... I assumed that was some kind of cap in what 8 will address...

I didn't go to Viewer because I wasn't primarily interested in viewing. I want to be able to process and save. When I saw the documentation says it doesn't open the IMG format, I thought I could better spend my time with Q. (esp since I can process, save and export from there). Although 8 was slow to open the surface, I was able to easily zoom in and around to view once it was open.

My main task is to select areas which have selected land cover (raster, bil) and soil type (vector, shp) in 12 States of USA (vector,shp). I'm starting by first subtracting the states I don't want, to reduce file size. I'd really appreciate any tips on how to do that efficiently in 8 using large files.

I will look at Viewer shortly; if I find I can process what I want from there, I'll think further about buying Radian (and maybe more RAM).

Dimitri


4,941 post(s)
#25-Sep-17 08:07

I have SSD so didn't think there would be much to gain speed-wise by adding RAM

Not so. It never fails to amaze me that with no irony whatsoever people will begin a comment with "I'm not an expert..." but then go on to discount the advice of experts. :-) Human nature, I suppose.

SSD is faster than rotating memory but slower than system RAM. There is no incompatibility whatsoever with the advice to max out your system's RAM and to also use fast disk drives. SSD is just a very fast disk.

At such low prices it is simply criminal not to have at least 4 GB of RAM

Also Is that up to date ? 4GB ?

Of course. It is simply criminal not to have at least 4 GB of RAM. That comment was exactly correct back when it was written and it is still very accurate today. Manifold is used all over the world where there are, indeed, many people who think of 2GB as perfectly ample. They also run 32 bit Windows XP.

The advice in that topic was written a long time ago but it is still perfectly correct: If you want to work with big images or big data in Release 8, install plenty of RAM. It is absurdly cheap so there really isn't much of an excuse not to max out the amount of RAM you can put into your motherboard. If all it can handle is 32GB, install 32GB. If it can handle 64GB, install 64GB. If you can afford an SSD it's "penny wise and pound foolish" to spend for an SSD but to fail to max out the RAM in your system.

I'd really appreciate any tips on how to do that efficiently in 8 using large files.

Switch to Radian, specifically Future. That's why Manifold created that technology. There is no finer tool for doing spatial engineering with larger data.

I thought I could better spend my time with Q. (esp since I can process, save and export from there).

So, why don't you just use Q? There has to be a reason why not, or else you would have just done it already, true?

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.