Subscribe to this thread
Home - General / All posts - Making images from layouts - running for hours
Ian
181 post(s)
#29-Nov-17 00:24

I am trying to make images of layouts and I have left the computer running for 36 hours and it still can't do it. The layouts are A0 size and cover regions of New Zealand but don't include any imagery layer. I have 16 of these to do they started off not to bad taking an hour or two at 150dpi. They progressively took longer so I dropped to 96dpi but this didn't seem to make any difference. The last one I cancelled the operation after 36 hours. I am making images as I want to export he files as jpegs so they can be uploaded onto a clients website. Has anyone go any ideas of what the problem is or how to best achieve what I trying to do? Thanks

mikedufty

777 post(s)
#29-Nov-17 01:18

Have you got enough free hard disk space on the relevant drive. I suspect the images are too big to fit in memory and I think manifold leaves all the memory management up to windows, does seem to struggle with really large files if you don't have lots of memory and disk space.

Ian
181 post(s)
#29-Nov-17 01:43

Have plenty of disc space - hundreds of GB. The images aren't that huge - from memory around 40MB at 96dpi and 98MB at 150dpi.

atomek

421 post(s)
#29-Nov-17 16:58

Having similar issues with large, repetitive tasks like importing huge shapefiles, merging surfaces or generating images from map components, I'm always observing ram memory consumtion in windows task manager. When it comes close to max it either slows down terribly, stalls or just crashes. Learning from experience, I tend to do a batch or two, (save if necessary, as with import operations), close the Manifold .map file, re-open, have a go with another batch and so on. Good luck.

Dimitri


4,332 post(s)
#29-Nov-17 05:58

What version of Manifold are you running?

Ian
181 post(s)
#29-Nov-17 09:15

Manifold 8. Everything is update as far as I know as well

Dimitri


4,332 post(s)
#29-Nov-17 19:11

Thinking outside the box a bit...

I am making images as I want to export he files as jpegs so they can be uploaded onto a clients website.

What's the purpose of the jpegs? Would a large A0 sized PDF with hires image do? If so, use Future and print the layout to PDF.

atomek

421 post(s)
#29-Nov-17 20:12

Good idea. Then you can turn the pdf into a jpg with e.g. X-Change Viewer

antoniocarlos

471 post(s)
#29-Nov-17 20:39

I don't think Future fully supports importing M8 layouts yet.


How soon?

Ian
181 post(s)
#29-Nov-17 20:58

I am not familiar with Future or a PDF with hires image. I have already exported the layouts as PDFs but they are over 220MB each. Can anyone give me any ideas as to why I am having the problem creating the image in the first place?

antoniocarlos

471 post(s)
#29-Nov-17 21:28

Can you give a bit more information about how your are going about producing the pdfs? What printer dirver are you using if any? If you are exporting to PDF what settings? If F6 (make image) not an option?.

Here is an example of a PDF (zipped) of a map with a background image is is 42wx21h inches. 42 Megabytes. The Manifold 8 map files is about 110 MB.

http://www.geoinformaticapr.com/uploads/8/1/7/3/81739222/pr_100_in_x_40_in.zip

ACGT


How soon?

Ian
181 post(s)
#29-Nov-17 21:33

I produced the PDFs for printing but they are to large to upload onto the web site so I want to make the jpegs. I started using the Tools - Make an image and the first few worked ok but they progressively got slower until it was taking hours and it was crashing Manifold when I would open another project to work on while the create image was working in the background.

KlausDE
6,054 post(s)
#29-Nov-17 23:08

On PDF-export choose options and set resolution to a reasonable value. Mfd8 always creates rather large pdf-files that adobe professional or other packages can reduce in a second step.

antoniocarlos

471 post(s)
#29-Nov-17 23:12

It should not take 36 hours to create a PDF from a Layout. There is something wrong with the process.


How soon?

Ian
181 post(s)
#30-Nov-17 00:03

Yes I agree, I have no idea what it is though and am hoping someone has some suggestions on what's going on and how to address it

antoniocarlos

471 post(s)
#30-Nov-17 00:20

OK. So. Please describe the process in detail. Tell us the system configuration and what Manifold 8 Build you area running. Start from the development of the .map files. Don't leave anything out. What size are the Map Files? Are all the data part of the map file? Are all the data in the same projection? Can you share one of them through a shared drive? Here to help.

ACGT


How soon?

Ian
181 post(s)
#30-Nov-17 02:51

I am running the latest version on Manifold 8 - 8.0.30.0, 64 bit, personal edition. I have it installed on two computers which are both having the same problem. The newest computer has a 250gb solid state drive (Manifold is installed on the ssd) and then over a TB on the hard drive. 32GB of ram (or it maybe 16, working off my older computer at home at the moment and can't remember). The maps are created using two data sets NZLRI (NZ land resource inventory) and LUCAS (a land cover data set) both in the same NZTM projection. Queries were then run to analyse the data for land cover by land class (not sure if you want all of the steps for this) this data was then split into the 16 regions of NZ and shown in a table. The resulting data set for the whole country was then displayed in one layer. I then had to produce a map for each region (16 maps). To do this I used the layer with the regional boundaries and shaded out all of the regions but the one I was producing a map for. Normally I would have made 16 maps but in this case they were so big I used one map with 16 layers, each having one region visible and the rest shaded out. I then created 16 layouts based on the one map and would turn on whichever shading layer I wanted to make the maps. Once I hade made a layout I would export it as a PDF. I then went back to make the jpegs. Open each layout go tools - make an image and put in 150dpi and let it do its thing. The first few were okay then they just got real slow. The whole project file is just over 4.2GB. I imagine i have missed out some info but it is a starting point anyway. I have upload one of the small jpegs which was done at 96dpi

Attachments:
West Coast LO Image.jpg

ColinD


1,821 post(s)
#30-Nov-17 05:36

You can check if memory is getting low by using task manager and also keep checking your Temp file size. Is the Temp file on your ssd or hdd? Maybe that is too small and filling up?


Aussie Nature Shots

Ian
181 post(s)
#30-Nov-17 10:57

I am looking on my old computer now which has much lower specs and space than my new one, I have manifold open and am trying to make an image. There is plenty of space in the temp files, from task manager cpu is running at around 60%, memory at 50% and disc mostly 0-3%.

Dimitri


4,332 post(s)
#30-Nov-17 11:55

Key questions:

On what disk volume is your TEMP path folder?

What is the total free space on that disk volume?

On what disk volume is your Windows page file?

What is the size specified for your Windows page file in Windows?

What is the total free space on the disk volume used by the page file?

The whole project file is just over 4.2GB.

Do you have your Release 8 tools options set to use compression for the .map file? If so, the uncompressed form of the .map file could be 5 or 10 times larger... 42 GB, maybe.

Release 8 is slow with very large images. That is well-known, and was one of the motivating factors for creating Radian. But even accounting for that it is easy not to realize that you are commanding 8 to create an image that, uncompressed, could be 50 GB in size. For Release 8, working with a 50 GB image is at least an overnight deal.

The large amounts of free space required by the Windows pagefile and TEMP storage can surprise people. See the Memory Requirements topic. The "Working with Large Images or Surfaces" and "Performance Tips" topics may also be useful.

Ian
181 post(s)
#30-Nov-17 19:28

I am not sure how to find the answer to the key questions, have done a search but can't find much on them. I have 141GB free on my ssd and 1.4TB free on my hard drive so that seems like plenty to me and shouldn't be restricting temp file volume - unless there is a some limit put on it that I don't know about?

antoniocarlos

471 post(s)
#01-Dec-17 00:19

Here is a thought, not sure if it will help.

I would assume that the .map file is too big and make 16 individual map files (not sure if you did this), one for each region. I would clip out any data within these map files that is not going to be printed out (JPG or PDF). I would probably also run a normalize topology with a large precision parameter (500 meters?) on both the land use and land cover layers to make their memory footprints smaller.

ACGT


How soon?

Dimitri


4,332 post(s)
#01-Dec-17 07:11

unless there is a some limit put on it that I don't know about?

How would you know if you didn't look? :-)

I am not sure how to find the answer to the key questions

Easy. Visit www.google.com ... if you are travelling, visit www.google.com/ncr so if you are changing planes in some foreign country google won't redirect you to a local country page in a foreign language.

In the big box in the center of the page under the big Google caption enter a phrase for which you would like to find information. Example:

windows page file

or...

windows pagefile

... and press Enter. Google will provide numerous pages that explain everything you might want to know about what a windows pagefile is, how it is used, how it is configured, how to tell if some maximum size has been set and so on.

There is a seriously inconvenient catch to the above "google" process: you have to read the pages it recommends to get the benefit of their information content. That's annoying, for sure, but until Google improves their system to eliminate that requirement, it is what it is.

You can apply the same process to learn how to get the information you need to answer the other key questions.

So... if your pagefile is set to an upper limit of 50 GB and you are trying to do something that requires 200 GB that's a problem.

It also helps if you would actually read the links I recommended. There you will learn that you should have four times as much free space as you think you might need. 4 * 50 = 200. 200 is more than 141, if your TEMP path points to a folder on your SSD. Possibly another problem. Easy to solve: point your TEMP variable to a folder on your hard disk.

The above is routine Windows admin stuff, like not running out of disk space or pagefile space for big jobs. If you want to run bigger jobs you have to pay closer attention to those factors.

Ian
181 post(s)
#01-Dec-17 11:01

You are a bit quick on your assumptions Dimitri. Would seem to me you can't have lived too much if you think its cool to talk down to people and treat them like idiots, reveals quite a bit about you though. I have been working through the issues that you and others have highlighted. Some of it I understand and am familiar with while other bits are new to me. I have got to the point of possibly needing to to change my temp folder to my hard disc but I was going to talk to the company tech guy who set my computer up before I do that in case I screw something up - computers aren't my specialty so I'd would rather check first than have to try and fix a mess later. If that is the problem why would I be able to create the images to begin with and not now - specially as I have cleaned up my temp folder and its still not working.

Dimitri


4,332 post(s)
#01-Dec-17 13:40

if you think its cool to talk down to people and treat them like idiots

Oh, I wasn't treating you like an idiot. I was being sarcastic, the intent being to tweak someone I assume knows perfectly well how to do a search for failing to take advantage of that.

Consider this:

how to find the answer to the key questions, have done a search but can't find much on them.

I just don't buy it that somebody smart enough to be doing GIS at any level doesn't know how to use Google to learn how to find out what size their pagefile is set to. Sorry. Not possible. Whatever you enter into Google, be it "page file", "pagefile" or even

What is the size specified for your Windows page file in Windows?

... you get zillions of hits and pages providing wonderful, clearly explained, super-helpful, step by step instructions on the very first page of Google results.

Now, I agree that sarcasm is not always well-met. I might have written "Now, wait a minute, you did a search but you couldn't find much on, say, how to see what size your pagefile is set to?.... how did you search?"

If you don't feel comfortable setting the size of the pagefile, no problem. I just asked some basic questions, the answers to which must be known as part of doing big projects on Windows. If you answered those questions you could move on to the next step.

This stuff is complicated given that it involves fundamental limitations of both Windows and Release 8 and it is easy to step over those limits, even when everything is working correctly. It's best to check off all the "must know" basics for simple causes before moving on to more complex ones.

tjhb

7,545 post(s)
#01-Dec-17 21:24

Was that an apology? Hard to tell.

Dimitri


4,332 post(s)
#02-Dec-17 04:31

No, I don't apologize for mild sarcasm. That would be an insult, right up there with suggesting somebody is too fragile to be told to RTFM.

Instead, I prefer the implied compliment inherent in... "Look, we both know you are a smart guy who can easily handle this if you read up on what you don't know. "

adamw


7,307 post(s)
#01-Dec-17 11:19

I then went back to make the jpegs. Open each layout go tools - make an image and put in 150dpi and let it do its thing. The first few were okay then they just got real slow.

Is it that making an image becomes slower the more layouts you go through in the same session of Manifold or is it that some specific layouts are fast and other specific layouts are slow?

If it is the former, that looks like a memory leak (a bug in Manifold or in one of the system components that the process stresses) or some type of accumulating fragmentation (not a bug, but also unpleasant), and you can solve it by relaunching Manifold and continuing from where you stopped previously. (I assume the process is: take a layout, make an image, export the image, delete the image, move to the next layout. If you are not deleting the generated images, it is better to move them to separate MAP files - ie, via copy / paste.)

If it is the latter, fine, we then need to figure out what it is about the layouts that are slower than others, there should be some difference from those that are fast.

antoniocarlos

471 post(s)
#01-Dec-17 12:22

I like Adam's recommendations better than mine.


How soon?

Ian
181 post(s)
#01-Dec-17 17:49

The first time around it was in the same session. I got through a few images (some were slower than others but I assumed that was just as they covered slightly different areas and were different sizes) but the overall process was taking a lot longer the more I did. I thought maybe the images in the project were slowing things down so I deleted them and carried on. Didn't make any difference though. I had another manifold project open that I was working on at the time and it started to take ages to do things in that one go slow - not respond. So I closed the image making project. Yesterday I cleaned up my temp folder increased my page file size to a minimum of 70 gb maximum of 100gb - was set at 50gb - not sure why it only had 50 listed as page file size but when I changed it to custom I had to put minimum and maximum - yes I could have done a Google search but as I mentioned I'm trying to sort all this out while still working and meeting deadlines. I then opened the project and tried to make a 150dpi image of one of the layouts and it ran for over 6hrs and still couldn't do it. Was wondering then about redirecting manifold to the temp folder on my hard discs as there's so much more space on there but when monitoring my storage space and with task manager open while i was trying to make the image, cpu capasity was only at 15% disc space used was at 1- 5% and memory was around 15%, so that didnt seem to be the issue.

adamw


7,307 post(s)
#02-Dec-17 10:42

So, it looks like the main problem is indeed related to a specific layout (or layouts). That is, the layout you tried to render last couldn't render in 6 hours, while the layouts before it rendered far faster.

What's different between the layout that can't render in 6 hours and the layouts that render faster? If they are simply showing different areas of the same component, can it be that the "fast" layouts were showing areas near the border and were half-empty?

I realize every experiment is taking you a long time because the operations are slow. It is rather unfortunate that I cannot yet suggest you to simply open the MAP file in Future and render from Future (which should be much faster) - we are pretty close to the point where that suggestion would work, but we aren't yet there.

Here is something that might help right now. Try rendering one of the "fast" layouts and the "slow" layout with, say, 60 DPI and measure the times. Then take the time to render the "fast" layout with the DPI that you want and that will allow you to produce an initial estimate of approximately how long it should take the "slow" layout to render at the same DPI before accounting for various overflows and edge cases. If that estimate is going to be, say, 2 hours, then it's not useful because you already waited more than that, but if it is, say, 8 hours, it will indicate that pausing after 6 hours was too early, you should have waited for 2 more hours.

Also, you can contact tech support, report the slowness and offer to upload the MAP file. Even if that's not a bug, they might suggest something useful after taking a look at the data (like: unselect all pixels / change the projection on this component to that, etc).

Ian
181 post(s)
#03-Dec-17 21:37

Thanks Adam. You were correct about the different areas of the same component and there being more blank space with the quicker ones. I did what you suggested to get an idea of the time. One of the quick ones took less than 10 minutes where as one of the big ones was still running at an hour 10 minutes. Had to pull the pin as I couldn't use any other projects while it was running. Will try again (with a high dpi) at the end of the day and leave it running over night at the office, hopefully 15hrs will be long enough. May try the tech support option - not sure if I have any tokens left though. So from where I sit it seems like just the size of the file that is causing the problems, and the way I have set the project up. In the future I would set things up a little different to try and reduce the size of each area to be printed.

adamw


7,307 post(s)
#04-Dec-17 07:20

You may send without tokens - just say that you are reporting a potential bug / area for (big) improvement and that you do not require a response, plus offer to upload example data (or archive the MAP file with a password, upload it to one of the file sharing services and include a link to it in the email).

Dimitri


4,332 post(s)
#04-Dec-17 08:24

You may send without tokens - just say that you are reporting a potential bug / area for (big) improvement and that you do not require a response

Bug reports are sent directly to tech support and do not require any tokens if you do not ask for a response. Send suggestions to sales, also no token required if you don't need a response.

If you require a response or have any questions, that's cool, but that makes it a support request and requires a token.

If a crash happens that's a bug report:

I started using the Tools - Make an image and the first few worked ok but they progressively got slower until it was taking hours and it was crashing Manifold when I would open another project to work on while the create image was working in the background.

Release 8 is not guaranteed crashproof if you launch it with insufficient Windows resources, but crashes are interesting enough that tech support will ask for file upload. They'll usually report back what they find even if you don't ask for a request.

By the way, Radian, in contrast is designed to be crashproof even if launched with insufficient Windows resources.

Ian
181 post(s)
#04-Dec-17 10:18

Thanks, will do that when I get a minute, will be interested in what they find. I have Radian and had a bit of a play around with it, but I am short on time at the moment so haven't given it a good go. Tim B helped me with project i am having issues with and we were going to compare the analysis in Radian but again I was short on time so we didn't get that far. From what I have seen though it doesn't seem to have the same options/capabilites for doing map layouts - I could be wrong on this as I haven't had a really good look. I use map layouts and PDF and Jpeg exports for reports and stand alone products a lot so it has held me back from moving over to Radian, though I do like what I have seen and it does seem faster.

mikedufty

777 post(s)
#04-Dec-17 15:05

In terms of getting your job done, rather than fixing issues with Manifold, have you looked at Klaus's post above about PDF compression?

If you have satisfactory PDFs that are just too big, you can massively reduce the size by enabling jpeg compression of the images in them (I think manifold uses no compression or zip compression).

I know adobe acrobat can do it, I usually use Foxit Phantom PDF, which is a lot cheaper. Not sure if there are free alternatives, but I suspect there are.

Dimitri


4,332 post(s)
#04-Dec-17 15:15

it doesn't seem to have the same options/capabilites for doing map layouts

Layouts are way more robust and predictable in Future than in 8, and they are much easier to use. They have fewer features, as yet, but that will change soon. The main reason to use them now is Radian's far greater capacity and speed.

Printing using print-to-PDF is also a very good thing in Future. Using a good print-to-PDF driver in 8 as mikedufty recommends is not a bad idea, but that's going to be way more robust in Future.

mikedufty

777 post(s)
#05-Dec-17 07:10

I'm not suggesting a print to pdf driver in 8. Use the save to pdf from Manifold, then open the resulting pdf and compress it with suitable pdf software. Results of printing in M8 (to PDF or other) tend to be pretty bad, i my experience, especially if there is any transparency.

Ian
181 post(s)
#05-Dec-17 09:06

Don't currently have any suitable pdf software to do that, will look into it. Making the image certainly isn't working, still no joy after 16hrs.

I downloaded Future (have Radian already installed) but couldn't install it. Before I go to the Tech guys thought I would check here first. Have followed the install instructions and installed the extra software needed have right clicked on bin64 and run as administrator and get unknown error message. Have tried following the steps to make sure I have full admin rights (currently running windows 10) but when I right click on the metro interface nothing comes up so I cant select run as admin. Any suggestions?

ColinD


1,821 post(s)
#05-Dec-17 10:17

Because Radian is installed you don't need to install any of the Future builds. Just unzip and run the 64-bit or 32-bit manifold.exe (found in the bin64 or bin folders)


Aussie Nature Shots

Dimitri


4,332 post(s)
#05-Dec-17 11:10

Any suggestions?

Manifold Future is a portable installation. In the download page section for Manifold Future, see this: "See the Future Tour Part 1video to learn how to download and use a portable installation."

Also... Read the Installations topic. The topic contains a section on Portable Installations.

Also... Portable installations are very popular for many purposes. A Google search will reveal many useful pages.

Ian
181 post(s)
#05-Dec-17 23:59

Have done all of that and followed the steps on the video still getting unkown error when I try double click on it. If i run as administrator does the same.

tjhb

7,545 post(s)
#06-Dec-17 00:22

Is it the latest build, 9.0.163.11? Beta builds do expire/time out.

Also, you may need to Unblock the download. Before extracting the contents, right-click on the .zip file, press Unblock. (This is a Windows mechanism which can be turned off.)

Ian
181 post(s)
#06-Dec-17 01:35

The only one for download is 9.0.163.10. Don't have an unblock option when I right click. Will try downloading again and give it another go

tjhb

7,545 post(s)
#06-Dec-17 01:56

9.0.163.11 is available from the Cutting Edge section of the forum.

Normally the latest build is also available from the downloads page on manifold.net, but this time they must have forgotten to update it.

And yes, 9.0.163.10 has already timed out (after less than 2 weeks). I think they normally last longer than that.

Dimitri


4,332 post(s)
#06-Dec-17 05:28

this time they must have forgotten to update it.

Yep. An oversight... now repaired. On timeouts for Cutting Edge builds: these are often aligned with the end of the month.

Ian
181 post(s)
#06-Dec-17 21:07

Downloaded and working now. Thanks

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.