Subscribe to this thread
Home - Cutting Edge / All posts - File size
orerockon

351 post(s)
#10-Oct-17 01:37

Just having downloaded future and knowing nothing beyond clicking around and parousing the videos, I opened a small map file (4 megs) and saved it. As advertised it opens more or less instantly but the size has ballooned to 53 megs. Does the additional space store information so it can open instantly or is it now an "uncompressed" map?

Dimitri


4,193 post(s)
#10-Oct-17 07:18

If you opened a Radian format .map file and then, without doing anything else, immediately save it, it saves as the same size.

Usually when people are surprised to see a .map they save is bigger than what they started with, one of two things has happened:

1. The .map includes an imageserver which has persistent cache turned on so they end up saving lots of cache tiles. See the "Cache" section of the web server topic.

2. The "without doing anything else" does not apply, because they did something else (import, etc.).

tjhb

7,452 post(s)
#10-Oct-17 07:35

There are benefits in the Radian storage model which have an obvious cost in file size.

But raw file size simply does not matter now. Local storage is trivially cheap, it is not a factor. Only access speed matters.

For transmission we should still compress. 7-zip compression seems by far the best.

orerockon

351 post(s)
#10-Oct-17 16:48

I should have been clearer when I say opened a small map it was created and saved in M8, then opened and saved in Radian. And the file has a few hundred very small drawings (100m lines), a couple scripts, a table and a map. Just surprised me a bit. And a USB can only hold so much lol.

artlembo


2,835 post(s)
#10-Oct-17 19:16

I agree. Some of the file sizes I am seeing are explosive - in the many gigabytes. My large Mfd8 files, now saved in 9, are getting to the point that a 250GB hard drive is starting to feel it, and a jump drive can no longer hold it.

As Tim says, there are reasons for this, and it likely improves performance in 9, so I am willing to live with it. However, I think it is worthwhile to think about what these sizes are.

Dimitri


4,193 post(s)
#10-Oct-17 19:36

Was your Release 8 .map file stored with compression?

orerockon

351 post(s)
#11-Oct-17 18:23

Yes mine was. M8 attached. Try it on your end, the MF file is too big to attach (19968 Kb).

Attachments:
Centerlines tmp1.map

Dimitri


4,193 post(s)
#11-Oct-17 20:22

When a Release 8 map is compressed, the file size gets reduced. You can see a similar effect by taking a Radian .map and zipping it. A good example is the Wildfires.zip file I've attached, which is 10 KB as a zipfile. Unzip it into a .map and it grows to 1 MB, 100 times bigger. That's a super extreme example, of course, as zipping a Radian .map won't normally make it 100 times smaller.

Release 8 can be set to automatically compress map files because the technology within 8 is not capable of accessing the innards of a .map all that much faster than a good unzip algorithm can decompress it, so it is not a bad tradeoff. But try to open a very large image in 8 and you're going to think right away that "heck, rather have a way bigger file if that makes it go faster."

For an extreme example of that effect, watch the video at https://www.youtube.com/watch?v=Gvib5LqqE2o

There is simply no way Release 8 (or, for that matter, most other applications) could handle even 20 GB worth of images as fast as that video works with 110 GB worth of images. With Release 8 I get the feeling each step (pan, zoom...) would be the sort of thing that you would launch at the end of the day so it could cook overnight. With Radian it is instant. You do pay for that instant speed with larger file sizes, but, heck, those are cheap.

But you get other benefits with those as well. With Release 8 if you make a small change in a big vector file it is really painful to save that file so you don't have to worry about losing the change. With Radian / Future / 9 if you make a small change the save, just like the open, is instantaneous.

Newegg.com sells plenty of 4 TB hard disks for $119 and you can buy 8 TB for around $250. Storage has become very inexpensive, but the cost of your time keeps going up, so if a larger file size makes GIS significantly faster for you, well, that's not a bad trade off.

By the way, in terms of zip drives, it is way cheaper to buy one of the new, slim 1 TB portable external hard disks for around $55 than it is to buy a USB flash memory zip drive.

Attachments:
Wildfires.zip

orerockon

351 post(s)
#11-Oct-17 21:14

I do have a 500 gig portable and I like the convenience, but there are certain disadvantages as compared to a usb or hdsc. (Ill leave that up to your imagination lol). Seems like a nice option would be to zip or not to zip, without going into options...miscellaneous or whatever it will be called. Maybe as save as... As you pointed out there can be a large, marginal, or substantially no benefit to saving uncompressed.

Dimitri


4,193 post(s)
#12-Oct-17 07:47

Ah, but the minute you zip the file you kill Radian speed. One might think, "oh, that's OK, I'll just get in the habit of unzipping a file whenever I want to use it, or... I won't mind if whenever I open a file I have to wait 20 seconds for Future to unzip it in background..." but that doesn't work for many reasons, four of which immediately come to mind:

1. Zipping takes longer than unzipping. Are you really going to wait around 20 seconds every time you want to save some small change to a project?

2. People get addicted to the instant open and the instant save. Waiting for 20 seconds, or even only 10 seconds, for big data to unzip becomes intolerable. [Data that is so small it unzips quickly you don't care about being stored uncompressed...]

3. A key part of using Radian technology is making nested storage a part of your lifestyle. You cannot beat keeping vast, endless archives of data at your fingertips in the form of projects that can be nested one within another without any performance hit. Start zipping files and you lose that speed and organizational convenience. All that archiving goes away because at each level you get a massive, cascading performance hit.

4. Once you get used to the nested, archival storage lifestyle you end up doing far less duplication of data, because what you want can be nested into a project without needing to duplicate the same data for each project. If you look at the routine duplication of data people do who use systems that do not provide zero performance hit access to vast archives, well, it might just be they end up spending more space on disk for all that duplication despite having smaller file sizes. I grant that the lazy user in all of us may not take advantage of such organization, but well-run shops will insist on it, even if for no other purpose than to avoid version skew.

I say get with the program and think in terms of 8 TB disks. TB are cheap but your time is valuable. Think TB, not GB. :-)

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.