Subscribe to this thread
Home - General / All posts - Discussion Question 1 - file sizes
artlembo

3,072 post(s)
#27-Nov-17 14:17

Just wanted to fire off a quick question to think about (I have a few others). These aren't questions off the top of my head, but rather some things I've come across as I've been working with real world things. I'm beginning to assemble the soil data for the US. So far, I have NY, PA, and MD:

it is about 3M areas, but as you know, it has lots of vertices. This file is 22GB in size. The corresponding ESRI geodatabases that make up the individual States is 6GB.

I know that disk space is cheap, but there is a fairly hefty inflation factor for the data sizes.

Do you think there is a need to do something regarding the space that a .map file takes up (perhaps allowing compression like in 8, with the warning that if you want to work fast with really large data, you have to keep the .map uncompressed).

Or, maybe next year, a 4TB drive might actually only cost $50 so who cares? (I got my 4TB drive for $59 on a huge Best Buy sale two weeks ago).

Thoughts?

Attachments:
soils.jpg

Dimitri


4,899 post(s)
#27-Nov-17 14:27

Do you think there is a need to do something regarding the space that a .map file takes up (perhaps allowing compression like in 8, with the warning that if you want to work fast with really large data, you have to keep the .map uncompressed).

No. The correct solution is to always work faster with any data, big or small. Never ask people questions about doing things faster. Just do them faster.

I got my 4TB drive for $59

Yes, exactly.

We will be reducing map sizes. Right now they err on the large size. Just like there are many optimizations yet to be done with raw speed of the engine (SQL, etc.), there are plenty of optimizations that can be done to reduce .map size while still retaining speed and reliability. We'll do those.

Likewise, if somebody wants to archive data or interchange (sending through email, etc.) we'll introduce a highly compacted, form for archiving data and for facilitating migration / interchange between different Manifold applications and generations. You can zip a .map file now to reduce size but we can do much better than that for archival storage.

adamw


7,968 post(s)
#01-Dec-17 08:06

To add a bit more to what Dimitri is saying on file sizes.

9.0.163.11 adds support for compressed MAP files = MXB. With this, we have two formats: the format for everyday work which optimizes for speed and does not care much about space (MAP), and the format for archiving and exchanging data which optimizes in the reverse direction (MXB). We think this is the best way to go in the long term, because biggest optimizations for speed and for space mostly oppose each other.

Also, in the last weeks we did a big internal work aiming to analyze and optimize the use of space inside MAP files. We built a framework that allows MAP file structures to claim the space they are using along with additional metrics for it and we can now see the big picture much better. We identified several key areas for improvements to MAP file structures. We are going to put that into work after we release 9.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.