Subscribe to this thread
Home - General / All posts - compression and queries in M8
vincent

1,695 post(s)
#06-Jul-18 21:33

Hi,

I'm running queries in M8. Sometimes I see in the progress bar : "Compressing Data" and "[*] Caching Data".

This compression is really slowing the execution of the queries when it is active.

The Map file compression is turned off is the global parameters. 11 Gb of RAM are still free when running (on a x64 machine).

Is there a way to avoid this slow down when running queries ?

Thank you.

tjhb
8,094 post(s)
online
#06-Jul-18 22:33

How do you know that the compression is not speeding things up?

It seems likely that that is its purpose. Of course it could be wrong. It might be based on false or outdated assumptions about hardware, for example. But it might be that the algorithm in use is faster on compressed data than on raw.

What does "compressing data" mean here? Hard to know. It might be removing redundant coordinates, or interior triangles, or...

It would be interesting to know what the query is doing.

vincent

1,695 post(s)
#09-Jul-18 15:38

Looks like M8 32bits is not compressing data. Processing time : less than 75 minutes.

M8 64 bits, compressing data : 147 minutes.

My script is based on many queries. All these queries are in one loop, that runs a thousand times or more :

1 - Inserts line into a target drawing upon a selection with a where clause;

2 - Inserts points in another drawing along a line from the target drawing from step 1;

3 - Updates a field in the points drawing;

4 - Add height to the points from a surface;

5 - Updates a field in the point drawing, using an inner join with its own table;

6 - Updates another field in the point drawing, using an inner join with its own table;

7 - Updates another field in the point drawing using a where clause;

8 - Updates another field in the point drawing using another where clause;

9 - Inserts points into another drawing (the target gets 5 millions points at the end);

10 - Deletes lines from the target at step 1;

11- Deletes points created at step 2;

vincent

1,695 post(s)
#10-Jul-18 19:58

Finally, M8 64 bits does not always compress while running queries. I cannot find what is triggering the data compression.

tjhb
8,094 post(s)
online
#10-Jul-18 22:49

I don't know, but it might be a situation where Tech could give a useful answer, given one or more support tokens, if you can formulate a tightly-focussed question. (And they might give permission to post the answer here.)

Following with interest, happy to test anything you can post or send by email.

If you're seeing consistently faster performance in 32-bit Manifold 8 than in 64-bit ("consistently" might be a stretch, you may not have much time to waste on repeated tests), then a crude but helpful approach might be to write a timestamp to the Manifold log after each step 1 to 11. Which step takes much longer in 64-bit? If just one, and always the same.

But who knows--asking Tech seems like a better start.

vincent

1,695 post(s)
#11-Jul-18 20:38

I don't know how much time it's gonna take, but I will test and, good idea, send a question to support.

My hint now is the following : Manifold starts compressing data when a .map file is reaching X size.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.