Subscribe to this thread
Home - General / All posts - compression and queries in M8
vincent

1,736 post(s)
#06-Jul-18 21:33

Hi,

I'm running queries in M8. Sometimes I see in the progress bar : "Compressing Data" and "[*] Caching Data".

This compression is really slowing the execution of the queries when it is active.

The Map file compression is turned off is the global parameters. 11 Gb of RAM are still free when running (on a x64 machine).

Is there a way to avoid this slow down when running queries ?

Thank you.

tjhb

8,480 post(s)
#06-Jul-18 22:33

How do you know that the compression is not speeding things up?

It seems likely that that is its purpose. Of course it could be wrong. It might be based on false or outdated assumptions about hardware, for example. But it might be that the algorithm in use is faster on compressed data than on raw.

What does "compressing data" mean here? Hard to know. It might be removing redundant coordinates, or interior triangles, or...

It would be interesting to know what the query is doing.

vincent

1,736 post(s)
#09-Jul-18 15:38

Looks like M8 32bits is not compressing data. Processing time : less than 75 minutes.

M8 64 bits, compressing data : 147 minutes.

My script is based on many queries. All these queries are in one loop, that runs a thousand times or more :

1 - Inserts line into a target drawing upon a selection with a where clause;

2 - Inserts points in another drawing along a line from the target drawing from step 1;

3 - Updates a field in the points drawing;

4 - Add height to the points from a surface;

5 - Updates a field in the point drawing, using an inner join with its own table;

6 - Updates another field in the point drawing, using an inner join with its own table;

7 - Updates another field in the point drawing using a where clause;

8 - Updates another field in the point drawing using another where clause;

9 - Inserts points into another drawing (the target gets 5 millions points at the end);

10 - Deletes lines from the target at step 1;

11- Deletes points created at step 2;

vincent

1,736 post(s)
#10-Jul-18 19:58

Finally, M8 64 bits does not always compress while running queries. I cannot find what is triggering the data compression.

tjhb

8,480 post(s)
#10-Jul-18 22:49

I don't know, but it might be a situation where Tech could give a useful answer, given one or more support tokens, if you can formulate a tightly-focussed question. (And they might give permission to post the answer here.)

Following with interest, happy to test anything you can post or send by email.

If you're seeing consistently faster performance in 32-bit Manifold 8 than in 64-bit ("consistently" might be a stretch, you may not have much time to waste on repeated tests), then a crude but helpful approach might be to write a timestamp to the Manifold log after each step 1 to 11. Which step takes much longer in 64-bit? If just one, and always the same.

But who knows--asking Tech seems like a better start.

vincent

1,736 post(s)
#11-Jul-18 20:38

I don't know how much time it's gonna take, but I will test and, good idea, send a question to support.

My hint now is the following : Manifold starts compressing data when a .map file is reaching X size.

adamw


8,289 post(s)
#20-Jul-18 14:17

This looks like autosaving of the MAP file prior to running a script.

You can turn it off in Tools - Options, Scripts, Save file prior to running scripts.

vincent

1,736 post(s)
#29-Nov-18 13:09

I found an explanation for this behavior. If I run my script, containing a lot of queries and processing, while leaving one or a few open windows (map, drawings), data compression is occuring. If I close all windows before launching the script (which launches a lot of queries), this is not happening. The time needed to run the whole script is REALLY shorter if all the windows are closed when the script is launched.

KlausDE

6,271 post(s)
#29-Nov-18 14:27

Then code wrapped in app.ActiveDocument.BatchUpdates = True/False should be as quick as code with all windows closed plus the final redraw.

vincent

1,736 post(s)
#30-Nov-18 13:38

I was not aware of that property. It looks useful.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.