Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System

8,884 post(s)
#29-Apr-19 17:30

Here is a new build.

The focus was on rendering, we were reducing blinking, minimizing delays, making transitions smoother, etc. We were also fixing outstanding issues and doing much more testing than usual, preparing to release the public build. We are planning to release the public build in a few days. If you can, please give this cutting edge build a run, check any issues you may have had before and report if any of them are still there or partly there - we'll try to fix them. Or just use the build normally and report any issues you encounter - we'll try to fix them as well. After we release the public build, we'll start a new series of cutting edge builds. The focus will initially be on legends / printing / curved labels.

SHA256: 10e453a5dee87cc0ef68acd5f83fe28b1f67a672ab455e02620757d9063e95f7

SHA256: d61c4ebc98db5119125c9b16c32187340dfcd133294e74b9b77a2b22255a3e72


8,884 post(s)
#29-Apr-19 17:31


Rendering maps better schedules screen updates. Rendering threads get reasonable time to produce first screens (0.5 sec, could be extended to 1 sec depending on the internal heartbeat) and until that time passes, the screen does not change. If rendering completes earlier, the screen is updated immediately (previously, there was a delay of up to 0.5 sec, 0.3 sec on average). This makes rendering of small amounts of data complete faster, and helps rendering of large amounts of data proceed with much less blinking.

Rendering a map with layers that are all slow and cannot produce any data for a long time keeps last used screen for up to 2 sec (after that, the map window gives up and paints the new screen even if it is blank).

Re-rendering a map after changes to layers (turning layers on and off, selecting objects, etc) coordinates with rendering threads to minimize blinking.

Resizing a map window coordinates with rendering threads to minimize blinking.

(Fix) Zooming a map by scrolling the mouse wheel no longer sometimes exits zooming prematurely (and restarts it from the new zoom level).

Rendering a map with an overlay after zooming it by scrolling the mouse wheel no longer sometimes shows the overlay twice - once as part of the scaled screen and also as a true vector overlay.

Rendering web images from image server data sources first paints data at reduced resolution and then gradually details it. This quickly fills the screen with rough data at the cost of about 10% more fetched data (frequently served from cache). The optimization works for images in parent data sources that reference table in the image server data source. If the window is small, the optimization is turned off.

Rendering a map with a lot of layers limits the number of rendering threads to make better use of system resources. The number of rendering threads depends on whether the window is active or not - rendering an active window can use as many threads as there are CPUs (plus a couple more to maximize utilization as threads switch between layers), rendering an inactive window can use up to two threads. If an active window becomes inactive, its existing rendering threads are allowed to complete and the limit is only applied towards new threads. If an inactive window becomes active, new threads are created immediately.

Logging rendering time for a map no longer logs the average number of threads used during the rendering session (with dynamic adjustments to thread limits it is hard to make this value useful). If the rendering session was interrupted with some of the layers being re-rendered due to changes in their data, rendering time is not logged (this was creating inaccurate reports before).

(Future areas for improvement in rendering: (a) preview blinking - previews already blink much less than before, but we can do more, we have several other changes planned for previews, reducing blinking will come as part of those changes, and (b) maps with lots of layers - limiting the number of rendering threads in flight does wonders for performance, but we can do more by reducing storage requirements, particularly when layers change mid-way through rendering, this will also come later.)

New query functions: TileInterpolateKrigingRegression, TileInterpolateKrigingRegressionPar - do regresssion kriging. The parameters are those used for regular kriging plus one extra parameter for regression model (auto / linear / quadratic, specified as a numeric value). There is a new transform that uses the functions: Kriging with Regression.

(Fix) Composing intermediate levels for images correctly averages data for images with the alpha channel regardless of whether channel values have to be inverted or scaled during rendering.

ArcGIS REST dataport supports servers with exportImage command (this allows using some servers that could not be used before).

Reading TIFF files better detects images with 4 channels where the 4th channel is not alpha.

Reading DBF better detects fields with floating-point values (DBF fields are 'numeric' with no separation into integer vs floating-point, whether a field is integer or floating-point is determined at runtime).

(Fix) Writing geometry to GDB normalizes areas.

SHP reads multi-patch geometry values. Opening a SHP file with multi-patch geometry values disables writes to the geometry field (there is no good way to roundtrip these values due to current limitations of our geoms, read-edit-write would be losing data).

(Fix) Exporting a drawing to GPKG no longer sometimes fails with 'cannot change table schema'.

(Fix) Reading GPKG no longer sometimes fails to use a spatial index with a complex name due to improper quoting.

(Fix) Writing geometry values to GDB no longer loses Z data.

(Fix) Exporting a very big image to TIFF no longer sometimes fails when producing data for intermediate levels.

(Fix) Pasting a folder into another folder in a different instance of Manifold no longer sometimes creates two folders (one in the target folder, another at the root of the target data source).

End of list.


5,765 post(s)
#29-Apr-19 18:21

I'd just like to say Congratulations! to the Engineering team for such a dramatic improvement in rendering.

I've been working with .12 all day today and there's no way you can go back. It's so much smoother you get used to it instantly and it's easy to forget what life was like before .12.

Try firing up a portable installation of .11 and run that next to .12 with a larger or more complex map and try doing the same thing side by side in .11 and .12, like undocking a map, resizing the window and seeing how the rendering is so much better in .12.

Things that used to drive me nuts, like how resizing a column in a table window would cause a redraw in the associated drawing if that drawing was open in a window, are now gone. Select/deselect and no blinking. So much better! :-) This was a huge amount of work but it sure has paid off.

9,111 post(s)
#30-Apr-19 02:07

The speedups seem to be deep in the DNA. For example, opening a single large image was "instantaneous" before. Now it is a different kind of instantaneous. (After repeat comparisons, 168.12 versus 168.11.) I could get really used to this!

And everything just works.

What a great way to show clients their own data.--What???? We need this.


5,765 post(s)
#30-Apr-19 07:49

The speedups seem to be deep in the DNA.

Yes. This may be the most extensive build in terms of very widespread internal interactions since the Radian builds. What adamw wrote in the release notes is about a tenth of the many optimizations and other improvements that were done. That, of course, makes it especially important to hammer on this build to surface any issues that made it through internal testing.

Hopes are to issue a public build tomorrow, so it would be great if everyone could try out .12 on various projects, paying special attention to how they render as different windows are opened and different work gets done. There is obviously a lot of confidence in .12, but, hey, you never know, so the more hands pounding on this the better. :-)

An example of the sort of error that might be found: in an alpha build I noticed that in a map I was working with which had a Google satellite layer above a Bing streets layer the Google layer rendered with slight transparency, despite being set to 100% opacity in the Layers pane. I reported that and it got fixed before .12 was issued.

So please, hammer on .12 and right away report any cases where projects that rendered perfectly before show any errors in rendering. If at all possible, check to see if the issue that seems to be happening also occurs in .11, in which case it is not something specific to the changes in .12.


3,118 post(s)
#30-Apr-19 17:16

The NYC taxi data I provided way back when responds very well under certain circumstances. If you recall, the data was messy, and included a lot of 0 latitudes and longitudes. But, that's a good thing, it really stresses the system.

1. if you create a new map, and drop the taxi_zones in, and then slide the enormous pickup or dropoff points, it displays very quickly. Zooms to the taxi_zones is instantaneous, and shows all the dropoff points instantaneously, too.

2. but, the initial loading of the dropoff points takes over 20 seconds to display. But, once it is drawn, it zooms and pans instantaneously.

gotta run. Hopefully more tests later.


8,884 post(s)
#01-May-19 06:33

The initial full zoom render takes this long because it is a cold start and because data is split into more individual pieces than usual (because it is points, areas / lines are usually much less numerous). Full zoom renders after the first one are going to perform somewhat better. We know how to deal with points as well, have plans for future improvements.

Overall, current build seems to perform better than previous builds on this data at detailed zooms, and about the same at full zoom. (Good.)


3,118 post(s)
#01-May-19 14:41

yes, the detailed zooms are excellent, as are the pans.

685 post(s)
#01-May-19 15:08

I like the blue line under the tab showing which tabs are still rendering.

The following image shows rendering times for turning a tab on and off on top of some complicated tabs. The layer being turned on and off is the Rock Wall Points 2 layer. It is not complicated having only 8 points in it styled in one color. The green layer is a hill shaded DEM with 1 meter resolution covering half of our county. The black lines are the 1-meter contour lines covering the same area. There are 186,000 records in that table.

Zooming out to the full extent of the contours and turning that tab off takes 0.236 seconds to render. Turning the tab back on takes 11.649 seconds to render.

Turning the hill shade tab on and off with the contour lines off takes 0.013 seconds (off) and 0.227 seconds (on).

Render times in M9.jpg


8,884 post(s)
#01-May-19 15:37

Thanks for taking the time to test and write a note!

The 11.649 seconds to render all contours is the first target for future improvements. The problem is kind of tough because contour lines tend to go all over the drawing, they are frequently very long. But we have some ideas, both for full zoom and for detailed zooms. :-)

Turning even a simple layer on and off in a map that includes a complex layer will have to re-render the complex layer currently. We might add some smarts here as well.

Other than that, the times seem pretty good.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.