This thread describes the last batch of updates to the 9.0.169.x series of builds which were merged to 9.0.170.

The build can be downloaded from the Product Downloads page.

The build is going to be officially announced tomorrow, but can be downloaded and used today. (We are preparing a fat portable ZIP with various optional components for easy downloading of "everything".)

Changes

Renamed transform templates: Viewshed Area, Visible from All -> Visible Area, All Observers; Viewshed Area, Visible from Any -> Visible Area, Any Observer; Viewshed Image, Visible Count -> Visibility Index, Observer Count; Viewshed Image, Visibility of All -> Visibility Index, All Observers; Viewshed Image, Visibility of Any -> Visibility Index, Any Observer, renamed transform template parameters: Min camera -> Min V angle, Max camera -> Max V angle.

Viewshed transform templates set the refraction parameter to 0.13 by default.

(Fix) Viewshed transform templates correctly handle images with non-square pixels. The radius parameter passed to the TileViewshedXxx query functions is now in degrees / meters instead of in pixels, to allow uneven scales.

(Fix) Various UI elements that do not use menu access keys no longer show them.

(Fix) Pressing an unambiguous access key for a submenu now opens the submenu instead of merely selecting it.

Initializing the Manifold object model from a third-party application turns off services that prefetch data from opened MAP files to be less intrusive and give the application more control over the CPU / memory.

Renamed command for vector editing: Show Metes and Bounds -> Show Traverse.

Reading a traverse file in ESRI format supports polar directions.

The Record pane allows specifying display parameters for the traverse: direction type (North azimuth / South azimuth / Quadrant bearing / Polar) and direction units (Degrees / Degrees-minutes-seconds / Radians / Gradians (gons)).

The Record pane allows specifying display parameters for the clicked or selected coordinates using the context menu. Ends of straight line segments can be set to define segments using: angle + distance / direction + distance. Ends of circle arcs can be set to define arcs using: angle + arc / angle + chord / angle + radius / arc + chord / arc + radius / chord + radius. Ends of circle arcs can also be set to use one of the directions: chord (makes the arc non-tangent) / radial (makes the arc non-tangent) / tangent (makes the arc tangent unless the difference between the starting tangent of the arc and the tangent of the segment preceding the arc is too large).

The map window supports a new snap mode: Snap to Relative Bearing. The mode is similar to Snap to Bearing, but uses the bearing of the last entered segment as a base.

The map window allows switching to the default cursor mode using Shift-Escape.

The layout window allows switching to the default cursor mode using Shift-Escape. Switching to the default cursor mode no longer clears the Record pane (and no longer cancels changes made to the layout frame selected into it).

(Fix) Saving a non-map component as a map from a map window no longer sometimes forgets to save the coordinate system used in the window into the map.

(Fix) The Component pane correctly reports the coordinate system used by the map window as XY even if the coordinate system of the relevant component uses YX axes. (Map window always renders X horizontal and Y vertical.)

(Fix) Painting and printing layouts correctly handles components with YX coordinate systems.

Renamed commands for collations: Edit Collation -> Edit. Dropdown menus used by collation pickers put favorite collations at the top of the menu.

Renamed commands for coordinate systems: Change Coordinate System -> Reproject Component, Edit Base Coordinate System -> Edit, Edit Coordinate System -> Edit, Edit Metrics -> Edit, Use Default Metrics -> Use Default Values. Dropdown menus used by coordinate system pickers put favorite coordinate systems at the top of the menu.

The CoordConverterMake and CoordConverterMakePath query functions no longer fail when coordinate converter cannot be created due to an impossible combination of parameters in either of the coordinate systems, or due to an invalid coordinate conversion path or a path that references a missing or invalid grid file. Instead the functions return a NULL object. (Failing the function does not behave well in query constructs with multiple threads. The change allows to recover from a failure.)

The Reproject Component dialog is made modal. (Previously, it allowed clicking into windows underneath it, switching to a different component, etc. This was a leftover from when the old times when panes were modeless dialogs as well.)

The Reproject Component dialog is reworked to display coordinate conversion path using a specialized picker control instead of a list. If the source and target coordinate systems are the same, or the same save for coordinate system metrics and / or axes, the conversion path is fixed (at 'default' plus a short description of the case) and cannot be customized. Selecting a new target coordinate system preserves the selected conversion path if it remains available. Selecting a conversion path from the list of available paths displays path definitions similarly to the definitions of coordinate systems.

(Fix) The Reproject Component dialog correctly handles images with coordinate systems set to YX axes. The coordinate system of a produced image is set to use XY axes.

The Reproject Component dialog for images allows direct editing of pixel scale (local scales). The Auto checkbox (on by default) sets pixel scale to values that produce an image with roughly the same number of pixels by X and Y. There are two adjustments that make the number of pixels with Auto not exactly the same as in the original image: first, the exact scales are rounded to two digits (a scale of 1.31923 ft becomes 1.3 ft), second, if the rounded scales differ by less than 25%, they are made the same (equal to the smaller value) to make pixels square.

The Reproject Component dialog for images allows direct editing of image origin (local offsets). The Auto checkbox (on by default) sets image origin so that the coordinate of the pixel in the left bottom corner of the image is 0, 0.

The Reproject Component dialog for images allows customizing tile type and tile size for the new image. The Auto checkbox (on by default) sets tile type and tile size to coincide with those of the original image.

The Reproject Component dialog for images reports the dimensions for the image about to be created and the size of produced image data in bytes. The exact size of image data in the storage will vary due to a number of factors (compression, visible / invisible masks, intermediate levels, storage overhead), but the reported size is good for a ballpark picture.

The Reproject Component dialog for images computes bounds of a new image with greater accuracy (32x32 intermediate coordinates instead of 8x8).

The Reproject Component dialog checks all parameters prior to performing the reprojection or generating a query that will attempt to perform the reprojection, and fails if the reprojection cannot succeed due to an impossible combination of parameters in the coordinate systems, etc. Attempting to produce an image with more than 16 TB of pixel data also fails.

(Fix) Clicking Edit Query in the Reproject Component dialog no longer opens the window with the generated query below the active component window.

The base coordinate system picker allows copying and pasting coordinate system definition to or from the clipboard. SRID:xxx definitions that only makes sense in the context of a specific database are converted to JSON prior to the copying.

The coordinate system picker allows copying and pasting coordinate system definition to or from the clipboard. SRID:xxx definitions that only makes sense in the context of a specific database are converted to JSON prior to the copying. There are two paste commands: Paste pastes everything, Paste without Metrics pastes everything except coordinate system metrics.

Parsing coordinate systems from WKT and WKT2 recognizes many additional keywords.

(Fix) Reading GDB puts drawings and labels into the same folder as their tables.

(Fix) Reading E00 no longer fails to read files with compressed rasters.

(Fix) Exporting image to BIL supports multi-channel images with pixel values other than UINT8.

Reading XYZ adds postfix to the name of the table instead of to the name of the image (Filename Image + Filename -> Filename + Filename Tiles).

End of list.

So much hard work and thinking to explore here, and share more widely, even if 80% overwhelming.

Congratulations and I hope you all get a long holiday!

The official announcement is now ready.

There was one last change that managed to sneak in - menus with favorites have been simplified to follow a common, cleaner structure: list of favorite items -> More (formerly: Edit Xxx) -> Favorites (formerly: Edit Favorites).

The download page also features a fat portable ZIP which includes all the regular portable has plus the following optional components:

• DLLs for common database clients, vetted latest versions for MySQL, PostgreSQL, SQLite;
• DLLs for IronPython runtime;
• GRIDS.DAT.

Compared to the previous public build, the size of the regular portable installation package went down from 87 MB to 61 MB. The size of the fat portable installation package is 323 MB.

32-bit and 64-bit DLLs for database clients have different versions. Most databases now stop providing 32-bit modules and only provide 64-bit ones. 32-bit modules will probably continue to be able to connect to the database and work with it for a long time, but they will not support everything the 64-bit modules do and the difference in the supported features will grow over time.

Enjoy!

looks like one of the largest leaps forward in a long time. Just in time for Christmas vacation to begin exploring all the new functionality, and testing the speeds.

Very curious to see how processing LAS/LAZ files compares with LASTools and FUSION, and what multiple threads begin to offer.

There was one last change that managed to sneak in...

In case anyone else is wondering whether this means that the release builds now available for download (26/11/2019 UTC) are different from the early builds posted yesterday (25/11/2019 UTC), the answer is yes.

Apart from the simplifed Favorites menus, you can check which build you have by the SHA256 hash values.

The builds corresponding to the full public release have the hashes now given on the Downloads page (of course), e.g.

manifold-9.0.170-x64.exe

  fd96a29d2256788a192d43d76eb4fc1b148faac3a810e137820519cc5f34d2c6

manifold-9.0.170-x64.zip

  373bdd43e73cda792b0ef36235a543263bfac2e7d75a38438ccea61124bada3c

The earlier builds had hashes

manifold-9.0.170-x64.exe

  3a74fb5e1f2e7fba140289dfb7021c7e1f5d35b7e4fcb46d40a23d00b87c4fa3

manifold-9.0.170-x64.zip

  dfa5ae62bb299037770a3ce954e65516fbf6f51601e638085cb427591114b2d0

Apart from the simplifed Favorites menus, you can check which build you have by the SHA256 hash values.

At the risk of creating confusion where there was none, you can't check by looking at the version date in the About box, since both builds show date 25-Nov-2019.

The download page also features a fat portable ZIP which includes all the regular portable has plus the following optional components:

...

• DLLs for IronPython runtime;

[and a separate IronPython is also available, on the same page].

I'm so happy about this. It effectively settles IronPython as a first-class language for Manifold. For the fat portable, users no longer have to do anything extra to make Python available for use, while for the skinny portable and the full install there is now a common Manifold source to copy.

It was easy for enthusiasts before, now it is easy for everybody--and now an enthusiast can post in Python without having to add "er, this is IronPython, you have to go over here to get it and install it first".

This is, as it says, the IronPython runtime, with its dependencies. It does not include the Python standard library. When that is necessary we will need to refer to a full IronPython distribution, but that is normal.

It does not include the Python standard library. When that is necessary we will need to refer to a full IronPython distribution...

Occasionally we don't need to.

For example we can use the deque class without access to collections in the standard library, because under the hood collections depends on the built-in extension module_collections (written in C for CPython, ported to C# for IronPython) and in particular, collections.deque (IronPython) just refers to _collections.deque (C#).

So while, without the standard library, this can't work

import collections

nor this

from collections import deque

this does work

import _collections

as does this

from _collections import deque

Only deque and defaultdict are included in _collections, all other collections classes are in Python, but those two classes are supremely useful, since both need optimal speed.

(These nuances are new to me, feeling my way.)

It effectively settles IronPython as a first-class language for Manifold.

Yes.

We find over time that Python has to be a first-class language, because it is just so popular, particularly in scientific computing. C# and VB.NET have an advantage over Python in that (a) they come automatically with .NET and automatically have access to a very rich set of standard classes in the .NET Framework, and (b) they don't come bundled with a counterproductive religious split between language versions like Python does (Python 2.7 vs Python 3.0). But even then, Python has more than enough great qualities to earn its place.

We will continue writing all scripting examples in C#, VB.NET and Python. We will also make sure that any extensions to the object model that we make are Python-friendly - easily accessible, feel natural, have no additional overhead due to the use of something that Python implementations have to emulate.

Go, Python! :-)

(b) [C# and VB.NET] don't come bundled with a counterproductive religious split between language versions like Python does (Python 2.7 vs Python 3.0).

There is something of an analogy for C# though, as you and Riivo have noted in the past. That is the split between C# 5.0, supported by compilers included in versions of the .NET Framework, and C# 6.0, 7.0 and 8.0, which require a Visual Studio compiler.

Some of the features added in the later versions are really appealing, e.g. proper support for tuples (easy construction, methods returning tuples, tuple deconstruction aka unpacking), switch by type or by lambda expression, [..] range and [^i] index syntax, ?. and ??= ... lots of things, many quite like established features in Python.

We can't use the new features in Manifold scripts because (if I understand correctly) the Microsoft C# scripting engine is Framework-dependent, therefore stuck at version 5.0. [I don't think I do understand correctly. From Adam's explanations in the past I think it is a question of switching to a post-Roslyn compiler framework, which for reasons over my head is quite difficult.]

I don't get the impression this split is religious or even political, though, just messy.

That's a good point.

I grew so accustomed to .NET Core splitting off from .NET Framework that I forgot the split even exists. But it absolutely exists, and we are going to feel it the moment we start using .NET Core because scripts written for .NET Core simply won't run on .NET Framework unless the author specifically targets both flavors of .NET (which isn't great in practice). That's probably an even worse split than between two different flavors of Python.

Back in 8, I was unable to import pip packages with the Manifold IronPython implementation.

Can you import python packages like paycopg, censusgeocoder, etc.?

I haven't tried using pip from inside Manifold 8 or 9 (only from VS and only about once!).

In any case, though, IronPython can only import Python packages and extensions written entirely in Python, whether by pip or locally. (Plus any and all .NET libraries of course.) I gather Psycopg is written in C, so that's not usable from IronPython. Same with Numpy, SciPy as many people often lament.

Like Tim says, with IronPython you are mostly limited to packages that do not contain native code - which is a pretty severe restriction (I don't know whether the packages you mentioned contain native code = C/C++ DLLs, but they easily might). On the flipside, you can use anything .NET has - which is a pretty severe bonus. :-)

That said, when we are talking about supporting Python in 9, this does not have to be IronPython. We might support traditional Python in the future. Probably using whatever installation of Python already exists on the machine. IronPython has a lot of great things compared to CPython, particularly with multi-threading, but, well, being able to access traditional Python packages is quite significant and if we can allow a choice between IronPython and traditional Python, all the better. We'll see.

Yes. Both ArcGIS and QGIS support standard python, allowing you to load really cool packages like numpy, multiprocessor, pandas, etc. Having the ability to connect to those would be great. Lots of boutique pip packages would be great.