9.0.169.7

Here is a new build.

manifold-9.0.169.7-x64.zip

SHA256: d0520025a07a9099f6f5e1b4440a43f5bf116e54463e4aa5f2f38257ef7d960d

manifold-viewer-9.0.169.7-x64.zip

SHA256: 4e05455b5a42483770661400ae85484d1ecfa52ee634de968c90ad8519f6dca4

Changes

Folders in the Layers pane are toggled independently from layers. A layer is considered to be visible when it is turned on and all folders it is in are also turned on.

The drop-down menu for favorite coordinate systems displays coordinate systems which override metrics with a trailing '#' symbol.

(Fix) Tables with more than 10 fields no longer display with wrong field order in a table window.

Shift-clicking (instead of Ctrl-Shift-clicking) a field header in a table window no longer sorts records. (Supported clicks after the change: Ctrl-click = sort table on clicked field replacing previous sort order, Ctrl-Shift-click = add clicked field to existing sort order.)

Invoking View - Zoom to Native in a map window for an image adjusts zoom center to avoid smoothing image pixels. Resizing a window might need a re-adjustment.

Invoking View - Zoom to Native in a map window for a drawing, labels or map adjusts zoom center to make entered coordinates whole numbers (the adjustment is opposite to the one needed to avoid smoothing pixels for images). Resizing a window might need a re-adjustment.

New query functions:

• TileGeomToValues - takes an image and a geom, returns a table with all pixels under geom (fields: X, Y, Value).
• TileGeomToValuesX2 / X3 / X4 - variants of TileGeomToValues that return pixel values as an x2, x3, x4 (for images with multiple channels).

LAS dataport optimizes spatial searches without thinning with big rects and returns records progressively (instead of first collecting all records to return).

Snapping to coordinates in map window performs faster on data sources that support data thinning (MAP, LAS).

The object model supports expression contexts:

• Schema.Constraint has a new property: ExpressionContext (string, read / write).
• Schema.Field has a new property: ExpressionContext (string, read / write).
• Schema.AddConstraint has a new optional parameter for expression context (string).
• Schema.AddFieldComputed has a new optional parameter for expression context (string).

The object model supports searching for records via an r-tree index with thinning:

• Table has a new method: SearchBatchRTreeThin which takes the name of an r-tree index, search keys (ValueSet with a rect for the indexed geometry field), array of fields to output, X and Y size of a grid to use for thinning (eg, window size), and a boolean flag allowing (true) or disallowing (false) to reduce the number of coordinates in returned geometry values. Thinning is currently implemented by r-tree indexes in MAP and LAS files. If an index does not support thinning, SearchBatchRTreeThin works as SearchBatchRTree.

(Changes to the API doc reflecting the additions above are coming.)

Cutting edge builds expire on the first day of the third month following the build month (for example, this cutting edge build will continue working during August, September and October, and will expire on the 1st of November). One month or less before the expiration date, starting a cutting edge build will open the log window and output: "This cutting edge build is going to expire soon. Check for newer cutting edge builds on http://www.manifold.net". After the expiration date, attempting to start a cutting edge build will fail with: "This cutting edge build has expired. Check for newer cutting edge builds on http://www.manifold.net".

(Fix) Registering the ODBC driver for cutting edge builds no longer fails to use the 'experimental' name for the driver.

The ODBC driver uses a 4-component version instead of a 3-component one. (This is an old limitation that comes from the times when we did not support ODBC 3.x completely. We now do, so the limitation no longer applies, we can assign the ODBC driver the exact same version as the product.)

End of list.

I'm very interested to see the TileGeomToValues family of functions and would welcome any examples. Can these also be used to update the pixels under a geom in a performant way or is this something for a future function?

Also just a comment about the cache optimisations in the previous build. I suspect that it was something to do with these or other changes, but I went from only being able to generate watercourses/watersheds on relatively small DEM's to being able to generate watercourses/watersheds on SRTM data for the whole of the New Zealand mainland in 10 mins and 20 mins respectively on my 5 year old machine. Amazing work!

Landsystems Ltd ... Know your land | www.landsystems.co.nz

An example for TileGeomToValues - take an image, overlay a drawing with areas, return 3 highest pixels under each area:

--SQL9

-- take all pixels, return n highest

FUNCTION highestCoords(@t TABLE, @n INT32) TABLE AS (

  SELECT * FROM @t ORDER BY [value] DESC FETCH @n

) END;

-- prepare to convert coordinates from drawing to image

VALUE @conv TABLE = CALL CoordConverterMake(

  ComponentCoordSystem([german_alps]), -- to

  ComponentCoordSystem([sample_areas]) -- from

);

-- select 3 highest pixels in german_alps under each object in sample_areas

SELECT [mfd_id], SPLIT CALL highestCoords(

  CALL TileGeomToValues([german_alps], CoordConvert(@conv, [Geom])), 3

) FROM [sample_areas];

Regarding writing back into a raster, TileGeomToValues[Xn] functions are not really suited for that. What you are looking to do specifically and what do you want done in the case of conflicts (pixels covered by more than one object)? We have a couple ideas of our own, it's all completely doable, but we'd like to hear about uses first.

The performance boosts for watersheds that you see are pretty likely related to cache optimizations, indeed. Thanks for your kind words. :-)

Interesting example. A nice succinct answer to the transfer highest coordinate thread. I too am interested in a Vector to Raster function. Basically you would use an area to modify a raster within the area's extents.

An example of this would be for urban catchment delineation. One of our issues is with the quality of our full feature to bare earth conversion. Often things like canopies or parked vehicles distort the curb and gutter which is the primary flow path. To get around this, we depress the roadway the height of the curb. This gives us more representative streamlines and routes runoff towards catchbasins.

Another example of this would be for two dimensional flow modelling. To make the modelling software work, we have to use a bare earth raster. However, we all know that flow cannot travel through buildings. Our workaround is to raise the building footprints a few metres such that our software routes flows around the raised buildings.

In Manifold 8, we would accomplish these tasks by transferring the selection from the vector layer to the raster. We would then use the Transform menu to raise/lower the selected pixels. The transfer selection step essentially eliminates the issue of many-to-one conflicts because you can only select the same pixel once. Not sure what the best method of accomplishing this in M9 would be...

Hi Adam,

As Andrew rightly points out, in M8 this largely meant either transferring a selection from another component to an image or creating a freehand selection in an image and updating the selected pixels in some way. To me, this was one of those magical functions which I have used innumerable times and which stood Manifold 8 out from the pack. It is something that I have long hoped will be added to 9 and perhaps extended to include both freehand selections and updating pixels using other components and selections within these components (either with a selection flag or with values).

By way of examples, I have used the M8 functions (in conjunction with M8’s ‘Selections’ pane tools) to build proposed coastal marine structures into LiDAR DEM’s using architects CAD drawings. This is typically to create the base data for hydrodynamic modelling software to simulate the effects of proposed structures upon currents and the patterns of sediment erosion/deposition.

I also use it to:

- Insert or remove flood control structures in an ‘as built’ state to LiDAR data.

- As part of process to estimate the floodplain extent of rivers from LiDAR.

- Assess river flood inundation in response to flood control structure breaches.

- In preparation for various hydrological modelling similarly to Andrew’s examples.

I have also used it numerous times as a component of creating cartographic presentations.

I should also mention at this point that the ‘Modify Selection' tools in M8 where often invaluable for working with selections once transferred allowing them to be expanded, shrunk or smoothed.

My current pressing need is to allow me to complete a script which produces layers of marine connected and disconnected flood inundation at incremental intervals. This is to support an NZ wide effort to create public facing indicative flood extent viewers in response to sea level rise or other flooding events, similar to the NOAA sea level rise viewer:

https://coast.noaa.gov/slr/#/layer/slr

I have labored to create a performant script in ArcMap but in parallel have been replicating the process in M9. I have all of the steps in M9 apart from setting the marine disconnected pixels to a value of 99 (green pixels in NOAA’s viewer) for which I need to be able to update pixels under drawing objects.

In ArcMap the un-optimised process on my test data takes ~20 hours to run on my test data set. In contrast, I estimate that with the ability to update my pixels, in M9 the same will take ~10 minutes and would offer the additional benefit of being able to process all data sets at once.

This is something I obviously want to pursue as we will be assisting other authorities around the country with building their own data sets. It also seems perfectly suited to M9’s grid analytics and in addition to saving countless labor hours will allow me to really showcase the utility of M9 to my employers and others.

and what do you want done in the case of conflicts (pixels covered by more than one object)?

Typically I have used this as per the M8 implementation of transferring a selection and then updating the pixels in some way which avoids this problem. In the case that we can update pixels under multiple (potentially overlapping) drawing objects, could the transfer rules not be called into play to determine how to deal with this?

Landsystems Ltd ... Know your land | www.landsystems.co.nz

Excellent posts by Dan and Andrew. I would just say "+1", but would like to add one simple example, and two comments, one regarding overlaps and the other regarding selections.

The example: flattening lakes and waterbodies to a known elevation. This is very similar to Dan's inundation examples of course. It also seems similar to filling sinks to a known elevation, except for being restricted to given vector objects, and applying per-object elevation defined by an attribute.

Re overlaps:

what do you want done in the case of conflicts (pixels covered by more than one object)?

Well mainly, no removal of overlaps by implicit normalization. In my opinion that is nearly always a bad thing, unless it is essential to make an algorithm finish in reasonable time, or to avoid circularity. Leave it to the user to remove overlaps in advance, if they are errors.

If overlaps or other conflicts exist, then I like the simple rule "last object wins". For now, this is not under user control, since Z-order is undefined. Arbitrary assignment (including multiple sequential assignments when multiple threads are in play) is normally perfectly OK in the meantime. If and when it is not OK, the user has the option of normalizing topology in advance, or manual clipping in a defined order, or detecting and harmonizing crossed lines. (Only the user can know whether this is necessary.) When it becomes possible to define Z-order by the value of a field, then "last object wins" should I think become "highest Z-value wins".

Lastly regarding selections. The Manifold 8 approach of first selectiong pixels under geometry, then applying a transform to selected pixels, is convenient enough but seems to me (a) to involve an unnecessary step, and (b) not to fit the Manifold 9 image tile model well at all. Applying values directly from geometry to touching pixels seems more efficient and more natural to me.

All of this is possible already with 9, but it feels like a lot of work. It would be really great to see it built in and easy.

Applying values directly from geometry to pixels seems more efficient and more natural to me.

Agreed, but the ability to make irregular selections by freehand or component transfer and work with these is also extremely useful. As such I would like to see both.

Landsystems Ltd ... Know your land | www.landsystems.co.nz

Can you give an example, of where that is a necessary (or best) way of working?

In particular, drawing a freehand selection of pixels.

Thanks a lot to all of you, this is exactly the feedback that we wanted.

It looks like extending the selection in images from full tiles to pixels and keeping it easy to alter / manipulate just the selection in queries would be a big step forward so we are perhaps going to do this first and other things second.

(We have been planning to allow selecting individual pixels in images opposite to just selecting tiles from the very beginning. The current model of selection in an image being a subset of records = full tiles in terms of pixels was mostly a temporary stopgap. We did this limited selection model for images because there are multiple ways to go about storing selected pixels, and we wanted to spend some time on raster analysis tools first, both simple tools like filters and more complex tools like watersheds, to see what their preferences regarding how to store selections would be. We now have a pretty good idea regarding what we need, this involves reorganizing / extending images themselves, too. This is a fairly big chunk of work that we are planning to do in the near future. As a birds-eye view, we have three such big chunks of work on our plate: "geometry", "rasters" and "servers". We are currently busy with other things, but we make gradual progress on each of these chunks in parallel.)

We have been planning to allow selecting individual pixels in images opposite to just selecting tiles from the very beginning

Can I ask why you have never said so?

It is not a rhetorical question. Really, why.

Neglecting to disclose important design intentions like this (even through two long successive betas) only wastes goodwill and time.

Can I ask why you have never said so?

That's been discussed in the past, as part of plans to implement rasters, and to backfill various 8 capabilities into 9.

I tried to look through previous discussions on image selections and our plans regarding that in the beta areas of this forum and it seems we didn't say anything like "image selection is always going to be whole tiles" but - at least in what I have been able to find quickly - we didn't say "we are going to switch from selecting full tiles to selecting pixels" either. I guess by not saying that in the end we do want means to restrict operations to individual pixels rather than tiles, or by not saying it clearly enough, we might have created a wrong impression here. Apologies for that.

Part of this is perhaps because while we always wanted to be able to restrict operations to individual pixels rather than tiles, we were considering different ways of doing this and in some of those ways we would have things that wouldn't be called "selection" (eg, several approaches we considered were using just "masks").

Again, apologies for the confusion.

(Here is a post from Dimitri that talked about selecting pixels: ... pixel level selections ... There are probably other posts like that, but I agree it's pretty buried.)

Hey Adam, Glad too see pixel selection is in the works. Having access to individual pixels for manipulation of their channels and elevation would be fantastic. It would also be very useful to have access to the pixel's visibility state as well.

In Manifold 8, we would turn some pixels off to force streamline terminations at that pixel's centroid. Combining this with the ability to transfer vectors to rasters would allow for much more control over the raster itself.

Great news. Thanks for the explanation and birds eye view. I look forward to seeing some of these features in future builds.

Landsystems Ltd ... Know your land | www.landsystems.co.nz

Speaking of snapping...

Snapping only seems to work when creating a new element. Can it be turned on for editing an existing element?

Yes, we will do this.