Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.169.2
adamw


8,650 post(s)
#04-Jun-19 17:24

9.0.169.2

Here is a new build.

manifold-9.0.169.2-x64.zip

SHA256: 416ef49d9eb61d09ecfa65c90edb7cd0ab7a8914363ccfae5709b0b7f47dec2e

manifold-viewer-9.0.169.2-x64.zip

SHA256: f429c7abbdfc477f6683a121c692f6669e9a0296a026dfa2c8a45ce555028a07

adamw


8,650 post(s)
#04-Jun-19 17:26

Changes

The list of layers in the Layers pane uses a single column. Pressing Enter or double-clicking a layer starts editing it. Pressing Space toggles the active layer as well as all other selected layers if the active layer is selected.

Selecting layers in the Layers pane follows group hierarchy. Selecting or unselecting a layer with children selects or unselects all children. Unselecting a layer with parents unselects all parents (to preserve the invariant that if a layer is selected, all its children are selected as well).

Moving layers in the Layers pane follows group hierarchy. Moved layers never change their group level, layers never move between groups.

Grouping layers in the Layers pane preserves group selection. Actions that create groups with selected parent and unselected children are denied.

The Layers pane includes a filter box that allows filtering layers using parts of their names. Applying a filter disables moving and grouping commands as their effect could be invisible. Deleting / editing / toggling selected layers can affect layers hidden by the filter (this is similar to how the selection works in tables). Selecting or unselecting layers can affect layers hidden by the filter as well (eg, selecting a layer will select all its children, including those that are hidden).

The Layers pane allows expanding and collapsing layer groups (Plus / Right Arrow expand, Minus / Left Arrow collapse). Switching between windows preserves collapsed states of layers per-window. Moving or grouping commands automatically expand groups that contain selected layers to make the effect of commands visible.

Clicking a + / - icon of a layer in the Layers pane expands or collapses the layer.

Command windows mark their log messages with '(Command : CREATE ...)' or '(Command : [datasource]::[datasource] : CREATE ...)' instead of with '(ad-hoc)' to help distinguish commands from each other.

(Fix) The definition of 'Spain Grid' (a built-in coordinate system carried from Manifold 8) uses correct value for X scale adjustment.

Parsing coordinate system from WKT (PRJ) has been extended to recognize a number of new coordinate systems.

New query functions for coordinate systems:

  • ComponentCoordSystemScaleXY - takes a component object, returns an x2 value for local scales for X and Y.
  • CoordSystemScaleXY - takes a coordinate system definition, returns an x2 value for local scales for X and Y.
  • CoordSystemBase - takes a coordinate system definition, returns the base lat/lon coordinate system.
  • CoordSystemForceXY - takes a coordinate system definition, returns the same coordinate system with axes forced to XY order. If the original coordinate system definition is an EPSG or SRID code, the new coordinate system is the same code with overrides.
  • CoordSystemPlain - takes a coordinate system definition, returns the same coordinate system with local offsets set to 0 and local scales set to 1.
  • CoordSystemWkt - takes a coordinate system definition and a strict boolean flag, returns the WKT (PRJ) representation. If the coordinate system cannot be represented as WKT, the function returns a NULL value. If the strict flag is false, the function maps some coordinate systems that cannot be represented as WKT to close variants (eg, the Lambert conformal conic variant for Michigan is mapped to Lambert conformal conic).

New query functions for measurements, low-level:

  • GeomAreaGeo - takes a geometry value and values for major axis, eccentricity, tolerance, returns geodetic area. The geometry value must be an area in lat/lon, with X (lon) and Y (lat) expressed in degrees. The returned value is in the squared units of major axis (typically, meters).
  • GeomBearing - takes a geometry value, returns projected bearing. The geometry value must be a line with a single branch, the bearing is computed from the first to last coordinate of the line. The returned value is in degrees.
  • GeomBearingGeo - takes a geometry value and values for major axis, eccentricity, returns geodetic bearing. The geometry value must be a line with a single branch in lat/lon, with X (lon) and Y (lat) expressed in degrees, the bearing is computed from the first to last coordinate of the line. Geodetic bearing is computed using Vincenty's formulae. The returned value is in degrees.
  • GeomBearingPoint - takes a pair of point (x2) values, returns bearing. The returned value is in degrees.
  • GeomBearingPointGeo - takes a pair of point (x2) values and values for major axis, eccentricity, returns geodetic bearing. The point values must be in lat/lon, with X (lon) and Y (lat) expressed in degrees. Geodetic bearing is computed using Vincenty's formulae. The returned value is in degrees.
  • GeomDistancePointGeo - takes a pair of point (x2) values and values for major axis, eccentricity, returns geodetic distance. The point values must be in lat/lon, with X (lon) and Y (lat) expressed in degrees. Geodetic distance is computed using Vincenty's formulae. The returned value is in the units of major axis.
  • GeomLengthGeo - takes a geometry value and values for major axis, eccentricity, tolerance, returns geodetic length. The geometry value must be an area or a line in lat/lon, with X (lon) and Y (lat) expressed in degrees. Geodetic distance is computed using Vincenty's formulae. The returned value is in the units of major axis.

New query functions for measurements, high-level:

  • CoordMeasureMake - takes a coordinate system definition, a unit name and a geodetic boolean flag, returns a measure object (similar to coordinate converter) that can be used to perform measurements.

If the coordinate system is lat/lon and the unit is lat/lon, measurements are Euclidean, performed in the specified lat/lon coordinate system.

If the coordinate system is lat/lon and the unit is metric, measurements are geodetic.

If the coordinate system is metric and the unit is lat/lon, measurements are Euclidean, performed in the base lat/lon coordinate system of the specified coordinate system. The geometry values are converted to lat/lon as part of the process.

If the coordinate system is metric, the unit is metric, and the geodetic flag is false, measurements are Euclidean, performed in the specified metric coordinate system.

If the coordinate system is metric, the unit is metric, and the geodetic flag is true, measurements are geodetic. The geometry values are converted to lat/lon as part of the process.

  • CoordMeasureArea - takes a measure object and a geometry value, returns area. The geometry value must be an area in the coordinate system of the measure object. The returned value is in the squared units of the measure object.
  • CoordMeasureBearing - takes a measure object and a geometry value, returns bearing. The geometry value must be a line with a single branch in the coordinate system of the measure object, the bearing is computed from the first to last coordinate of the line. The returned value is in degrees.
  • CoordMeasureBearingPoint - takes a measure object and a pair of point (x2) values, returns bearing. The point values must be in the coordinate system of the measure object. The returned value is in degrees.
  • CoordMeasureDistancePoint - takes a measure object and a pair of point (x2) values, returns distance. The point values must be in the coordinate system of the measure object. The returned value is in the units of the measure object.
  • CoordMeasureLength - takes a measure object and a geometry value, returns length. The geometry value must be an area or a line in the coordinate system of the measure object. The returned value is in the units of the measure object.

Removed query functions: GeomScaleToSystem, GeomScaleToSystemCff, GeomScaleToUnits, GeomScaleToUnitsCff. These functions have been superceded by functions in the CoordMeasureXxx family.

New query functions for watersheds:

  • TileWatershedAreas / TileWatershedAreasPar - take an image and a number for minimum flow, return a table of watershed areas. The XxxPar variant takes an additional parameter for thread configuration. The image must be single-channel.
  • TileWatershedLines / TileWatershedLinesPar - take an image and a number for minimum flow, return a table of watershed lines (streams). The XxxPar variant takes an additional parameter for thread configuration. The image must be single-channel.

Watershed functions return a table with the geometry values as well as values for stream ID (int64), target stream ID (int64), Shreve order (float64), Strahler order (float64), flow (float64), total flow with children (float64).

Watersheds are computed using a new algorithm that is radically different from the algorithm used by 8. Due to differences in algorithms, watersheds computed by 8 and 9 are similar, but frequently not equal. Most of the differences come from plateaus, where the behavior of streams is wide open to interpretation. The algorithm used in 9 is much faster than that in 8 and can handle vastly more data.

New transform templates: Watershed Areas, Watershed Lines.

(Transforms and functions to fill sinks to specified level / trace upstream lines are coming.)

End of list.

BerndD

125 post(s)
#05-Jun-19 15:32

As for grouping layers I am wondering if we can transfer the folders from the project pane to the layer pane with contained layers already grouped. Then the folder should have a check box to toggle the visibility of that folder.

Ignoring the folders in the layer pane eliminates all the efforts that a user puts into organizing his or her data in the project pane.

I think organizing layers first and then using them in a map is the right way to do.

At the moment I have to toss all layers into a map and then I have to organize them after the fact.

If I create another map, I have to organize all layers again.

Sure, I am missing individual layer groups per map, but I think that's worth it. Usually I am using grouped data the same way in all my maps.

Just a thought and curious how other users want to see that handled.


The Future of Spatial Data // www.drahola.com // www.geocockpitug.de

tjhb

8,846 post(s)
#05-Jun-19 16:00

Yes.

Perhaps it would be enough if, when a folder is dragged into a map, the first eligible component within it (drawing or image) would appear as a new leading layer, and all further components from the folder as child layers beneath it. (Nested folders could be treated the same, i.e. producing nested leading layers with their groups.)

Then the grouped layers would be easy to arrange and rearrange as necessary--or to ungroup, if a group structure were not wanted. Often, we might want some other component than the first to become the leading layer, but this would also be easy to adjust. The most important thing is for the folder contents to come in initially as a group, so that Project pane structure can be readily reused as map layer structure.

The folder itself need not become a layer, I think.

BerndD

125 post(s)
#05-Jun-19 16:08

Dragging an entire folder into a map sounds like a great idea.

I think having an actual group name instead of taking the name of a leading layer would be beneficial.

Having a group name is giving the user a better understanding of what the contents of that group actually are, for example "Environmental data" or "County data".

I know it comes with quite some effort to introduce group folders in the layer pane, but it really might be worth it.


The Future of Spatial Data // www.drahola.com // www.geocockpitug.de

adamw


8,650 post(s)
#05-Jun-19 16:20

We are using leading layers instead of non-layer entities because that allows us to use groups in layer tabs - that's not yet in the builds, but coming.

If we grouped layers under folders which wouldn't be layers themselves, then layer tabs either:

(a) wouldn't be showing them, which is bad because layer tabs and layers pane show different things, or,

(b) would be showing them but those folder tabs would be essentially traps for the user - if you make a folder tab active, all panes would go blank, all interactive tools would go disabled, etc, because a folder is not a drawing, not an image, not anything.

Maybe we overestimate (b).

tjhb

8,846 post(s)
#05-Jun-19 16:35

I agree that (a) and (b) would both be bad.

But, taking Bernd's point about the value of keeping folder name as group name, perhaps there is also:

(c) That folder tabs could not be activated at all (thus avoiding (b)), only their child layers could be. Single left-click on a folder tab would do nothing, but double-left click could toggle group visibility, and left-click-drag could reorder (move whole group). Folder tabs would be visual placeholders (perhaps with slightly different font or tab colour), along with right-click options for opacity, ungrouping, reordering, etc. (duplicating some controls available in Layers pane), plus controls to collapse/expand display of tabs for child layers.

I still like the leading layers concept (especially bearing in mind that a leading layer can either be empty, or contain e.g. a bounding box or some other content relevant to its group), but Bernd makes good points too.

BerndD

125 post(s)
#05-Jun-19 16:53

As for b), you can have the first layer in that group be the active layer.

If you double click, the group toggles on or off.

If you just click, it can open a popup with all layers in that group where the user can mark one of the layers as the active layer (see image). This active layer would be the correspondent to all linked panes and tools.

And if you add the ability to drag layers up and down, you will also have a tool to arrange the stacking of the layers within a group.

Attachments:
Manifold_Group_Layers.png


The Future of Spatial Data // www.drahola.com // www.geocockpitug.de

adamw


8,650 post(s)
#06-Jun-19 08:41

Thank you both, Bernd and Tim, for great notes! Definitely some food for thought.

danb


1,708 post(s)
#05-Jun-19 20:12

(Transforms and functions to fill sinks to specified level / trace upstream lines are coming.)

With regards the trace mentioned here. If could the following options be considered?

  • The option for upstream or downstream tracing between reaches
  • The option to trace upstream or downstream from multiple sources simultaneously while preserving the source reach in the output as a catchment identifier or for use in dissolving multiple traced watershed areas into distinct catchments.
  • The option to include one or many upstream barriers to allow tracing of partial hydrological catchments

Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


8,650 post(s)
#06-Jun-19 08:52

We are going to have at least upstream tracing from points. We'll look into the rest.

To clarify, when you say "reaches", do you usually specify them as lines or areas?

Thanks for the note!

danb


1,708 post(s)
#10-Jun-19 06:40

Hi Adam, on second reading, we might be talking about slightly different things, but I will clarify how I use these hydrological datasets and the terminology that I use. Firstly, to answer your question, when I say reach, I use it interchangeably between ‘reach watersheds’ (watershed areas) and ‘reach watercourses’ (watershed lines). A reach to me is therefore a watershed area and its corresponding watercourse line, identified in the two datasets by a common unique reach identifier.

In my data table, I will have a common attribute [REACH_ID] in both datasets which identifies a common reach watershed and its corresponding reach watercourse. M9’s attributes [Stream] and [Target] I would know as [FNODE] and [TNODE] (from and to node), likely from my ESRI upbringing.

The way we use this data is for catchment planning and prioritisation and to do this I need to be able to use SQL to trace upstream, downstream and in either direction between optional upstream or downstream barriers. Tracing is only between whole reaches i.e. no partial reach lines or reach watersheds and I need to be able to trace more than one catchment at a time very rapidly in SQL. Passing source reaches is achieved using something like WHERE [REACH_ID] IN (1,4,6,7,8, …x) or by making interactive selections of source reaches for tracing in the GUI.

To this end I built myself something in M8. Essentially a single table with the traces built-in for every reach (all be it in a fast but kludgy way). This base table has a variety of other commonly used hydrological attributes and I am able to ‘hang’ any of our other datasets onto this base via joins or table relations (lookup tables are commonly the intersection of a such as example land cover with the reach watersheds. In this lookup I might have each of the unique reach identifiers and the percentage area of the reach in each of the vegetation categories). The REACH_ID then becomes the field allowing the data to be joined back to the base dataset.

Because the ‘traces’ are so fast, in SQL and because I carry a common reach identifier for each individual source ‘catchment’ (composed of multiple traced upstream reaches), I can hang various datasets on and trace/aggregate data combinations to ask all sorts of questions for one or more upstream catchments such as …

Which catchments have a high pastoral stock unit density within erosion prone areas? Which of these sub-catchments present the highest ‘risk’ in terms of the considered variables?

At what point within the headwaters of a catchment does the watershed fall below some threshold of intact natural habitat.

Etc. etc.

It also allows consideration of both the upstream and in reach contribution of a quantity such as nutrients, sediment or pathogen for every reach watershed within a catchment. This can be used to help hydrologically prioritise and spatially place remediation works within a catchment.

It can also be used to ‘balance’ catchments across multiple variables, that is split a regional scale catchment into x number of hydrologically intact sub-catchments, each of which have an approximately similar quantity of one or more variables such as erodible land.

Anyway, the long and short of it is that I would love to be able to do multiple source, rapid upstream and downstream catchment tracing and aggregation built-in to M9 SQL. This is not required necessarily to the same degree of sophistication as a more complete network analysis capability, but speed is the key, something that M9 seems to be very good at.

This has proved to be a useful tool that has been extended and modified to support a number of activities including:

https://www.waikatoregion.govt.nz/council/policy-and-plans/waipa-catchment-plan/

https://www.waikatoriver.org.nz/about-the-waikato-river-authority/purpose/


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


8,650 post(s)
#10-Jun-19 07:41

Thanks, that helps a lot.

To see that I understood it properly, we are not talking about additional functions that would take a raster and derive geometry using some new criteria - existing geometry is fine. We are talking about means (query functions) to take something resembling the current output of watershed lines / areas and do things like find all parents / children of specified stream ID, possibly with limits on such scans (stop after N steps up / down, or stop after total area / whatever exceeds P, or stop when hit objects X1, X2 ... Xn which act as barriers). Correct?

(We also need some means to make sure the IDs of watershed lines / areas coincide - right now there is no guarantee that if you run one, then the other, the IDs will be the same.)

danb


1,708 post(s)
#10-Jun-19 09:17

That's exactly what I was meaning. Many thanks for your interest and let me know if I can provide further clarification.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


8,650 post(s)
#04-Jun-19 17:27

A performance illustration for the new watershed algorithm.

Example 1, 1200 x 1200 raster, computing streams (lines)

8: 75.000 sec

9, single thread: 3.024 sec

9, multiple threads: 1.727 sec

Example 2, 1200 x 1200 raster, computing watersheds (areas)

8: 175.000 sec

9, single thread: 3.843 sec

9, multiple threads: 2.705 sec

Multiple threads help more with more data.

Mike Pelletier


1,618 post(s)
#04-Jun-19 19:24

Looks like a big effort has been done hopefully to get ready for vector tools as well as to play nice with ArcSDE coordinate system definitions. That's good.

Unfortunately, by default the new version is rotating my layers in ArcSDE 90 degrees to the east (Colorado State Plane Central, NAD83, ft) and it tells me that I cannot set value when I try to repair the initial coordinate system because of it being read only. Assume the problem is XY switching.

Also, the coordinate system dialogue is not wide enough to display all the SDE coordinate system info and cannot be resized.

Mike Pelletier


1,618 post(s)
#04-Jun-19 20:27

The XY switch occurs also with shapefiles with .prj files as well. Without thinking through the implications, can we make the Force XY checked by default? PS: my note of rotating 90 degrees was wrong.

adamw


8,650 post(s)
#05-Jun-19 14:51

Can we have the GDB and SHP+PRJ? I tried to reproduce the XY issue and I could not - I exported a PRJ for EPSG:2232, attached it to example SHP and it imported with XY axes, not YX (from what I see, EPSG:6428, EPSG:2877, EPSG:3502 would all have behaved the same, although I didn't check them).

Mike Pelletier


1,618 post(s)
#05-Jun-19 17:49

Here's a shapefile of Colorado Counties that you can easily match up with Google street map. By default it swaps the XY. Assigning EPSG:2232 aligns it correctly but interestingly when I check the metrics it reads meters instead of feet. The shapefile is definitely in feet and so it would be good if it defaulted to feet if possible.

Not sure the "XY" button is the best graphic for opening up the metric dialogue. Perhaps it should say "Metrics"?

Attachments:
ColoradoCounties.zip

adamw


8,650 post(s)
#06-Jun-19 09:07

Thanks a lot. This is our bug, we will fix it. Apologies for the inconvenience.

danb


1,708 post(s)
#04-Jun-19 20:03

I'm very exited to see the new watershed tools and the promise of additions to these in upcoming builds. I was however busy being bowled over by the speed of the watersheds function when on my first outing I encountered ...

This limit has been a real pain for various experiments and processes that I have undertaken with M9. Is there any chance it can be removed or a parameter for a user defined limit added? I don't mind waiting a bit longer for it to complete as everything else is so fast

PS will it be possible to get at the intermediary grids flow direction, flow accumulation etc as these are useful to participate in other grid processes and calculations such as Topographic Wetness Index etc?


Landsystems Ltd ... Know your land | www.landsystems.co.nz

tjhb

8,846 post(s)
#04-Jun-19 23:26

Dan, can you give details? Size of image, proportion of invisible pixels if any, and especially (assuming you are using the *Par version) the number of threads used?

What happens if you try specifying a very high number of threads, e.g. 512? They won't all get work at once--it will reduce effective parallelism--but it should (I hope) reduce the object count per batch.

danb


1,708 post(s)
#05-Jun-19 05:02

Hi Tim,

No invisible pixels, 1x1m lidar grid of dimension 2637x5840. I used a value of 500px to initiate a stream as anything much lower yielded the 'too many areas' message.

On my machine ...

SystemCpuCount():

2019-06-05 15:51:08 -- Transform (Watershed Areas): [fill_tonga] (38.009 sec)

2019-06-05 15:52:06 -- Transform (Watershed Lines): [fill_tonga] (28.376 sec)

512:

2019-06-05 15:42:18 -- Query: (Command : CREATE ...) (74.006 sec)

6:

2019-06-05 15:45:32 -- Query: (Command : CREATE ...) (24.055 sec)

Whichever way you look at it, a superb addition.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

danb


1,708 post(s)
#05-Jun-19 05:46

PS The 512 and 6 core tests were watercourse timings.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


8,650 post(s)
#05-Jun-19 14:52

We'll remove that limit - both watersheds and tracing will be able to trace an arbitrary number of areas (well, up to 2 billion, until we raise that further).

Also, yes, we'll likely add functions to output flow direction / accumulation, as well as functions to build watersheds from those.

atrushwo46 post(s)
#04-Jun-19 21:50

I'm also very excited about the new watershed features. I'm a bit confused about the new algorithm though. Can someone please explain to me why the watershed lines terminate? Blue lines are M9, Red lines are M8. My obvious preference is for the watershed lines to be continuous.

Attachments:
Capture.JPG

danb


1,708 post(s)
#04-Jun-19 22:24

A guess. Because you are not using a filled DEM as a source? I will try this now and report back if it makes a difference.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

danb


1,708 post(s)
#04-Jun-19 22:51

That was it. Unfilled DEM on the left filled DEM on the right, same streamline generation parameters.

Attachments:
Watercourse M9.jpg


Landsystems Ltd ... Know your land | www.landsystems.co.nz

tjhb

8,846 post(s)
#04-Jun-19 23:04

Nice!

StanNWT
140 post(s)
#04-Jun-19 23:19

Hey Tim,

How about we both try and use the new watershed operation on that big DEM that I sent you? See if something breaks?

tjhb

8,846 post(s)
#04-Jun-19 23:28

I'm in. Not sure I want to fill sinks on that though (until it is built in to 9). Might put up with breaks for now.

adamw


8,650 post(s)
#05-Jun-19 15:03

Yes, I'd wait until filling sinks and until we remove the limit for traced areas (which affects watersheds because they call a variant of tracing areas internally) - we have both things in the works right now, it should not take too long.

tjhb

8,846 post(s)
#04-Jun-19 23:51

P.s. Dan, the way sinks have been filled for the crater lake looks false in this case (your attachment), in a significant way. Downstream flows and orders would be allocated very incorrectly.

(I think: the crater is a true sink.)

danb


1,708 post(s)
#05-Jun-19 01:49

Hi Tim, It was a quick and dirty example to test if this was the problem. I filled the sinks with no threshold value so completely artificial in this case.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

atrushwo46 post(s)
#05-Jun-19 15:04

Thanks Dan! Now I'm eagerly awaiting fill sinks! Will test in M8 to try and reproduce your results.

rk
330 post(s)
#05-Jun-19 07:59

9.0.169.2 Clicking anywhere on layer in layers pane activates opacity. On/Off is unreachable.

Attachments:
169_1_on_off.jpg
169_2_on_off_opacity.jpg

Dimitri


5,519 post(s)
#05-Jun-19 09:44

For any row that has the focus (dotted outline cursor on the row), toggle off/on with the spacebar. That applies to all selected rows as well if a selected row has the focus.

BerndD

125 post(s)
#05-Jun-19 15:25

Thay's Ok, but might present an issue when working with a tablet using touch or a pen. I would rather see toggling layer visibility by clicking on the square icon behind the field that holds the opacity number.


The Future of Spatial Data // www.drahola.com // www.geocockpitug.de

adamw


8,650 post(s)
#05-Jun-19 15:27

That's a good idea - we can click on + / - icons for groups, so why not on the toggles. We will try to add it.

BerndD

125 post(s)
#05-Jun-19 15:37

Please, keep the space bar toggle for convenience when working on the desktop.


The Future of Spatial Data // www.drahola.com // www.geocockpitug.de

adamw


8,650 post(s)
#05-Jun-19 15:06

That's a change of controls - we used to have multiple columns with opacity / toggle being in separate columns, activated via Enter or double-click, and now we have a single column = opacity, so Enter or double-click activate it, and toggling is done by pressing Space.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.