Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.169.3
adamw


10,447 post(s)
#14-Jun-19 17:32

9.0.169.3

Here is a new build.

manifold-9.0.169.3-x64.zip

SHA256: 798e4901e93ddb8fe38ce5c7afbbd2f99c832600bcebab75f4553d3243a9d191

manifold-viewer-9.0.169.3-x64.zip

SHA256: 2581ddd1a9c0786e90e905212a713274f7c975afc5b220192bca0d2c41089605

adamw


10,447 post(s)
#14-Jun-19 17:34

(We have changes for layer groups coming in the next build. We are going with folders instead of leading layers, to have the same concept of folders apply everywhere where we have hierarchy.)

Changes

Clicking a toggle icon for a layer in the Layers pane toggles the layer (and all selected layers if the clicked layer is selected) on and off.

Updating image style from the Style pane accepts changes made both to image palette / channel map and to shading. Shading controls are enabled even if the image is rendered using a channel map with shading not applied.

Switching cursor mode in a layout window clears active layout frames.

Alt-click, Alt-drag, Ctrl-click, Ctrl-drag reset cursor mode in layout window. (Inserting a new layout frame and then Alt-clicking an existing frame will activate the existing frame instead of inserting one more new frame. Same for Ctrl-clicking and Ctrl-dragging which will start selecting frames.)

Alt-click, Alt-drag, Ctrl-click, Ctrl-drag reset cursor mode in map window. (The behavior is the same as for layout frames. In addition, Alt-clicking a record in drawing / image / labels, then Alt-clicking a different record will put the new record into the Record pane. Same for Ctrl-clicking and Ctrl-dragging which will clear the Record pane and start selecting records.)

(Fix) Dragging the first coordinate of an area branch in a map window no longer fails to move the matching last coordinate of the branch.

Dragging coordinates of geometry values in a map window only allows dragging the focused coordinate. To drag a coordinate other than focused, first click it to make it focused. (This protects from accidental drags and makes drags more predictable.)

Painting coordinates of an edited geometry value in a map window always puts the focused coordinate on top of all other coordinates.

Painting an outline of a geometry value in a map window uses halo for line / area outline, thick border and halo for the focused coordinate.

Painting an outline of a geometry value in a map window paints a box around the whole value if it is too small at the current zoom, to make it more visible. The additional box is only painted for existing records when the coordinates are not being edited (the Coordinates tab in the Record pane is not active).

Painting a snap reticule in a map window uses halo.

The Record pane disables moving to the previous / next record if it is editing a new object.

Moving to the previous / next record in a layout or map window from the Record pane pans / zooms the window to the new record. If the record is already completely visible (although possibly small), the view is not altered. If the record is small enough to fit into the window at current scale, the window is panned to include the record. If the record is too big to fit into the window at current scale, the window is zoomed to the record.

Alt-clicking in a drawing / labels layer in a map window picks up to 10 records under the clicked location. The first picked record is put into the Record pane, other picked records can be moved to using Move to Next. (That is, if there are multiple picked records, the virtual record order used by the Record pane is altered to put the picked records first and unpicked records next.)

Moving to the previous / next record in a map window from the Record pane uses a background thread to (show the progress dialog and) allow canceling the move if it takes too long. (This is useful for slow data sources and for components made on queries.)

The Record pane allows moving only to selected records using a toolbar button. (Turning the mode on keeps the current record unchanged, but moving to the previous / next record will only move to selected records. If there are no selected records in the specified direction, the current record stays unchanged.)

The Record pane shows and allows editing pixel values in tiles. Pixel values are shown as X - Y - value (possibly multi-channel) - toggle that indicates whether the pixel is present or missing. Pixel values can be selected, values can be edited and toggled, editing or toggling a selected pixel value alters all selected pixel values. Changed pixel values are highlighted. (Saving changes to the edited tile will currently require updating intermediate levels manually in the map window - we will perhaps automatically update them if they were updated before the changes.)

The Record pane shows field names in the list of field values as readonly.

(Fix) Writing multi-point geometry values with multiple branches to SQL Server no longer fails.

Composing contours for many heights (thousands+) performs much faster.

Composing traces for many values (thousands+) performs much faster. The limit onto the number of unique values is removed.

New query functions:

  • TileWatershedAreasSinks / TileWatershedAreasSinksPar - takes an image, returns watershed areas for all sinks (no minimum flow parameter necessary). TileWatershedAreasSinksPar is a parallel variant, takes thread configuration as an extra parameter.
  • TileWatershedAreasUpstream / TileWatershedAreasUpstreamPar - takes an image and a drawing of points, returns upstream watershed areas for each point. TileWatershedAreasUpstream is a parallel variant, takes thread configuration as an extra parameter.
  • TileWatershedAreaUpstream / TileWatershedAreaUpstreamPar - takes an image and a coordinate, returns upstream watershed area for the coordinate. TileWatershedAreaUpstreamPar is a parallel variant, takes thread configuration as an extra parameter.
  • TileWatershedLinesUpstream / TileWatershedLinesUpstreamPar - takes an image, a drawing of points and the minimum flow value, returns upstream watershed lines for each point. TileWatershedLinesUpstreamPar is a parallel variant, takes thread configuration as an extra parameter.
  • TileWatershedLineUpstream / TileWatershedLineUpstreamPar - takes an image, a coordinate and the minimum flow value, returns upstream watershed line for the coordinate. TileWatershedLineUpstreamPar is a parallel variant, takes thread configuration as an extra parameter.

New transform: Watershed Area, Sinks - computes watershed areas for all sinks.

(Fix) Parsing a coordinate system from WKT (PRJ) no longer sometimes erroneously returns a system that uses YX axes.

(Fix) Exporting a TIFF or ERDAS IMG no longer sometimes fails when writing data for intermediate levels.

Exporting a TIFF or ERDAS IMG skips writing data for intermediate levels if the image is too big (intermediate levels will strain the disk) or too small (intermediate levels are unnecessary, easy to compute on the fly).

There is a new dataport for web CSV, similar to web GeoJSON, to allow easy linking of web URLs with CSV data.

The Options dialog allows specifying a value for the cache size. The default is '(auto)' and that can be overriden with an explicit value in GBs. The minimum allowed value is 4 GB (which is the default) and the maximum allowed value is 64 GB, however the dialog only lists values up to the installed physical memory. The option only applies to 64-bit mode, the 32-bit mode ignores it. Any changes to the cache size are only applied after restarting the application. (This allows increasing the cache size in the 64-bit mode, which could help performance of some operations. Increasing the cache size can also hurt performance depending on the scenario. Generally, it makes sense to keep the cache size no bigger than half of the installed physical memory, perhaps slightly lower than that if you are regularly running multiple instances of Manifold. Whenever in doubt about the effect on performance, use '(auto)', this is conservative and safe.)

End of list.

tjhb
10,094 post(s)
#15-Jun-19 01:48

It's going to be pretty hard to go back from this build, isn't it? A situation where, once you know, you can't unknow.

So many things to test and try out, converging from many development paths, from ergonomics and usability to completely new tools. A watershed I think. (And so much work! Man. Thanks.)

[Oh and, as always, vast understatement. E.g. "The limit onto the number of unique values is removed."]

adamw


10,447 post(s)
#15-Jun-19 15:15

Thank you. :-)

The optimizations in contours / traces are indeed pretty significant, and watersheds benefit a lot from them as well.

Speaking of the limit on the number of unique values in traces, there is still a limit of 2 billion records in output. If the number of produced traces / contours / watersheds exceeds 2 billion, the operation will still fail. However, that's a much higher limit than 20,000 unique values. This limit of 2 billion is something that we have in several places, most notably in tables in MAP files - any single table can only store up to 2 billion records. We are going to remove that limit in the future, too.

joebocop
514 post(s)
#15-Jun-19 17:22

There is a new dataport for web CSV, similar to web GeoJSON, to allow easy linking of web URLs with CSV data.

Wow, you guys are fast, thank you again; my 5mbps internet connection thanks you for the CSV option over GeoJSON!

One detail that might be useful is the ability to assign a name to the table component exposed by the csvserver dataport. The connection string is URL-encoded and contains a SQL query as a parameter, so the table component name ends up being long and weird, making it somewhat unwieldy to query.

Here is a screen capture of the "same" (except for the format parameter) URL using geojsonserver (Features) and csvserver (yikes).

At any rate, thank you again.

Attachments:
comp_name.PNG

adamw


10,447 post(s)
#16-Jun-19 10:36

Erm, this is a bug - the parameters should have been cut and the table should have been named after the object (and if there is no object, it should have been named 'CSV'). We'll fix it.

tjhb
10,094 post(s)
#16-Jun-19 03:50

I am confused about what the TileWatershedAreasSinks/Par functions and the Watershed Areas, Sinks transform do.

The first DEM image I tried the function on was too ambitious, ~90000 x 100000 pixels. I set up the transform, pressed Edit Query, made no changes to the query and ran it. It did eventually complete, in 40873.545 sec (11h 21mn 14s), but the resulting table and its drawing were blank. No errors.

(I note in passing that I had increased the maximum cache size in advance of that test to either 8 or 16 GB, I apologise for not having a note of which. I reset to (auto) afterwards.)

I have had more success with a smaller image, 8000 x 8000 pixels. This is raw one-second SRTM data for part of the central South Island of New Zealand. The automatically-generated query completed in just 63.952 sec (1mn 4s), and was fully populated--almost. One large lake and one of two areas of sea were left blank (no areas).

But putting that aside, I am confused as to what the result drawing is showing. What I had expected from the release notes is that only the sinks would be processed, and the result would contain areas that represent them (as if they were all lakes, which many are though not all of course). Then, either by manual flooding or perhaps an upcoming new function, we could fill some or all of those sinks to enforce complete drainage.

Instead, I get an almost full coverage of the image, with the exception of that one lake and the western portion of sea. They can't all be sinks, can they? That would mean that nothing drains at all:

Contrasting this with the result of performing watersheds on the same image shows that at least most catchments do drain (with the default minimum flow value of 100):

(BTW notice the two blank squares in the sea here, at the extreme top left and bottom right.)

I also don't understand what the Value returned by TileWatershedAreasSinks/Par represents. Nor even, actually, why no minimum flow parameter is necessary for the sinks functions--given that they seem to produce watersheds with almost complete coverage.

So I don't understand much!

The two tests were on different machines, both with 32GB RAM and 1TB SSD TEMP space. The first test (11 hours, blank result) was on a laptop running Windows 10 1809 17763.557. The second sink test (8000 x 8000, fast) as well as the Watershed test shown above were on a desktop running 1809 17763.503.

Attachments:
SRTM south Watershed Areas 100 Drawing.png
SRTM south Watershed Areas, Sinks Drawing.png

tjhb
10,094 post(s)
#16-Jun-19 04:07

One more thing, in case it's important. The first test used a DEM image converted in 9.0.169.3 from a Manifold 8.0.27 project. The converted table therefore has a Levels field and both kinds of tile index, i.e. the old Level_X_Y_x BTREE and the new X_Y_Tile_x RTREE. (I have sometimes wondered why the Manifold 8 converter still produces both.)

The images in the second project were freshly imported from GeoTIFF into 9.0.169.0 (not .2) and therefore have no Levels field and only the new kind of tile index.

adamw


10,447 post(s)
#16-Jun-19 10:57

That shouldn't matter. (Although we'll check if it does, thanks for mentioning it. We'll get rid of the level field when migrating images from MAP files created by 8, too - we wanted to do that for some time now.)

tjhb
10,094 post(s)
#16-Jun-19 22:40

I don't think it does matter: I repeated the first test (90k x 100k) with data freshly imported from ESRI FLT format, so that it had no Level field and just the new index, and it gave the same blank result. This was also on the second machine (the desktop), which probably makes hardware and minor software differences look insignificant as well. (It was quicker--6h 42mn.)

You have said that the SRTM result looks strange. This blank result also seems strange, in the opposite direction, since while SRTM data (1-second, version 3) has been largely corrected for hydrology (waterbodies and major rivers were stepped down monotonically), this other DEM has not had river hydrology corrected at all. (I know because I made it.) I was expecting a plethora of sinks here. (By the way, this new tool will be hugely useful for quality control and error correction.)

hugh
200 post(s)
#17-Jun-19 04:06

A few days ago I had the same result trying to do Transform (Watershed Areas, Sinks) with a very large dem image. After many hours it completed but the result was an empty image component.

2019-06-15 02:54:05 -- Transform (Watershed Areas, Sinks): [Combo-start-Imgn47w115_13 2] (3030.378 sec)

Log attached. Note that I ran this on a laptop with no Nvidia GPU

The MXB file is temporarily available here:

https://c7riochico.net/downloads/SW-Montana_hugeDemR9_one-demimage.mxb 2.4 GB

On a very much smaller DEM no problem and the attached shows comparison of watershed lines transform with streams sinks filled in 8 (both flow 100). This is a small file with comparisons:

https://c7riochico.net/downloads/SmallPartBitterrootDEM_R9.map

Manifold 8 file:

https://c7riochico.net/downloads/SmallPartBitterrootDEM_m8.map

Attachments:
log1.txt
Streams_m8m9comparison.jpg

adamw


10,447 post(s)
#17-Jun-19 09:06

Thanks a lot, this is very useful. We'll look into why the transform fails to produce anything on a big image and we'll use the smaller images to double-check the results of the transforms vs 8 - after we are done with means to fill sinks in 9.

adamw


10,447 post(s)
#17-Jun-19 12:21

For some reason, the link to the MXB file above reports that the file is 151 MB. Attempting to download the file creates a 151 MB MXB, and that file then refuses to open.

You can upload the full file to our FTP server, if you want. Ask tech support for instructions if you need them.

hugh
200 post(s)
#18-Jun-19 14:59

The file did not upload completely first time and I did not realize. Should be complete now at the same link:

https://c7riochico.net/downloads/SW-Montana_hugeDemR9_one-demimage.mxb

If this still does not work I will ask for FTP info though that will take a while too with slow internet here.

adamw


10,447 post(s)
#19-Jun-19 11:54

This worked, thanks. (A 2.4 GB MXB that expands into a 5.1 GB MAP that contains a big image. Will check what happens with watershed areas for sinks.)

hugh
200 post(s)
#19-Jun-19 17:25

The file should also contain this component:

Combo-start-Imgn47w115_13 Table 2 Watershed Lines Drawing

As happened with Tim's largest DEM, it is an empty component produced at completion of this transform:

2019-06-15 02:54:05 -- Transform (Watershed Areas, Sinks): [Combo-start-Imgn47w115_13 2] (3030.378 sec).

adamw


10,447 post(s)
#16-Jun-19 10:52

The results look strange, indeed. We'll investigate. (It looks like there is some mix-up in the internal machinery of watersheds and finding watersheds for sinks decided to find all watersheds with no limits onto the minimum flow instead.)

Regarding no errors but no results when running on a 90k x 100k image, was there anything in the log? We'll do our own tests, but if you remember, that'd help (you can try checking the log files for the test date).

Regarding blank squares in two corners on the second screen - does the image have data in those corners? If it does, could you share the MXB with the image so we can take a look?

The Value returned by TileWatershedAreasSinks/Par should be total flow. The minimum flow parameter is not needed because the function is supposed to compose a single watershed for each sink no matter what the total flow for that sink is.

Thanks for the notes!

tjhb
10,094 post(s)
#16-Jun-19 22:50

Regarding no errors but no results when running on a 90k x 100k image, was there anything in the log? We'll do our own tests, but if you remember, that'd help (you can try checking the log files for the test date).

Nothing unusual in the log. The log for the first 90k x 100k test is attached as Log1. For the repeat test on the same image last night, I have attached the log as Log2 and the query used as Query2. This is the auto-generated query, unedited (although I had used Options to adjust the name of the output table and drawing).

Everything finished normally both times; the query Results panel said Result: 0.

Attachments:
Log1.txt
Log2.txt
Query2.sql

danb

2,064 post(s)
#16-Jun-19 23:12

Thanks for following up on this Tim/Adam. I was equally baffled by the results.

Could I once again however ask if we might be able to have the option to derive a filled DEM layer (and also optionally to specify a fill threshold in DEM Z units) as these intermediate images are useful for other tasks. For example we use the sinks (filled DEM - raw DEM) as a component of identifying areas where water may naturally accumulate. This is with a view to identifying wetland areas, wetland restoration and areas with the potential for performing surface water cleaning functions.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

tjhb
10,094 post(s)
#16-Jun-19 23:57

Dan, I agree that a function to automatically fill sinks will be very useful (we can do it ourselves, with some work). But if I understand it, it is the watershed sink areas that are the intermediate step--they will be used to fill the DEM sinks on the image (by fast painting, like transferring heights from drawing to surface in 8).

The sink areas themselves will also be useful for other purposes, such as identifying possible wetlands as you say, or finding unexpected errors in a DEM. Sometimes a difference image will be more useful, sometimes the sink areas will be more flexible and faster.

danb

2,064 post(s)
#17-Jun-19 04:06

But if I understand it, it is the watershed sink areas that are the intermediate step--they will be used to fill the DEM sinks on the image

Tim are you able to expand on this a little as I currently don't understand it? I can appreciate the utility of the sink catchments, but not how they are used to fill the sinks for watercourse/shed processing.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

tjhb
10,094 post(s)
#17-Jun-19 06:07

I'll try, but will probably just show that I don't understand correctly myself.

What do sink watershed areas show (when things are going perfectly)? The boundary of each such area delineates a sink. If a drop of water falls inside the boundary, it will flow into the sink and never drain (unless there is an underground pipe or a pump). If water falls outside the boundary, it will flow somewhere else--usually through a series of other watersheds until finally it drains to the sea. Or this rainwater might still be "unlucky", and wind up flowing into a different sink somewhere further downhill.

The boundary of a sink watershed is like the boundary of any other. It defines which way rainwater will flow, into the catchment, or away from it. And as for any other watershed, the boundary will normally go up and down, following a series of ridges and saddles (and possibly flat areas).

A normal watershed drains from the lowest point on its boundary. To make a sink watershed drain (artificially perhaps), we likewise need to raise the minimum height of all points within it, up to the height of the lowest point on its boundary.

The lowest point on the boundary is pretty easy to measure, and then it is easy to raise all heights within the sink watershed (or catchment) to that minimum height (by painting pixels). Then the sink is filled and will drain.

[Edited a bit.]

danb

2,064 post(s)
#17-Jun-19 07:40

Thanks Tim, understood, but it still feels a bit back to front to me. I guess the order of fill sinks > flow direction > flow accumulation etc etc is too ingrained.

I would however still like access to both without having to search for the lowest point in the poly.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

tjhb
10,094 post(s)
#17-Jun-19 08:00

It always iterates once I think. Flow direction, then watershed sink areas. Then fill sinks. Then flow direction, flow accumulation, then watershed areas and/or streams with threshold.

(Or watershed areas can be done without filling sinks, roughly the same as steps 1&2 only.)

adamw


10,447 post(s)
#17-Jun-19 09:16

To clarify, the functions / transform to find watersheds for all sinks appeared before the functions / transform to fill sinks not because we intend to find watersheds for all sinks as an intermediate step during filling - we do not need to extract geometry to fill. There is plenty of common machinery, however, that's true.

adamw


10,447 post(s)
#17-Jun-19 09:10

Yes, of course, we will have means to fill sinks (take an unfilled image and produce a filled image). This is in the works, likely coming in the next build.

tjhb
10,094 post(s)
#16-Jun-19 23:44

Regarding blank squares in two corners on the second screen - does the image have data in those corners? If it does, could you share the MXB with the image so we can take a look?

Yes, it does contain data in those squares. They should be all zeroes, and the image Style shows them as such (e.g. I can map 0 to blue, and the sea is blue).

First, bear in mind that for this SRTM image (but not for the image used in the other test), I am using a "virtual" crop so to speak (very handy), i.e. setting the Rect property for the image to an 8000 x 8000 pixel subset of the data from its table.

The watershed areas in the result drawing extend partway into the upper left and lower right tiles; the rest of each of those tiles has no watershed areas (the blank corner squares).

Here is something curious. I am using the new Alt-click tool to inspect each corner tile and reading from the Pixels tab in the Record panel. The visible data in both corner tiles should be all zeroes, and there should be NULL values for edge pixels outside the image Rect. That is the case for the lower right tile. But for the upper left tile, the Pixels tab shows all positive values in the range ~500-700, no zeroes and no NULLS. That is also the case for other zero-filled tiles along the left-hand edge of the image... and clicking on many tiles wholly on land (all values > 0), the Pixels tab often shows many zero values, sometimes all zeroes. I don't know what to make of this.

Another small oddity while using this tool. I have a map containing the watershed areas drawing (top) and the source image (bottom). I make the drawing invisible, then attempt to Alt-click tiles in the image, but by mistake I have left the drawing tab active. Duh. But now, even though the drawing is not visible, the map display puts blue boxes around (hidden) areas in the drawing, and shows records for the areas the drawing in the Record panel. Should it? Or should the Alt-clicks to do nothing, if the drawing is active but hidden?

I will check what happens in the corners, and to the Alt-click tile readout, if I make the current "virtual" crop into a hard crop, i.e. get rid of the extra, unused tiles in the table. (By reprojecting pro-forma into a new component, a handy way to do this without export-import.) It may or may not be relevant to share the whole (uncropped) project.

tjhb
10,094 post(s)
#17-Jun-19 00:15

In the meantime here are screenshots for the upper left and lower right corners.

The "a" images are at full extent, with the upper left or lower right tile selected with Alt-click and (some of) its pixel values shown. (For the lower right the NULLs are further down the list; for the upper left there are no zeroes or NULLs.)

The "b" images are the same at close zoom, with watershed areas shown.

Attachments:
LRa.png
LRb.png
ULa.png
ULb.png

tjhb
10,094 post(s)
#17-Jun-19 00:39

Here are screenshots showing Alt-click Pixels readout for tiles entirely within two southern lakes. Lake Hawea should show constant elevation of 347m (~670-870 shown in list), Lake Tekapo constant 534m (~1700-2100 shown).

Attachments:
Hawea.jpg
Tekapo.jpg

adamw


10,447 post(s)
#17-Jun-19 09:35

We'll try reproducing this on synthetic data (Alt-clicking a tile shows wrong values in the Record pane), but it would have been great if we could have the above image, too.

tjhb
10,094 post(s)
#17-Jun-19 10:02

I would like to supply the image(s) I've had problems with, of course. But... I have found other problems, and my head hurts.

I will try to post links to the data and more explanations tonight. Otherwise tomorrow morning.

What can I say now? That I think there is a major problem with the Record tool for tiles, when the image Rect has significant offsets. But there is something else too.

I will send data yes. Difficult not to overcomplicate things more than I already have (apologies).

adamw


10,447 post(s)
#17-Jun-19 10:17

Sure, whenever you can. We'll try to find the issue on our own, too (have a couple of ideas what it might be related to). I'll report if we find it.

tjhb
10,094 post(s)
#17-Jun-19 10:44

Thanks. More soon.

tjhb
10,094 post(s)
#18-Jun-19 07:50

Found out more today, will post tomorrow.

tjhb
10,094 post(s)
#21-Jun-19 02:54

Sorry for the extra delay. I have focused the issues a bit more.

Project SRTM 1s testing c.mxb (Dropbox, 54MB) contains an SRTM image at 1 second resolution, 8000 x 8000 pixels, initially in EPSG:4326, using YX axis order.

With this coordinate system, alt-clicking on tiles gives incorrect Pixels readout in the Record pane, as discussed earlier. Further, tiles in the right quarter and the bottom quarter of the image (approximately) can't be alt-clicked.

Run query [draw tile boxes] to give a grid of areas for all tiles. The grid of tile boxes shows a large offset to the northwest.

The tiles which can't be alt-clicked fall outside the coverage of the grid.

If we use Repair initial coordinate system to assign Latitude / Longitude rather than EPSG:4326, or assign EPSG:4326 with XY axis order (or just select "Force XY axes"), then all of the issues disappear. Pixel readout in the record pane is now correct, all tiles can be alt-clicked, and [draw tile boxes] creates a grid of boxes with no offset.

On the other hand, it seems that tiles in this image can't be selected with the mouse, even after the change to XY axis order.

Also, changing axis order does not prevent the Watershed areas transform from leaving square gaps in the upper left and lower right corners. That still happens.

Attachments:
draw tile boxes.txt
XY.jpg
YX.jpg

adamw


10,447 post(s)
#21-Jun-19 13:08

Thanks a lot, this helps.

We found the issue with picking / selecting wrong tiles. It was indeed related to the YX axes. Will fix.

We found the issue with the selection in images sometimes not painting correctly. It was related to the image rect not starting at 0, 0. Will fix.

We also looked into watersheds in corners. This is expected behavior. The explanation:

The ocean areas in the top-left and right-bottom corners of the image above are both flat and not lakes (on the border of the image). Because they are not lakes, we try to compute watershed lines / areas inside of them. But because they are flat, watershed lines / areas inside of them form dense regular structures, each location on an edge corresponds to its own stream, these streams never merge and just run more or less parallel to each other.

This is common to both 8 and 9. The dense structures, however, are slightly different.

8 creates something like this (I took a different surface and zeroed out a couple of tiles in the left-top corner, then created streams with min flow of 20):

9 creates this (same data and parameters):

The configuration of streams in 9 is different because unlike 8, 9 makes streams in these circumstances follow the shortest path. As a result, some of the streams in 9 end up being shorter than 20 and are discarded. This creates a blank corner. (The empty diagonal strip is there because with a limit of 20, streams only "start" when they grow to be bigger than 20.)

The exact same thing happens with watershed areas.

As a practical matter, regions like these have to be removed before watersheds are computed. Eg, for the SRTM image for which the empty corners were reported first, it would make sense to make all pixels with values of 0 and below invisible. If the image contains pixels with these values inside the land, we could first select tiles with the ocean water and only clear pixels of 0 and below in those tiles. (We are going to provide transforms to do this easily.)

Attachments:
streams-corner-8.png
streams-corner-9.png

tjhb
10,094 post(s)
#21-Jun-19 15:34

Brilliant Adam. Thanks for the update and the explanation (and patience).

It makes sense to remove or mask ocean pixels before making watersheds. The ocean is neither a watershed nor a sink. It's impressive that the algorithms just handle these cases, but is wasted effort.

adamw


10,447 post(s)
#17-Jun-19 09:31

I am using the new Alt-click tool to inspect each corner tile and reading from the Pixels tab in the Record panel. The visible data in both corner tiles should be all zeroes, and there should be NULL values for edge pixels outside the image Rect. That is the case for the lower right tile. But for the upper left tile, the Pixels tab shows all positive values in the range ~500-700, no zeroes and no NULLS. That is also the case for other zero-filled tiles along the left-hand edge of the image... and clicking on many tiles wholly on land (all values > 0), the Pixels tab often shows many zero values, sometimes all zeroes. I don't know what to make of this.

The upper left tile containing non-zero values outside of the image is not necessarily wrong - if the image was manually cropped by editing the Rect property of the image component, maybe the tile did contain those non-zero values, cropping would not have removed them. Having valid values outside of the image rect is fine, that's a valid scenario - you are using a sub-image of an existing image. These values should not participate in any analysis carried on the (sub-)image - we will double check that they don't for watersheds just to be sure.

If you click into a tile that should contain non-zero pixels (based on what you see on the screen / according to some other logic) yet the Record pane seems to contain all zeros, that does seem strange, indeed - we'd like to see the image with a tile that does it.

But now, even though the drawing is not visible, the map display puts blue boxes around (hidden) areas in the drawing, and shows records for the areas the drawing in the Record panel. Should it? Or should the Alt-clicks to do nothing, if the drawing is active but hidden?

That's an oversight. No, if a layer is hidden, you shouldn't be able to Alt-click into it. We'll fix it, thanks for the note.

BerndD

162 post(s)
#17-Jun-19 18:30

Great job. I like the record pane a lot better now.


Organizations that want to adapt to CHANGE are using products that can adapt.

www.yeymaps.io

Mike Pelletier

2,122 post(s)
#17-Jun-19 18:45

Great to see the progress in this build.

I'm hoping there is means cooking for selecting many inflection points (coordinates) in a line or area within the drawing rather than using the list of coordinates, which is harder.

Here's a suggestion, currently clicking Ctrl ends the edit session. How about holding down Ctrl changes the cursor to a select tool that selects individual inflection points with a click and release and a click and drag makes it a freeform select tool. If you selecting a currently selected inflection point it gets deselected.

Not sure if I like the new 2 click requirement to move inflection points. Slower than before and I don't recall accidental drags as you mentioned. Haven't really used it enough to know if it was a problem though.

I used to use Mapmaker.com and always loved the speed of clicking on the line between 2 inflection points and a new inflection point gets added and you could then drag it into place all in one click + drag motion. Very fast. The current means of pressing Ins is good for adding lots of new inflection points between 2 existing ones but if I had to choose between the two tools, I think the Mapmaker method is better.

adamw


10,447 post(s)
#18-Jun-19 08:09

Yes, means to select coordinates from the map window are coming. We are just thinking about how to do it better.

(Our current thinking involves two main alternatives: (a) allow Ctrl-click / Ctrl-drag like you suggest - this was our original plan as well, making Ctrl do the same selection for coordinates as it does for objects outside of editing tools, or (b) add a separate mode to the editing tools to select coordinates and then just use normal click / drag. With (b) you'd start editing / entering coordinates, then decide that you want to select some using the cursor, activate the select mode by choosing it from the drop down in the toolbar or by pressing an accelerator key - likely just a letter with no modifiers, eg, S, select coordinates, then switch the mode back to editing, drag selected coordinates around, etc. We are going to have editing modes in some form no matter what, because we cannot fit all editing into a single mode, there are too many things to fit. So the question mainly is whether selecting coordinates with Ctrl should be available in all modes or whether it should be a separate mode.)

The requirement to first click a coordinate (or otherwise make it active, eg, by scrolling to it in the coordinate list) you are about to drag comes partly from editing geometry with tons of coordinates - when you decide "hey, I'll drag this" and start dragging, you frequently end up dragging not what you wanted, because there are multiple coordinates under the cursor or close to it. Clicking first, dragging second makes sure you are dragging what you wanted to drag.

BerndD

162 post(s)
#18-Jun-19 18:44

I think it is good as it is now.

It would be nice if we could go into edit mode with ALT/click and then we would just hover over the coordinates/vertices and it would always highlight the closest coordinate with a bigger square. Ideally this pre-selection would correspond with the coordinate list in the record pane. Once we have found the coordinate we wanted, we do a left mouse click and drag it.

Another great addition would be to hover over a polyline or a line segment showing a circle (to indicate it is not an existing coordinate) along the line and when we click on it (left mouse click) it will add a new coordinate that we can drag instantaneously. Please, also see https://openlayers.org/en/latest/examples/snap.html

I do like the idea of identifying the coordinate point first. It's safe.

Also, I would like to suggest to show the record pane content when performing CTRL/click.

That way you can make it so that with CTRL/click the focus in the record pane is "Value" whereas when clicking ALT/click the focus in the record pane can be coordinates.

It is almost second nature, that a) you want to see the values of the attribute label when clicking on an object and b) you also want to be able to edit these values right away.


Organizations that want to adapt to CHANGE are using products that can adapt.

www.yeymaps.io

adamw


10,447 post(s)
#19-Jun-19 08:23

Thanks!

The workflow in the link is neat, we like the idea of a tracking cursor used that way.

We currently have Alt-click open the Record pane in the Values tab, and Alt-Shift-click open the Record pane in the Coordinates tab. Ctrl is currently used for selection.

BerndD

162 post(s)
#19-Jun-19 14:01

Thanks for the short-cuts. My bad.

I need to dig into these more intensively.


Organizations that want to adapt to CHANGE are using products that can adapt.

www.yeymaps.io

Mike Pelletier

2,122 post(s)
#19-Jun-19 21:41

Excellent. Bernd's example is the same but a notch better than what I was referring too with MapMaker because of the dot tracking on the line.

Regarding the selecting coordinates during an edit. I think holding down Ctrl while selecting coordinates would work great most of the time and would be fast, but as you say there is lots to consider so I'm sure a good solution will be found.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.