Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.171.2
adamw


9,283 post(s)
#05-Apr-20 16:18

9.0.171.2

Here is a new build.

manifold-9.0.171.2-x64.zip

SHA256: a553070bcfd1dd3ef9f2393e5306182414d81cfb51559a1c41a35784708fd180

manifold-viewer-9.0.171.2-x64.zip

SHA256: 9b2069b63727a4e43470b6b475ef48c1b70a2b19a6fa5564fa687ef54b6d1c00

adamw


9,283 post(s)
#05-Apr-20 16:20

Functions

The TileFilter and TileFilterSum query functions produce missing pixels if the filter shape covers no valid pixels (were producing pixels with the value set to 0). This is consistent with other TileFilter functions.

There are new query functions for tile filters: TileFilterStDev / StDevPop / Var / VarPop / Count. Each function has a GPGPU variant. (The GPGPU.DAT file is updated.) Each function has a corresponding transform.

The TileGeomAvg / Diversity / DiversityIndex / Major / Max / Median / Min / Sum query functions take a channel number as a parameter, to work with a single channel in a multi-channel image.

There are new query functions for geom tile statistics: TileGeomStDev / StDevPop / Var / VarPop. The functions take a channel number as a parameter, to work with a single channel in a multi-channel image.

There is a new query function for capturing a sample pixel value under a geom: TileGeomSample. This is the fastest way to capture data for a geom from an image. The function takes a channel number as a parameter, to work with a single channel in a multi-channel image. (We might add functions which take multi-channel samples in the future.)

The GeomConvexHull query function takes a boolean parameter which controls the behavior of the function when the produced convex hull is a single point or a straight line segment. If the parameter is false (former behavior), the function returns NULL. If the parameter is true (optional new behavior), the function returns an area with coincident coordinates.

Join (drawing -> drawing)

The Join dialog allows joining data from one drawing to another (spatial overlays).

To perform a join between drawings, open a map containing both drawings as layers, make the drawing layer you want to transfer data to active, then invoke Edit - Join.

The Join dialog for drawings is the same as the one used for joining data between tables, however, the list of source components is limited to map layers. This both extends the range of possible joins in that you can use the dialog to join components from different data sources, and reduces it in that you don't see all drawings from the data source of the target drawing.

Joining data between drawings allows selecting a join condition from the list of usual spatial operations: adjacent to / contains / contained in / intersects / touches (the default), etc.

Joining data between drawings is always x:N, that is, a single target record gets values from zero or more source records - there's no uniqueness which sometimes exists with joins between tables. Consequently, transfer rules are always aggregates - min / max, etc.

Joining data between drawings uses multi-threaded GeomOverlayXxx function variants.

The update query, if saved, is created in the data source of the map to allow joining data between any pair of layers. (If the update query was created in the data source of the target drawing, the source layer might have been unreachable.)

Join (image -> drawing)

The Join dialog allows joining data from an image to a drawing (an extended version of the transfer heights feature in 8).

To perform a join from an image to a drawing, open a map containing both components as layers, make the drawing layer active, then invoke Edit - Join. The Join dialog will allow joining data from either a drawing or an image.

The only available join condition is: contains, but the name of the condition is separate from the similar condition used with joins from one drawing to another, so that it can be localized differently.

When joining data from an image, you select a channel instead of a field. Joining data from an image to a drawing is always x:N, that is, a single target geom gets values from zero or more source pixels. Consequently, transfer rules are always aggregates. The list of available aggregates is similar to that available for numeric fields, except the 'copy' rule (which for an aggregate means 'use any value out of multiple values from multiple matching records') is replaced with 'sample' (which does the same thing, just faster).

Joining data from an image to a drawing uses multiple threads through a nested SELECT with THREADS.

The update query, if saved, is created in the data source of the map to allow joining data between any pair of layers.

Miscellaneous

(Fix) The Stack Vertical command in layout window no longer stacks frames in bottom up order instead of top down order.

The ValueFieldChannelF1 localization string is replaced with ValueFieldChannel to force unique channel names even if the localized version of the string messes up and omits the {0} escape sequence for the channel number.

The Join dialog can be used on a persistent query, if the result table is writable. This allows joining data to, for example, a table alias (TABLE [datasource]::[table]) or a selection in an existing component (TABLE CALL Selection([component], true)). (You cannot join data to an ad-hoc query shown in a Command Window because that query has no persistent name and, as such, cannot be addressed from an update query.)

The Join dialog allows adding multiple fields at once using the Fields command (shows a dialog) in the dropdown menu for the Add toolbar button. The number of fields listed in the dropdown menu is limited to 20.

The Join dialog automatically copies coordinate systems for the new geom and tile fields from their specified source fields.

The Center and Convex Hull transfer rules now work for all types of geometry (were working for lines and areas).

End of list.

Things we are working on: registration / labels / join into an image.

vincent

1,913 post(s)
#05-Apr-20 21:54

Is there anything yet to modify pixels values based on areas from a drawing ?

Before using all the hydrologic tools, the DEM needs to be modified to take account of bridges and culverts. I'm doing it in M8 with Transfert Selection and an update query. Any way to do that in M9 ?

LandSystems23 post(s)
#05-Apr-20 22:16

Things we are working on: registration / labels / join into an image.

I was hoping that this might be a reference to it.

adamw


9,283 post(s)
#06-Apr-20 07:32

Yes, modifying pixel values based on geoms from a drawing is part of joining data into an image, and will appear in the near future.

LandSystems23 post(s)
#06-Apr-20 08:37

That's great news. Thanks so much for confirming this

atrushwo71 post(s)
#06-Apr-20 15:30

Morning Everyone,

Just took a set of watershed lines and attempted to use the join to image command to pull the maximum Channel 0 elevation from my raster surface. It mostly worked? About 10% of the watershed lines received a null value despite being contained by the raster. Oddly enough, the majority of the lines with missing values are at a -45, -135, 45, 135 degree bearing.

Am I doing something wrong or is this a bug?

PS. The file is over 170MB. I'd have to do a different test to post a sample.

Dimitri

6,064 post(s)
#06-Apr-20 16:09

Try it with watershed areas to see what you get. Note that it is the drawing (point, line, or area) that "contains" pixels in the raster.

atrushwo71 post(s)
#06-Apr-20 16:43

Watershed areas works fine. The issue is related to line objects.

adamw


9,283 post(s)
#06-Apr-20 18:13

We'd be very interested in the test file. You can upload the 170 MB file via FTP - send an email to tech support and say that you want to report a bug and have a big file to upload, and they will send you the login info.

We'll do our own testing as well.

Thanks for the note.

atrushwo71 post(s)
#06-Apr-20 20:50

Hey Adam,

I couldn't figure out tech support's instructions on how to use the FTP site... I put it on my dropbox instead. I'll leave it up for a few days then take it down.

https://www.dropbox.com/s/2u1n3u9bhvghleu/M9_Streamline_Join.map?dl=0

Dimitri

6,064 post(s)
#07-Apr-20 05:58

For future reference, using FTP can be easy or difficult depending on the FTP client you use. FileZilla and similar free FTP clients make it pretty easy. I recommend using an FTP client and not a browser, because good clients are way more efficient and convenient.

Tech support gives you the four items you need: the name of the Host (some clients call it the "Server"), the Port number (almost always 21), the Username (some clients call it the "Login"), and a Password.

To connect with FileZilla, in the illustration below just enter those in the boxes just below the main toolbar and press Quickconnect. You can then upload the files you want.

Attachments:
filezilla.jpg

atrushwo71 post(s)
#07-Apr-20 19:05

Thanks Dimitri, This is very helpful.

adamw


9,283 post(s)
#07-Apr-20 11:07

Thanks a lot for the file.

We found the issue. It was a bug on our side - some line segments were ending up being ignored.

We fixed the bug, the fix will be available in the next build. The number of lines in the file with NULL average height after the fix is 0 (was: 2875). The difference in average height between the old version with the bug and the new version without the bug, on lines where the old version was returning a non-NULL average, is not zero, but pretty small - around 0.003%.

atrushwo71 post(s)
#07-Apr-20 18:29

No problem, glad you figured things out.

Mike Pelletier


1,749 post(s)
#06-Apr-20 17:54

It's great to see how much mileage your getting out of the join dialog. Wow.

One thought is to add a the ability to use Contains Inner Centroid to list of geom joins. This is a common one and I believe is included in Arcmap's dropdown list. It's useful especially when geometry is not exactly drawn.

Basically its a qualifier on the source geom. Perhaps it would be possible to alter the dialogue to allow qualifiers on the source and target with minimal complexity to the UI and great extension of its utility.

adamw


9,283 post(s)
#06-Apr-20 18:15

We'll look into it. A new join condition would need a new pair of spatial overlay functions, but why not, if the condition is useful.

Mike Pelletier


1,749 post(s)
#07-Apr-20 19:38

Sounds good but what about the concept of adding qualifiers to the target and source to increase its flexibility? So instead of creating a new join condition, for example the current dialogue allows you to take a district layer, add a [total parcels] field, then join the sum number of parcels contained within each district. Great but now add the qualifier, that it contains the inner centroid of the parcel or maybe even majority of parcel. Another example would be to join the length of each roads that lies within each district.

The join dialogue is really awesome and they say no good deed goes unpunished right! :-)

For that matter, consider adding the ability to join multiple tables. Of course all this increases the complexity of the dialogue, so perhaps these additional tools are only shown after checking a box to show them.

With some component organization work by the user, the join dialogue also provides similar function as the Viewbots from 8. Cool.

Maybe add a checkbox to create a new drawing/table instead of modifying existing table. That would be handy.

adamw


9,283 post(s)
#08-Apr-20 08:15

Additional qualifiers for target and source objects can be viewed as part of a complex condition (eg, source object S matches target object T if T touches S and - a qualifier - if the part of S within T is more than 50% of S). I guess we could add a separate dialog specializing the condition via such options - a single combo with a condition name is perhaps too limiting because there might be multiple qualifiers, some may have numeric coefficients. This needs some thinking through on our part.

You can currently join data from multiple tables in multiple steps. Eg, suppose we have a table of districts and a table of parcels and there is also a table which contains records saying that the data for parcel X were reviewed on year Y. If we want to determine which district contains parcels which were not reviewed for the longest time, we can do it in two steps: (a) join last review year (max) from the review table to the parcels table using a table-to-table join, then (b) join first review year (min) from the parcels table to the districts table using a drawing-to-drawing join - and then we'll see which districts contain parcels which weren't reviewed the longest. If we then find that we want to perform this analysis often and would prefer to do it in a single step, that's also fairly easy - presumably we generated update queries for both (a) and (b), so we just write a third query which first runs the query for (a) and then the query for (b): TABLE CALL [query-a]; TABLE CALL [query-b];. Or, alternatively, we merge the text of the queries into one via copy / paste.

As regards generating a new component instead of modifying an existing one - in general we think it is better to have multiple tools that can be used in combination, with each tool focusing on a separate thing, rather than trying to make every tool do everything, so this might be a bit of an overkill. We'll see.

Mike Pelletier


1,749 post(s)
#08-Apr-20 19:03

Okay, thanks for the consideration of these items.

Another thought is putting a geom tolerance box in the dialogue to allow folks to relatively easily experiment with different settings. This would help when dealing with inaccurately drawn objects (often the case), similarly to the inner centroid qualifer mentioned above.

dyalsjas
148 post(s)
#09-Apr-20 16:59

I like Mike's idea to add qualifiers to the Join dialog.

I'd also propose expanding the Join dialog to support the various SQL operators supported by Manifold.

Adding (dynamic) rows to the Join dialog that encompass SQL operators could resemble the Sort and Filter dialog in MS Excel, or the old Database Commander software from Manifold.

As an example, I want to update the values in common data fields within several tables.

After updating the fields in table A, with values from Table B, I would like to further update some of the remaining null values within the updated fields in Table A, with values from Table C.

The current join dialog doesn't seem to support that yet.

A join from Table A to Table C replaces the values in Table A that had been updated with the values from Table B.

apo106 post(s)
#06-Apr-20 19:45

I agree on the common use of the Inner Centroid approach in may aggregation issues on my side. I recently generalized the introduction of a inner centroid geom columns in my drawings for this purpose.

I tried a bit more the joins and find this new way to do the thing perfect. Just one thing. Once I selected a list of new fields to join I saw that it is possible to select them (turning them to red or purple if they are new and selected). In M8 selecting elements and modifying a value replicate the change to all the selected records. I would found this useful in the joined list of fields in order to set the join pattern (sum, copy,...) at once for all the selected new fields, just an idea.

adamw


9,283 post(s)
#08-Apr-20 08:23

We'll try to apply the newly selected source field / transfer rule to all selected fields in the Join dialog - that's a good idea. Thanks!

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.