Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.171.4
adamw


9,135 post(s)
#30-Apr-20 16:53

9.0.171.4

Here is a new build. The focus is joining data into a raster.

manifold-9.0.171.4-x64.zip

SHA256: 03432543e6844ed7cf25d22e625ca35b76d3955030a555ce1be85769ec168751

manifold-viewer-9.0.171.4-x64.zip

SHA256: 6b8c5814eac1ebddb20bd05c4db946da5289e38b165887bccf3f671618cbf6fb

adamw


9,135 post(s)
#30-Apr-20 16:54

Join (image -> image)

The Join dialog allows joining data from one image to another (raster overlays).

The only join condition available is: matches. The join operation dynamically projects the source image to the coordinate system of the target image and transfers data between matching pixels.

The list of fields contains a separate item for each channel in the target image. Any channel in the target image can receive data from any channel in the source image. If the pixel type of the source image coincides with that of the target image, the transfer method is set to: copy. If the pixel types are different, the transfer method is set to: convert.

The Add button in the toolbar allows adding new image channels, up to the limit of 4 channels per image. Newly added channels can be deleted.

The 'Clear image pixels' option specifies whether to clear the target image before transferring data or to transfer data on top of existing data (the default).

The 'Resize image' option specifies whether to resize the target image before transferring data. During the resize, the coordinate system, including the pixel size, stay the same, but the image rect changes. If the 'Clear image pixels' option is off, the image may only grow. If the 'Clear image pixels' option is on, the image may either grow or shrink.

The update query generated for joining data between images includes the code to clear / not clear image pixels and to resize / not resize the image so that these operations are repeated dynamically as part of the update.

Joining an image to itself is allowed and can be used to remap channel values. The query generated for this case is significantly simplified.

Join (drawing -> image)

The Join dialog allows joining data from a drawing to an image (vector to raster).

The only join condition available is: contained in. The join operation dynamically converts the drawing into a raster in the coordinate system of the image and transfers data between the images.

The list of fields contains a separate item for each channel in the target image. Any channel in the image can receive data from a field in the drawing. Available transfer methods are: count (the default), min, max, sum.

The Add button in the toolbar allows adding new image channels, up to the limit of 4 channels per image. Newly added channels can be deleted.

The 'Clear image pixels' and 'Resize image' options work as they do for joining data between images.

Join (general)

Selecting a transfer method for a field that is not being transferred yet in the Join dialog tries to use the source field with the same name as the target field.

Adding new fields or new image channels prior to performing the join specified in the Join dialog is done in a background thread that tracks progress and can be canceled.

Joining data into a drawing or image layer in a map using the Join dialog automatically clears the Record pane with the picked object or tile, discarding all changes, and clears field statistics used by the Style pane. (Previously both panes would be left showing outdated data until the window is closed and reopened.)

The Join dialog logs the time spent performing the join.

Functions and Templates

There are new TileValueAvg / Count / Max / Min / StDev / StDevPop / Sum / Var / VarPop query functions that compute various statistics for a single tile. All channels except the first one are ignored, similarly to other numeric functions for tiles.

The Line Coordinate transform template is reworked to return a geometry value instead of an x2 value.

The Coordinate transform template is reworked to return a geometry value, the former version of the template that was returning an x2 value is renamed to Coordinate XY. The Coordinate with Z transform template that returns an x3 value is renamed to Coordinate XYZ.

There is a new TileFillBaseLevel query function that takes an image and adds all missing records with NULL tiles for tiles within the image rect.

There is a new TileClear query function that takes a tile and makes all pixels in it invisible.

There is a new TileChannelsConcatFill query function that takes two tiles and concatenates their channels. Unlike TileChannelsConcat which only makes a pixel in the resulting tile visible if it is visible in both of the producing tiles, TileChannelsConcatFill makes a pixel in the resulting tile visible if it is visible in any of the producing tiles, filling missing channel values with zeros.

There is a new TileChannelCopy query function that copies values of a single channel between two tiles.

There is a new GeomInflateRect query function that inflates or deflates a rect (an x4 value) by the specified size (an x2 value, positive values inflate the rect, negative values deflate).

There is a new GeomInflateRectTileSize query function that inflates a rect (an x4 value) to cover whole tiles of the specified size (an x2 value, values required to be positive).

There are new TileGeomOverlayCount / Max / Min / Sum query functions with parallel variants that take an image, a drawing and a field name and produce an image with the same parameters as specified, filled with data from the drawing. The referenced field has to exist and have a numeric type. For TileGeomOverlayCount, the field name can be an empty string.

Tables returned by the CoordConvertTileSet family of query functions operate slightly faster, particularly on systems with many cores.

There is a new ComponentBounds query function that computes the bounds of a component. The function takes a boolean flag controlling whether to compute bounds for a component that is not automatically maintaining it by reading the component records. (Example components that automatically maintain their bounds: images, drawings / labels in a MAP file based on a table with an RTREE index on the needed geometry field. Example components that do not automatically maintain their bounds: drawings / labels based on queries.)

Other

(Fix) Projecting an image using the direct sub-pixel method no longer produces incorrect results if the tile XY values in the resulting image can be negative (may happen when local offsets are not set to Auto).

The Collation dialog applies the filter to the language code in addition to the language name.

Projecting an image using multiple threads limits the number of used threads to the number of CPUs x 2, to protect from trying to create too many threads when used programmatically.

(Fix) Aggregating pixel values under a geom no longer sometimes (extremely rarely) returns a non-NULL value for a geom that is outside of the image.

End of list.

gjsa80 post(s)
#01-May-20 15:25

These are great additions. Two questions:

1. Is an example provided anywhere as yet for using the TileGeometryOverlayCount /Max/Min/Sum query functions ?

2. Am I correct that generating raster statistics within polygons (zonal stats) is a function that still needs a different approach with functions that do not yet exist in M9. I've been testing exactextractr in R which runs fast, but alas does not seem to handle split, multipart geometries.

adamw


9,135 post(s)
#01-May-20 16:17

1 - You can look at the queries generated by the Join dialog for image <- drawing joins, they are using those functions. The parameters are pretty simple - an image, a drawing, and a field name. The result is a table with tiles in the coordinate system of the image, with the same tile size, covering the same rect, but with pixel values generated from the drawing.

2 - No, we already have functions for this: TileGeomAvg / Count / Max / etc. Each function takes an image, a channel number to take values from, and a geom projected to the coordinate system of the image. The result is a numeric value with the specified statistical measure for all pixels covered by the geom. For examples of using these functions, look at the queries generated by the Join dialog again, for drawing <- image joins. Branched areas / lines are, of course, supported. (As are multi-points.)

LandSystems20 post(s)
#02-May-20 01:09

Congratulations on this build, it contains components that I have been waiting specifically for and there is much to explore and get my head round.

Meantime, I have one task which I am keen to be able to achieve, but I am unsure of how to approach it and perhaps it isn't easily doable using the current infrastructure.

I have two images, of which the second is a subset of the first. Lets call them image full and image subset.

Is it possible to either:

1. Alter those pixels in image full NOT covered by image subset and set them to a new value.

or

2. Get those pixels in image full NOT covered by image subset and copy them into a new image.

In M8 I would do this by transferring selections from a polygon to the image, inverting the selection and then doing what I needed to to the selected data. I would dearly love to be able to so the same in M9 as selections are such a useful workflow to me.

adamw


9,135 post(s)
#02-May-20 09:03

Yes, this is all doable.

Here's how to do item 1.

Take the subset image and set all pixel values to 1 using the Transform pane: open the image, open the Transform pane, Fill, Fill with = 1, Update Field. If you want to keep the original subset image, you can create a copy of it in the Project pane first.

Now put both images into a map, make the layer with the full image active and invoke Edit - Join. Set the source component to the subset image, add a new channel (channel 1) and set it to accept values from the subset image (channel 0 in that image). Keep both 'Clear image pixels' and 'Resize imaze' unchecked, click Join Component.

OK, now the full image has two channels. Channel 0 contains the image data. Channel 1 contains 1 if it belongs to the subset image and 0 if it does not. We want to alter channel 0 in this way: if channel 1 contains 1, we want to keep the original value, and if channel 1 contains 0, we want to use a fill value. We can collapse this logic into the following expression: newpixel = subset * oldpixel + (1-subset) * fill. Since subset can be either 0 or 1, either the first or the second part of the sum is 0, this is a frequently used technique.

Let's try using the expression. With the full image layer active, open the Transform pane again. Switch to the Expression tab. Set the expression to:

--SQL9

TileChannelCopy([tile], 0,

     TileChannel([tile], 1)  * TileChannel([tile], 0) +

  (1-TileChannel([tile], 1)) * 500, 0)

This is the above expression rewritten for tiles plus we call TileChannelCopy to put the result back into channel 0 of the original tile.

You should see the results right in the preview - pixels in the subset image should stay unchanged and pixels outside of it should be set to a constant value (500).

Press Update Field and you are done.

Item 2 is doable as well, but we are currently reworking and extending the functions manipulating masks so I would wait with the recipes until these reworks are complete, otherwise what I write will likely become outdated soon.

Last, on selections. Currently, we only allow selecting full tiles. This is going to change and we are going to allow selecting individual pixels. The reason we didn't allow selecting pixels initially is because 9 can work with far bigger images than 8, and making pixel selections efficient for images this big is noticeably more difficult than it was in 8. But we have been working on this for some time, we have an approach that works, so right now this is simply a question of priorities. This summer (tentatively), we are planning a slight rework for images which will greatly simplify their handling in the Transform / Select panes. That's when pixel selections will likely appear.

LandSystems20 post(s)
#03-May-20 02:22

Thanks so much for responding Adam and for laying out the workflow so clearly. I will give it a try tomorrow.

Glad to hear that there are also things in the works around masks which I look forward to seeing. Also many thanks for the heads up around writing selections. This remains my number one wish list item and I am very happy to hear that they are in the mix.

The ability to use all these tools efficiently irrespective of the size of data must cause no end of headaches. I am however very glad that you are taking the time to make these things possible and with such wonderful performance. It is very much appreciated.

LandSystems20 post(s)
#03-May-20 05:18

Back again. I couldn't wait until tomorrow so I gave your workflow a go, harvesting the SQL as I went and it works perfectly (in the GUI).

If however I try running the harvested SQL, the join query runs, appears to get to the end and then fails with:

Invalid key field value.

Are you able to shed any light on what I might need to do to fix this? I am wanting to use the SQL in a script which uses slightly modified source and subset images at each iteration to derive 30 or so output images.

adamw


9,135 post(s)
#03-May-20 07:13

We need to see the schemas of all involved components and the text of the query that is failing to determine why it is failing. If you could post an MXB with the query and the example images (however small, you can delete all tiles but one in each, for example), that would help.

Alternatively, you could try opening the query and selecting and running the statements one by one (select the text of the statement, then invoke View - Run Selection or press Alt-Enter to the same effect) - in sequence, because the order is important - to see which one fails.

I suspect the image table that is being written to has a BTREE index on fields other than MFD_ID or X / Y, inserting records into such a table requires providing data for all fields used in the index, the query ends up failing to provide some of the values and failing. Or maybe the index on X / Y / TILE fields has a different size specified for tiles than the one provided in the query.

LandSystems20 post(s)
#03-May-20 07:37

Thanks for your continued assistance Adam. I will package up and mxb tomorrow. hopefully it is just me being dumb.

With regards running the queries one by one. I have done this and it is the join query that fails. It runs as expected, says that it is updating records and then gives the error when it completes. I will also have a look at the indexes.

adamw


9,135 post(s)
#03-May-20 07:35

One more thing, and this is perhaps the most likely reason for the failure.

The query generated by the Join dialog does NOT include adding a new channel to the target image. We do not include this so that the update query does not try to add a new channel every time it is run. So, if you are using 30 different *source* images into the same target image, you can get away with running the Join dialog once and then running the update query - with different image names - 30 times. But if you are using 30 different *target* images, the query you are running has to include adding a new channel.

Adding a new channel to an existing image depends on what indexes it contains. If it contains a BTREE index on X-Y fields, this is just:

--SQL9

UPDATE [skull Tiles]

  SET [Tile] = TileChannels([Tile], VectorMakeX2(0, -1));

ALTER TABLE [skull Tiles] (

  ADD PROPERTY 'FieldTileType.Tile' 'uint8x2'

);

The TileChannels call in the UPDATE, as written, composes a tile with two channels, with channel 0 set to channel 0 of the original tile and channel 1 set to zeros (-1 is not a valid channel index, so this channel is set to zeros).

If the image contains an RTREE index on X-Y-TILE fields, you have to drop and re-create it:

--SQL9

ALTER TABLE [skull_r Tiles] (

  DROP INDEX [X_Y_Tile_x]

);

UPDATE [skull_r Tiles]

  SET [Tile] = TileChannels([Tile], VectorMakeX2(0, -1));

ALTER TABLE [skull_r Tiles] (

  ADD INDEX [X_Y_Tile_x] RTREE ([X][Y],

    [Tile] TILESIZE (128, 128) TILETYPE UINT8X2),

  ADD PROPERTY 'FieldTileType.Tile' 'uint8x2'

);

Hope this helps.

LandSystems20 post(s)
#03-May-20 07:39

The query generated by the Join dialog does NOT include adding a new channel to the target image.

I did wonder about this and I haven't inserted any additional code to add this. Thanks again for the example code this is really useful.

LandSystems20 post(s)
#03-May-20 09:39

That was it. Adding the channel and dropping/reinstating the R-Tree index was what was needed. Thanks very much!

LandSystems20 post(s)
#03-May-20 22:37

A couple of additional thoughts. As in my case, the subset image is always a subset of the full image. This being the case, presumably I could actually do away with the join and just insert the subset image into channel 1 because the pixel dimension and origin are the same and the tileX and tileY also appear to be preserved in subset. I am guessing that this would be even faster to process?

Also I notice if I merge the two images using the merge images command, this is lightening quick. Is it possible to see the SQL that this command issues to the back end as this would be very useful to harvest and understand.

Thanks again for your help.

Dimitri


5,993 post(s)
#04-May-20 05:53

Is it possible to see the SQL

Not yet. Merge is one of the few dialogs that doesn't have an Edit Query button. I've sent in a suggestion to add that.

adamw


9,135 post(s)
#04-May-20 08:54

If we are talking about two different images, even with the same dimensions / coordinate systems, inserting one into another is always going to be a join, as in, tiles with the same XY have to be matched to each other. If the images have a BTREE index on XY fields, this matching is fast.

We do not generate query code for the Merge dialog because we cannot generate good code for it. The merging process frequently involves multiple components and handling a big number of different tables - instead of a big number of records or values - using pure SQL is not terribly pretty. This doesn't mean that SQL is unsuitable for handling tons of tables, it is, but that usually requires either some helper constructs (eg, in 9 we can use SQL to inspect the coordinate systems of all components in a project with ease, but that's only because we have a helper MFD_META table) or a combination of a query and a script where the script is generating a series of queries. Without such helpers it all becomes a big stream of similarly looking statements which are pretty hard to read and very hard to adjust or reuse. If the Merge dialog could generate a script or a script + query combo, we would perhaps generate it, but since all our facilities generate pure SQL, we'd rather pass.

For example code merging two images together you can use the code generated by the Join dialog. Turn on the 'Resize image' option, map identical channels to each other, that's it. You can then adjust this code to merge one more image to an existing image. This would be slightly slower than the Merge dialog because the Merge dialog merges all images simultaneously, but still fast enough.

I guess we could write an aggregate function that takes multiple images together and merges them similarly to how the Merge dialog does it, that would make generating the query for the Merge dialog trivial. But this would be a pretty new concept - an aggregate that takes *tables*. Or maybe there are other ways to express the same thing. We'll think about it.

Dimitri


5,993 post(s)
#03-May-20 19:44

There's a new example on using the Join dialog to transfer data from a drawing into a raster. It's a preliminary example but is still useful.

adamw


9,135 post(s)
#30-Apr-20 16:55

Some performance numbers for vector to raster join compared to Manifold 8.

In Manifold 8, we take a drawing, copy and paste it as a surface, setting field = ID, type = int32, pixel size = desired pixel size.

In Manifold 9, we take the same drawing, create a blank image with int32 pixels, the same coordinate system and the desired pixel size, then we join the drawing into the image, setting channel 0 = max ID, clear image pixels = on, resize image = on.

Test 1.

32 areas -> 3,000 x 2,000 pixels

8: 8.500 sec

9: 0.234 sec

Test 2.

527,000 areas -> 10,000 x 10,000 pixels

8: 341.250 sec (+130 sec if you count the time to perform Copy and then open the Paste dialog)

9: 34.278 sec

There is potential for further improvements as currently on many data sets the rasterization is so fast that using multiple threads does not bring much - this could be improved by tuning the multi-threading workloads dynamically.

vincent

1,911 post(s)
#30-Apr-20 21:08

Is there yet a way to change pixels values based on area in a polygon ? It seems to only work the other way.

adamw


9,135 post(s)
#01-May-20 07:53

Joining a drawing into an image does exactly this.

Example: you have an image and an area and you want to set pixels under an area to the value of its Z field. To do this, you invoke the Join dialog on an image, select the drawing as a source, then set Channel 0 to take data from Z and set the transfer method to, eg, max. The transfer method determines what exact value will be used for a pixel under an object and what will happen when the same pixel is under multiple different objects. You want to use the value of Z and you have only one object, so you can choose either of max / min / sum. You keep 'Clear image pixels' unchecked, because we don't want to erase any of the existing pixel values. You keep 'Resize image' unchecked as well, because we don't want to resize the image. If the area is partly outside of the image, we will let the join process clip it. We press Join Component and the pixels under the area get set to Z.

If you want to, say, *increase* pixels under an area by the value of its Z field, that's also doable albeit we'll need a small bit of SQL (really small, read on): invoke the Join dialog on an image, select the drawing as a source, then *add* a new channel for Z and set the transfer method to max. Press Join Component. This will join data into a new channel without altering the first channel that you see in the map. You can check what's in the second channel by switching display to it in the Style pane (this currently requires closing and reopening the window, we'll fix that). OK, so right now we have the original image in channel 0 and the values of the Z field from the drawing in channel 1. In that channel 1, pixels under the area are set to its Z and pixels outside of the area are set to 0. To add channel 1 to channel 0, open the Transform pane, make sure the field combo at the top is set to the tile field, switch to the Expression tab, then use this expression:

--SQL9

TileChannelCopy(

  [tile], 0,

  TileChannel([tile], 0) + TileChannel([tile], 1), 0

)

You should see the results in the preview even before you press Update Field.

OK, so, what happened in the expression. We take channel 0 and channel 1 from the original tile and sum it (TileChannel and +). Then we copy the result into channel 0 of the original tile (TileChannelCopy). If you are going to keep joining drawing data into channel 1, you don't have to be updating channel 0 manually using the Transform pane and the Expression tab - press Edit Query in the Transform pane to generate the update query, then invoke Edit - Save as Query to save this as a component, and just run that query after the join.

In the future we are going to restructure how we deal with images and then we will be able to add channels using a plain UI transform with no SQL coding required.

Hope this helps.

adamw


9,135 post(s)
#01-May-20 08:13

An additional thought: in the case of repeatedly adding data from a changing drawing to an image it would perhaps make sense to save the original image data into a separate channel, to avoid adding data for the same object multiple times. Saving the original image data into a separate channel could be done simply by joining an image to itself: add a new channel copied from an existing channel. There would thus be three channels: channel 0 = drawing + original image, channel 1 = drawing, channel 2 = original image. Joining data from a drawing would update channel 1. The addition would take channel 1 and channel 2 and update channel 0. The display would only use channel 0.

vincent

1,911 post(s)
#01-May-20 14:43

To do this, you invoke the Join dialog on an image

Ok ! I was in a map, and the drawing was the active component.

then set Channel 0 to take data from Z

It would be more intuitive to have some grayed text in the 2 cells right to Channel 0 in the Join dialog, to show that something is possible (#1 cell: aggregator; #2 cell : field). I really had hard time understanding that I can actually do something with Channel 0, instead of adding a new channel.

vincent

1,911 post(s)
#01-May-20 15:13

It is now possible with this Join to Image from Drawing, and with Fill missing, to modify DEM prior to run the Hydrological Tools (for culvert and bridges buffers, stored in a drawing). Great work !

adamw


9,135 post(s)
#01-May-20 16:28

We will think about adding grayed text, thanks. (We would need to do it without filling the entire grid with it, but this can be figured out.)

vincent

1,911 post(s)
#01-May-20 19:54

A title line over the grid would do. With : Source field ; Operator ; Target Field. (or the reverse if I'm mistaking)

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.