Subscribe to this thread
Home - General / All posts - Examples of these new transforms
antoniocarlos

609 post(s)
#04-Jan-20 13:44

Hi are there any examples for these transforms. I cant find them and cant figure them out. Thanks.

  • Closest Euclidean, Direction - produces a raster with Euclidean directions, in degrees.
  • Closest Euclidean, Distance - produces a raster with Euclidean distances.
  • Closest Euclidean, Source - produces a raster with Z values of closest points using Euclidean distances.
  • Closest Weighted, Direction - produces a raster with path directions, in degrees. The 'Use equal weights' option switches between just walking around missing pixels and also using pixel values as walking costs.
  • Closest Weighted, Length - produces a raster with path distances. The 'Use equal weights' option works as above.
  • Closest Weighted, Source - produces a raster with Z values of closest points using path distances. The 'Use equal weights' option works as above.

How soon?

tjhb
10,094 post(s)
#04-Jan-20 19:40

I think you're right.

Although the underlying SQL functions are present in 9.0.170.2, the corresponding new transforms are not listed in the Transform panel.

This is the case whether we address an image or its table (in the second case, manually selecting Tile as the source field).


An apparently unrelated UI bug (noticed when looking at the above):

In any of 9.0.169.0, 9.0.170.0, 170.1 or 170.2, open an image table, go to Contents > Transform, and select Tile as the source field. The Limit, Limit High and Limit Low transforms are all listed twice. This doesn't arise if we address the image instead, or if the source field is something other than a tile.

danb

2,064 post(s)
#04-Jan-20 20:07

I haven't had a proper play yet, but to see the transforms, you need a map containing a drawing (I have only tried points so far) and a surface. The surface should have focus.

Attachments:
Transforms.png


Landsystems Ltd ... Know your land | www.landsystems.co.nz

tjhb
10,094 post(s)
#04-Jan-20 20:12

Oh thanks, that makes sense Dan.

Now I can press Edit Query to see the correct SQL.

It looks as if I guessed the syntax roughly right, so I will try with better test data.

[Yes it was my data that was at fault.]

antoniocarlos

609 post(s)
#04-Jan-20 23:20

Got it to work thanks.


How soon?

antoniocarlos

609 post(s)
#05-Jan-20 00:16

So I got it to work.

I would like to make a recommendation that the default display of the distance surface be smarter. It seems to me that is the image only has one band (I think it does) then it could automatically choose (Channel 0) and display the image (I still think of these as surfaces) using one of the color palettes available. The initial display is (not pretty) and in the absence of a legend it is not easy to interpret visually if the result is correct.

Cheers

Attachments:
Capture.JPG


How soon?

tjhb
10,094 post(s)
#05-Jan-20 02:12

Do you mean that a single-channel transform should operate by default on the first channel of a multi-channel image, and always produce a single-channel image as its output?

Or is this just about the default style for a single-channel image?

antoniocarlos

609 post(s)
#05-Jan-20 13:33

I was thinking about the default style of a single channel image (a distance surface in this case). I dont know if/when a single channel transform should operate on only one channel of a multi channel image.


How soon?

adamw


10,447 post(s)
#07-Jan-20 15:17

A single-channel transform should not operate on a channel of a multi-channel image. For now...

adamw


10,447 post(s)
#07-Jan-20 15:11

Agree. We are planning to apply some sensible style to produced data - as long as it goes into a new component - not just for this family of transforms, but for some others as well. Eg, when we do Slope, we know that the output is an angle and can style accordingly.

adamw


10,447 post(s)
#07-Jan-20 15:08

In any of 9.0.169.0, 9.0.170.0, 170.1 or 170.2, open an image table, go to Contents > Transform, and select Tile as the source field. The Limit, Limit High and Limit Low transforms are all listed twice. This doesn't arise if we address the image instead, or if the source field is something other than a tile.

This is an interesting effect, indeed. We'll fix it.

Thanks for the note!

tjhb
10,094 post(s)
#04-Jan-20 20:08

As far as the SQL functions are concerned, I can't work out the correct syntax either.

I imagine it should be something like this.

--SQL9

FUNCTION distances() TABLE AS

  (

  TABLE CALL TilesDistanceMake([Image], 0)

  )

  END;

TABLE CALL TileDistanceTilesClosest(

  CALL distances(),

  [Drawing]

  );

While this seems to parse (gives no error), I cannot get it to give a result (not even an empty table).

The distances() function could be inlined, enclosed in brackets:

--SQL9

TABLE CALL TileDistanceTilesClosest(

  (TABLE CALL TilesDistanceMake([Image], 0)),

  [Drawing]

  );

Again no error, no result.

The same occurs using TilesDistanceTilesLength().

On the other hand, substituting TileDistanceTilesDirection() gives error "Cannot parse query."

tjhb
10,094 post(s)
#04-Jan-20 20:48

I did the dumbest thing. I forgot to press "Save Changes" after adding points to my (empty) test drawing. Thus no result. Sheesh.

And my error with TileDistanceTilesDirection was in leaving off the <type> parameter. Duh.

So, syntax all correct.

The automatically-generated SQL uses a VALUE to store the result of TilesDistanceMake/Par, but we can equally use a function or an inline the CALL as above. The difference is that a VALUE is likely to be more efficient if we want to perform multiple analyses (that is, two or more of TileDistanceTilesClosest, TileDistanceTilesDirection, TileDistanceTilesLength, using the same source image).


I still don't understand the distance type options to TileDistanceMake/Par, or the corresponding options to TileDistanceTilesDirection. This is a fundamental thing to grasp, and I don't.

--Ah. The penny has mainly dropped.

I still don't confidently understand the purpose of TileDistanceMake/Par, or what the distance type options do there. Let me try (next post)...

tjhb
10,094 post(s)
#04-Jan-20 21:13

If the image has no null pixels, then options 0 and 1 do the same thing. That is, they produce a quasi-table of radial distances around every pixel.

If there are some null pixels, then if option 0 is used, the null pixels are simply skipped, with distances still measured radially; while for option 1, the paths away from each pixel bend to avoid null pixels.

If option 2 is used, raw distances are multiplied by pixel values, giving a cost surface. Again, if there are null pixels, then the paths away from each pixel may not be radial, but will bend to avoid null pixels.

While the result of TileDistanceMake/Par is nominally and syntactically a table, it is actually a special data structure adapted to store distance data for every pixel. It must be extremely clever to do this efficiently (and fast). We can't read the result: to the user it is a black box. In this way (only), it is a bit like the result of CoordConverterMake.

The resulting distance structure can be used by one or more of TileDistanceTilesClosest, TileDistanceTilesDirection, TileDistanceTilesLength and their parallel variants, to trace from each pixel to the nearest point in their specified drawing.

Hopefully that's it.


I'm not sure about the function names TileDistanceTilesClosest, TileDistanceTilesDirection, TileDistanceTilesLength. There must be a good reason why the second Tiles keyword is inserted in each case, but it seems unwieldy in use. TileDistanceClosest, TileDistanceDirection, TileDistanceLength could be friendlier. (Besides that, if length were not an issue, how about TileDistanceGeomClosest, TileDistanceGeomDirection, TileDistanceGeomLength?)

tjhb
10,094 post(s)
#04-Jan-20 22:33

I and no doubt others would be extremely interested in knowing more about what the result of TileDistanceMake/Par actually consists of. [I apologize for sometimes writing Tiles... rather than Tile... above for these functions.]

I suspect the maths involved would be the kind of thing it takes decades to learn.

Obviously it can't literally be a table of buffers for every pixel. That would be as large as (N x M) x (N x M) [corrected]! idiotically large and slow.

Perhaps it is a table of function coefficients, or of functions. That also seems too simple.

Likely to remain secret. Good.

adamw


10,447 post(s)
#07-Jan-20 15:36

TileDistanceMake/Par return a special table which captures information about the raster you want to produce (coordinate system / dimensions) as well as the type of distances you want to use when determining which object (called Source in functions) is closest to each pixel. This special table contains no records and passes its information via other means which work only in the context of a running query.

This is not the first time we use such a design. While it has its drawbacks in that some tables become 'magic' and have no records but have other data that you cannot see directly, it allows us to pass / cache arbitrary data between running query nodes without exploding query syntax with tons of artificial types which are subject to frequent change (we could have used a new type named TABLE_WITH_DISTANCE_DATA, but we'd need tens of such types and they'd change a lot).

On to function names:

The first Tile in TileDistanceTilesDirection is there because a function operates on tiles (takes a table with tiles as an input). The second Tile (well, Tiles) is there because a function produces tiles as well. Some functions that we have take tiles but produce geoms. We do not follow this notion of putting both input and output types into function names in all cases, though - eg, interpolation functions are TileInterpolate... even though they take geoms and produce tiles. But we do denote output types whenever they might differ between functions (TileViewshedAreaXxx vs TileViewshedTilesXxx).

Dimitri


7,413 post(s)
#06-Jan-20 18:34

There's a new topic in Help that has some useful info: Transform: Closest Rasters No doubt there will be more info on these, but that topic's a start.

tjhb
10,094 post(s)
#06-Jan-20 19:09

Wonderful topic! Well worth a few reads.

I see that I was wrong above in at least one important respect:

If the image has no null pixels, then options 0 and 1 do the same thing. That is, they produce a quasi-table of radial distances around every pixel.

That is false. Option 1 always produces "jagged" paths through pixel centres (except in the cardinal directions, where paths are straight), even if they don't need to avoid null pixels.

tjhb
10,094 post(s)
#06-Jan-20 19:36

One caveat/criticism of the topic (assuming I have understood correctly). In the section on lowest cost paths, slope is used as the cost of travelling through each pixel. That is mathematically valid, but seems unrealistic, in that it treats downhill and uphill slopes as having the same cost. Since our eventual routing is directional, this is important, depending to some degree on mode of transport.

I think this analogy breaks down for a slope surface, for a similar reason:

But that is exactly what water does when it flows downhill from higher terrain to lower terrain: the water always flows next to whatever is a lower spot.

If downhill travel is easier than uphill travel (or than travel over flat terrain), then the lowest cost path will maintain ("flow to") the highest downhill slope it can find locally, not to the next lowest slope regardless of uphill/downhill orientation.

I may be wrong.

tjhb
10,094 post(s)
#06-Jan-20 20:31

I think I have misunderstood the second point--the quote and my paragraph starting "If...".

Given a particular cost surface, the lowest cost path will indeed always "flow downhill" in the sense that is meant. I.e. it will always proceed from pixels with higher total cost towards pixels where total cost is lower.

I should have included the text preceding and following the part I quoted.

The result is what looks like a hill shaded terrain, but what we see is not a terrain elevation surface but instead an abstract cost surface: what seem to be higher spots are regions where each pixel has a high cost path to the closest green point. What seem to be low spots are regions where each pixel has a low cost path to the closest green point. If we start at any given pixel and then we always go next to whatever adjacent pixel has the lowest path cost, we will always stay on the lowest cost path to the closest green point.

But that is exactly what water does when it flows downhill from higher terrain to lower terrain: the water always flows next to whatever is a lower spot. In an abstract terrain where apparent "heights" are total costs of shortest paths, if we want to find the lowest cost path from any spot to the closest green point, we simply do what water does and "flow downhill" to the least cost path adjacent pixel. We can use Manifold's Watershed solvers to do that for us automatically.

That is right--and its purpose is to introduce the (great) next section discussing the use of watershed tools to delineate cost-based Voronoi areas around each destination point.

(I was muddling the slope of the source surface with the slope of the resulting cost surface.)

On the other hand, the caveats above and below about using slope as cost, regardless of uphill or downhill direction, are still well made I think. Making a cost surface from undirected slope (or cost "watersheds" from that result) is only appropriate in some circumstances.

...Which begs the question, which I have avoided: if we want to treat uphill slope as a positive cost, flat slope as neutral, and downhill slope as either neutral or a negative cost (gain), how would we do that?

How would we process our source surface then? Not ordinary slope but what?

adamw


10,447 post(s)
#07-Jan-20 15:48

There should never be a negative cost. You always spend some time getting through a pixel so the cost might be small, but positive. Same for zero cost - it just isn't realistic. The simplest way to transform a raster with both negative and positive values into a raster suitable to use as costs is perhaps just to subtract the minimum value plus some gap (so, if the source raster has values between -100 and 100, subtract -101 or something like that).

Mike Pelletier

2,122 post(s)
#07-Jan-20 16:13

Is there a way to take a slope raster and for each pixel change the cost depending on if the path is going up or down? So if the path is through a 10 degree slope going uphill the cost is 10 but when the path is downhill the cost is say 2, using some formula. When the path hits extremes like a cliff up or down the cost may need to be set very high. Possibly turn it into a null value depending on the purposes.

Dimitri


7,413 post(s)
#07-Jan-20 17:33

Is there a way to take a slope raster and for each pixel change the cost depending on if the path is going up or down?

No, not in these tools. Situations like that, with directional asymmetry, are not what tools like these (or the Distance tools in Spatial Analyst) are for.

Mike Pelletier

2,122 post(s)
#07-Jan-20 18:09

Thanks and I see your further explanation below.

On a separate note, one application that I'd like to be able to do (directional asymmetry, I believe) is to take streams and create a cost "floodplain" such that a user puts in a cost value and a raster is created showing the area flooded by the cost value along the entire stretch of the stream (potential riparian area). This is then compared with classified riparian vegetation from an aerial image along the stream. The purpose is to show areas where the stream could benefit from bank restoration, small rock dam structures, etc. to increase the potential for water to spread to dry areas during flooding and create more riparian area.

tjhb
10,094 post(s)
#06-Jan-20 20:02

To put this another way:

What the [Closest Path Length by Slope] result in the topic shows is the sum of (distance x slope) passing from each pixel to its nearest green point. So if two paths had equal length,* then any steep path would be more expensive than a flat slope, and a path travelling uphill at 45° would have the same cost as a path travelling downhill at 45°.

For some purposes or data, that would make sense. A good example might be routing electricity lines.

For other purposes, not.

Dimitri


7,413 post(s)
#07-Jan-20 05:22

For other purposes, not.

You've hit the nail on the head there, as to why tools like these are good for some purposes but not for others. If a task has directional asymmetry (where there are anisotropic effects based on directionality) these tools are probably not the best choice and a vector approach would probably be better.

I think raster tools, like GRASS, were more popular in earlier years because raster data was much easier to acquire for regions of interest. One pass with a radar scanner on a satellite and you get a whole region full of terrain elevation data - much easier than going out into the field and measuring a million locations point-by-point in the days before GPS, or before computed extraction of terrain heights from multi-angle aerial photography. Vector GIS tools were also very feeble, with very poor DBMS capability to handle attributes in the modern way.

But raster oriented tools, like ESRI's Distance tools and Manifold's Closest tools, are great for applications where data is raster in nature, and mostly continuous and isotropic. Take slopes as an example. Some applications using slopes don't really care about directionality, while others do. Your example routing electricity lines is a don't care example. Other examples are routing oil pipelines, where it's much more costly to run oil pipelines through terrain with lots of up and down slopes than it is through flat terrain.

Migration paths of people and animals are another. The up and down slopes in rough terrain, like the Appalachian chain of ridges (see this image) , average out in terms of "easy going down" / "hard going up" but the result is a higher "cost" going across the ridges than following the valleys. If you want to see how animals migrate through such terrain, or the paths along which earlier civilizations were linked, doing Path analyses using slope will provide insight.

It's true raster tools won't help much doing optimized routing in cases where vector tools are a better choice. A good example might be walking routes for tourism in our beloved Hong Kong, the topography of which features a high mountain ridge overlooking the densely populated strip of neighborhoods below. Many of the interesting neighborhoods are on steep slopes. To facilitate uphill travel, Hong Kong has a superb series of outdoor escalators to take you uphill without effort.

The key to least cost walking tourism is to take the Central / Mid Levels escalators up and to then take serpentine walking routes down through interesting neighborhoods. You take the effortless ride up and then it's an easy walk down.

But planning your tourist route using directionally symmetric raster paths isn't going to work. You need a vector map with one-way routes that have different costs depending on direction, so you can solve the graph problem through a directionally asymmetric network.

Conversely, trying that approach to understanding migration paths through terrain like this isn't realistic, not least because building a vector model over all possible routes is not so easy, and then doing geometrically more complex calculations for routing is also not so easy. A path solution for the entire terrain at once is a better approach, and that explains why people pay thousands of dollars per year to get such tools with ESRI. The ESRI literature I think would be worth reading for more examples of uses cases.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.