Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.167.8
adamw


8,242 post(s)
#30-Jul-18 17:09

9.0.167.8

This build is the last feature build in the current cutting edge sequence. We will issue either one or two more quick builds with fixes and cleanups and then we will issue the public build and move on to the next sequence.

manifold-9.0.167.8-x64.zip

SHA256: 492f2d4695a25e6ad8cdc1c38e1abbf6b37b41b0fbe1b3ff51965301adcdc29a

manifold-viewer-9.0.167.8-x64.zip

SHA256: 728daa14f811afc7911d63a3fedf1a16e273cc694941123c7277bc2f8435cbac

adamw


8,242 post(s)
#30-Jul-18 17:10

Changes

New query function: TileInterpolateTriangulationSeg - takes a drawing with geoms with Z values and interpolates them using constrained triangulation keeping segments from lines and areas. There is a boolean parameter that controls whether to improve triangulation of contours using DEST (true = use DEST).

New query function: TileInterpolateTriangulationSegPar - a parallel variant of TileInterpolateTriangulationSeg.

New query function: TileInterpolateGravity - takes the same parameters as kriging sans the model and interpolates data using gravity formulae, similarly to 8. (There is no parallel variant, because preparing interpolation data for gravity does not benefit from using multiple threads, although producing tiles does.)

The Help menu includes the new Context Help command bound to F1, which shows quick reference guide for the current component window. If there are no opened components, the command shows general keyboard and mouse reference.

Migrating views from MAP files created by 8 better scales views for lat/lon maps.

The context menus used by parameter pickers in the Select and Transform panes show icons for Value and Expression commands and show current choice. The icon for Value depends on the parameter type. The icon for Expression is changed from 'operator' to 'function'.

Parameter pickers in the Select and Transform panes clean xN values and produce VectorMakeXn(...) calls for the query.

The Vector Value transform is split into Vector x2 Value, Vector x3 Value, Vector x4 Value. This disambiguates the type of parameter value and allows parameter pickers to clean it (previously, entered text was going into the query verbatim).

(Fix) Previewing the result of a transform in a table window converts the result of the transform to the target field type. (Previously, the values were only converted when the transform was being run, and the preview could have been misleading.)

The Select and Transform panes specify component parameters using parameter pickers.

Transforms that use collate parameters embed the collate value as a plain numeric code instead of a fairly involved call to Collate(...).

Transforms that use regular expressions embed the case switch as a plain string instead of a CASE statement.

Many (the majority of) transforms specify the 'main' transform parameter which gets auto-set to the target field. (Example: Upper Case will set its only parameter to the target field, because that's the most common use. Previously, selecting Upper Case would set the parameter to an arbitrary field. Further, after the user would select the correct field, run the transform, then switch to another field and try to do Upper Case on the new field, the system would rather unhelpfully set the parameter to the previous already-uppercased field.)

The Select and Transform panes use parameter pickers for numeric and text parameters that have previously been specified using a textbox but can be set to be taken from a field. (This includes parameters like buffer size or tolerance. By default, the parameters are still set to use fixed values like 1 or 0, but the user can quickly switch to use a field. Quoting entered strings is no longer necessary.)

Renamed transforms: Enclosing Rect, Rotation Allowed -> Enclosing Rect with Rotation, Is NULL -> Null Values, Is not NULL -> Non-null Values. Renamed transform parameters: DX -> Delta X, DY -> Delta Y, Pattern (when used by Like) -> Pattern (%_).

Parameter pickers sort fields and components alphabetically. The Transform pane sorts target fields alphabetically.

End of list.

tjhb

8,399 post(s)
online
#30-Jul-18 23:01

So happy about this. Might be a bit quiet (busy) for a while.

adamw


8,242 post(s)
#31-Jul-18 08:49

A couple of notes regarding interpolation functions:

If the segments passed to constrained triangulation intersect, creating a potential conflict regarding the height at the intersection location, we don't generate an error and instead ignore one of the segments. This is intentional and we think this is the best course of action, other variants that we tried felt worse. (Generating an error is bad because it is hard to fix the source of the error without guidance as to where the intersection point is; this guidance is hard to provide because there might be millions of intersection points. Splitting segments at the intersection point needs to assign some height to that point; ignoring one of the segments has the exact same effect with the benefit of being easier to explain and a clear signal that yes, this is lossy, please resolve intersections before doing interpolation and put explicit heights onto intersection points using whatever rule is appropriate if you want to use all segments.)

We have been reviewing the implementation of DEST in 8 and have been making adjustments for various corner cases aiming to improve quality. We are mostly finished, there are just a couple of last adjustments pending for 167.9.

tjhb

8,399 post(s)
online
#31-Jul-18 09:45

That sounds great. In 8 I am used to detecting and correcting contour collisions before running DEST, and also ensuring line continuity (especially at turning points) so that DEST can be applied.

I can only think of one type of possible case where the algorithm in 8 may not give a perfectly smooth result--I have never been quite sure, but I know where to look for a test.

tjhb

8,399 post(s)
online
#01-Aug-18 05:40

Still testing... and having a terrific time. The new triangulation code is astonishly fast.

Once again Manifold has made new workflows and experimentation possible (and also, enjoyable). Many thanks for the hard work!

geozap101 post(s)
#01-Aug-18 16:05

I am trying 9.0.167.8 Viewer version on windows 7 64 bit.

I noticed that connecting to http://api.openstreetmap.org/api/0.6 as a Web server:osm data source shows nothing on screen even thought connection is estabilished. Also connecting to this wms server shows nothing on screen: http://gis.ktimanet.gr/wms/wmsopen/wmsserver.aspx This wms server works fine on QGIS and used to work fine at least until 9.0.167.5

I haven't used other versions of viewer since 9.0.167.5 and I don't know when those issues first apeared.

hugh
159 post(s)
#01-Aug-18 16:11

in viewer .7 and .8 got this message

2018-08-01 09:09:05 Render: [Data Source]::[Osm Drawing] (0.047 sec)

2018-08-01 09:09:15 *** (root)::[Data Source] Queried area is more than the maximum area that can be queried (4.000)

Dimitri

5,106 post(s)
#01-Aug-18 17:34

Queried area is more than the maximum area that can be queried (4.000)

Could it be that the OSM server is complaining you are trying to fetch too much? Try using a map that has a different image server, like Bing, to use to zero in closer to the area of interest, and only then turn on OSM. What happens then?

geozap101 post(s)
#01-Aug-18 19:19

Your suggestion works for the OSM server, but not for the wms server in my post.

Dimitri

5,106 post(s)
#02-Aug-18 06:17

not for the wms server in my post.

Sounds like the wms server in your post is doing something weird. Try other WMS servers to prove to yourself that Viewer works fine for WMS. It is always possible there is an error in Viewer, of course, but by odds of about 99 to 1 it is more likely something in the WMS server.

Read the documentation on servers, especially the part about strange limitations some server operators install. Note the following: "Finding out why a web server refuses service can involve some detective work."

There was a case a while back, for example, of a guy - either in France or in Greece, I don't recall - who hated ESRI so much that he configured his servers so they only worked with Q. But a side effect of that was they didn't work with much other software, not even with free software like Viewer, either. Sometimes you have to dig deep to find out what a specific server is doing when it fails to support the standard it claims to use.

Contact the web master for that server. Good luck with that, since, unfortunately, it is often exactly those web servers that are misconfigured which often have poor response to user inquiries.

geozap101 post(s)
#02-Aug-18 07:53

I tried with some other wms servers as you suggested and they work.

But that server not only works fine with QGIS and Google earth, but with 8.0.30 too. So even if they changed and maybe misconfigured something, manifold 8 (and other software) still manages to handle that change, while viewer doesn't. Later I will try it with an older version on viewer I don't have access to right now to check if it still works on them (it used to work at 9.0.167.5 for sure)

adamw


8,242 post(s)
#02-Aug-18 09:29

The log message with the OSM is the server telling us that the request area is too big.

(In this particular case the server does not even wait for the request and tells us upfront that 2x2 degrees is the absolute upper limit it will even consider. This happens because the data is vector; unlike with rasters, you get way more bytes to transport as you zoom out, because there is no universal information reduction mechanism like the pyramids. The server can refuse requests smaller than 2x2 degrees as well if they happen to be in densely populated places, but we deal with those refusals automatically by splitting requests more. We don't want to be automatically splitting requests for big areas because all this will do is generate gigabytes of traffic with the window sitting for hours painting a couple of pixels in the corner.)

So, for the OSM, just zoom in.

We are looking into the WMS.

adamw


8,242 post(s)
#02-Aug-18 10:07

OK, here's what happens with the WMS.

The server says that the image is in EPSG:2100 (a coordinate system specific to Greece) and that it can also provide data for the same image in EPSG:4326 (lat/lon, WGS84). We default to EPSG:2100, because that's the native coordinate system. QGIS defaults to EPSG:4326. The server returns valid but empty tiles for EPSG:2100 and valid and not empty tiles for EPSG:4326. Hence QGIS shows something by default and 9 shows nothing by default: the server does not raise any errors, it just says "here are your tiles" and returns tiles full of invisible pixels.

You can get the picture in 9 if, when you create the dataport, you set the preferred coordinate system to '4326' (there's a text box for this). This will override the default and select EPSG:4326 and since the server returns data for that system, you will see the data.

We cannot fix it on our side, because, unfortunately, it is not our issue, it is the issue with the server. If we stop trusting the server to return data in the native coordinate system it reports and will force it to EPSG:4326 whenever that is available, this will just make the results worse for servers that behave well and produce symmetric issues with servers that work well with their native coordinate system and do not work well with other coordinate systems they report as also supported. QGIS and 8 work with this particular server because once they see EPSG:4326 they ignore everything else, but this exact same logic is going to fail elsewhere.

What likely happened between 9.0.167.5 (where the server worked) and 9.0.167.8 (where the server returns an empty screen) is that the server was upgraded, and whoever was checking that the upgrade worked did not check all of the coordinate systems that the server reports as available.

geozap101 post(s)
#02-Aug-18 10:40

I understand. I tried with 9.0.167.5 and it has the same problem with 9.0.167.8, so they must have changed their configuration. Thanks for the help. Good to know there is a workaround to use with 9 the wms server I heavily depend on.

Dimitri

5,106 post(s)
#02-Aug-18 12:10

Make sure to write to the web master to let him know his server is misconfigured. I know that worldly people think that is a waste of time, that there is no chance anybody running such a server will respond to bug reports, but hey, it sometimes works.

I just ran into a misconfigured WMS server today that is operated by the US Forestry Service, at https://fsapps.nwcg.gov/afm/wms.php

It reports satellite sensor data for wildfires, important data these days given the loss of property and life going on right now in the US from wildfires. This particular server issues tiles as jpg, but it sends the data saying it is 'text/xml' as if it were text comments.

That's a pretty gross error. Saying the data is text, like metadata or comments, and then sending jpg will prevent the data from being displayed by any client that complies with WMS. If the client ignores such WMS directives it cannot handle them correctly when they matter. I don't know if they've ever tested their server or what they used to test it, but whatever it was, it wasn't a WMS-compliant client.

Anyway, I've sent them a bug report. Will the server get fixed? Maybe. Of the last ten such bug reports I've sent to government web server operators I've gotten only a single reply, but that was very thoughtful and appreciative, and got the error fixed right away. :-)

Dimitri

5,106 post(s)
#03-Aug-18 06:28

Works fine now in 9.0.167.9 No reply from the Forest Service about the bug in their server.

As unpleasant as it is to have to add a capability that assumes a "WMS" server is doing WMS wrongly, well, .9's ability to work around such bugs is one way to consume data from misconfigured servers. Glad this got done quickly as the data on that server is very useful for emergency work.

In particular, unlike the USGS servers it does not stop rendering as you zoom further in, and it provides better layers for VIIRS I band fire detection. That is a very big help to people trying to escape, or to stop, fast-moving fires. The image below with VIIRS I band datashows how the fast moving fire to the North of Clear Lake has engulfed Bartlett Springs, which it apparently overran in less than an hour. (Click on the image for a bigger image.)

Attachments:
clear_lake_fires.jpg

adamw


8,242 post(s)
#02-Aug-18 12:42

A correction to what I wrote above, you have to set the preferred coordinate system to 'EPSG:4326', not to '4326', to match how the server describes it.

Like this:

Hope this helps.

Attachments:
wms-data-source.png

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.