Subscribe to this thread
Home - General / All posts - WMS services, favourites adding \ to URL?
tonyw
504 post(s)
#11-Apr-19 23:33

Since about the last two releases of M9 I have been unable to get any images from WMS servers. When I check the URL in File > Create > New Favorite Data Source > Edit favorites I see a series of backslashes have been added to the URL. So http:// becomes http:\/\/ It looks like capital W but it's backslash forward slash together. The Project Pane populates with the names of the WMS layers from the WMS server but I M9 does not render any images if I double click on the layer names.

Here's WMS url if someone wishes to try, this layer is all the surveyed properties in British Columbia.

https://openmaps.gov.bc.ca/geo/pub/WHSE_CADASTRE.PMBC_PARCEL_FABRIC_POLY_SVW/ows?service=WMS&request=GetCapabilities

When I added the above URL to M9 (build 9.0.168.11) the URL now looks like this:

https:\/\/openmaps.gov.bc.ca\/geo\/pub\/WHSE_CADASTRE.PMBC_PARCEL_FABRIC_POLY_SVW\/ows?service=WMS&request=GetCapabilities", "SourceCache": true, "SourceCacheTemp": true, "SourceUuid": "e6e81c7c-7e47-4a23-8496-a5fdd2ba19a4" }

All the extra characters after GetCapabilities starting with , "SourceCache" are not present when I create the new favorite source and TEST he connection (connection successful).

Other observations: Open Street maps and Bing satellite images work. images render. However Google Satellite does not, no image renders.

Thoughts?

Attachments:
WMS servers1.JPG

Dimitri


5,359 post(s)
#12-Apr-19 05:40

When I added the above URL to M9 (build 9.0.168.11) the URL now looks like this:

When I copy and paste any strings into the Source box of a New Data Source dialog set to "wms" in the Type box, the dialog does not change the URL at all.

I think what is going on is that by your phrase above you don't mean that the New Data Source dialog makes any changes. It takes exactly what you enter as you enter it. Instead, I'm guessing that after you create the data source, if you then right-click onto that data source in the Project pane and look into the Properties, you'll see a JSON string that escapes special characters. That's correct, and exactly how it is supposed to work.

The JSON text string in the property has to work within other processes, such as SQL expressions, that use it. In such cases you have to escape special characters (a forward slash is a special character) with the backslash character. But that's a different situation than entering free text into a "Source" box for a URL, or entering such text into a browser URL box.

WMS works fine. If you'd like to test many different WMS services, visit the Product Downloads page for 9 and download the JOSM sources example map from the list of examples. There are many wms servers in there, for example, under Canada the Canvec server that provides an SCW-Toporama image.

As to why a particular server doesn't work for you, there is useful discussion in the Web Servers topic Notes section. There also have been threads in this forum on why, for example, Google might be blacklisting you (too much use, you're not using a free developer account, etc.). Classic problems with WMS is that WMS servers are routinely misconfigured, they aren't working (shut off for maintenance or just turned off), the URL changed and the webmaster didn't tell anybody but his friends, somebody in the bureaucracy decided the server was getting too much use so they now require access credentials, or they are just inconveniently configured. The WMS URL you provide is a classic example of the latter.

The particular WMS server URL you provided works fine, it's just very slow and it only provides layers when you zoom far in. Do this:

1. Make a data source for it.

2. Create a default Bing streets data source from the Favorites.

3. Create a new map and open it.

4. Drag and drop the Bing layer into the map.

5. Zoom into Vancouver, Canada.

6. Drag and drop the pub:WHSE_CADASTRE.PMBC_PARCEL_FABRIC_POLY_SVW Image layer from your data source into the map, as a layer above Bing.

7. Zoom far into the street grid of Vancouver, so you see approximately five city blocks wide by five high in the display.

8. The DataBC layer will fill in and appear. I'm using it from Europe so maybe that adds slowness.

Why it is that webmasters configure WMS services so they show nothing until you zoom far in? Good question for the webmaster publishing that server.

KlausDE

6,300 post(s)
#12-Apr-19 08:38

Just a short note, no time for testing:

I too found many(!) stored favorite wms datasources not working with version 10+ as they did before. To open the table triggers fetching tiles but to pan and resize a map does not necessarily - a cache thing??. Also transparency is always off while it seems it had been always on in previous builds. And with some favorite wms datasources I noticed the reversed color effect.

We should be able to control transparency and png/tiff/... format without detaching data from servers.

tonyw
504 post(s)
#12-Apr-19 15:25

...transparency is always off

Yes, I've noticed the WMS layers that I am able to receive are opaque (for instance the surveyed properties layer I mention above). If the WMS layer is the top layer then I cannot see any of the layers below. As the WMS server chooses the colour scheme over which we have no control, I was not sure if the opaqueness was an unfortunate choice of the server.

Dimitri


5,359 post(s)
#12-Apr-19 15:51

I was not sure if the opaqueness was an unfortunate choice of the server.

That's probably the case here. WMS layers are usually opaque when they serve tiles, using white color pixels, not transparent pixels. To avoid chasing red herrings that's something to find out for sure.

"transparency is always off" may have been an unfortunate choice of words. I've tried a bunch of WMS servers and they all seem to be working fine with transparency. You just have to look at each server in careful detail to be sure what is going on.

tonyw
504 post(s)
#14-Apr-19 00:46

I did some more testing of the provincial WMS server images. There is different behaviour between QGIS (3.2.0 Bonn) and M9 (9.0.168.11).

In QGIS, no matter which layer is on top, all the underlying layers are also visible. The map is of lakes and forest roads around Vancouver BC Canada.

1a. Roads WMS on top. Lakes WMS next. Coastline vector (shapefile) lowest layer). Details of all 3 layers are visible.

1b. Lakes WMS on top. Roads WMS next. Coastline vector lowest layer). Details of all 3 layers are visible.

2. M9. Coastline (shapefile) top layer. No interference with roads so roads WMS are visible. Note lowest layer lakes WMS are not visible.

3. M9. Coastline (vector) moved to lowest layer, not visible

4. M9. Coastline (vector) top layer, visible. Lakes WMS 2nd layer, visible. Roads WMS lowest layer, not visible

5. M9. Coastline (vector) top layer, visible. Lakes WMS 2nd layer, visible. Roads WMS lowest layer, not visible.

Source: BC Maps Library and WMS link https://catalogue.data.gov.bc.ca/dataset/bc-web-map-library

WMS: http://openmaps.gov.bc.ca/geo/ows?service=wms&version=1.3.0&request=GetCapabilities

Attachments:
1a. QGIS 3.2.0. Bonn transparent WMS layers, roads on top.JPG
1b. QGIS 3.2.0. Bonn transparent WMS layers, lakes on top.JPG
2. M9 Coastline top layer, visible.JPG
3. M9 Coastline not a top layer, not visible.JPG

tonyw
504 post(s)
#14-Apr-19 00:46

The rest of the screen shots

Attachments:
4. Lakes moved to top WMS layer, roads lower WMS layer not visible.JPG
5. Roads moved to top WMS layer visible, Lakes lower WMS layer not visible.JPG

tonyw
504 post(s)
#14-Apr-19 05:47

5. M9. Coastline (vector) top layer, visible. Lakes WMS 2nd layer, visible. Roads WMS lowest layer, not visible.

Copy and paste error, last description should read:

5. M9. Coastline (vector) top layer, visible. Roads WMS 2nd layer, visible. Lakes WMS lowest layer, not visible. Filename for 5th scenaria is correct.

adamw


8,447 post(s)
#19-Apr-19 14:24

We were able to reproduce the issue. This seems to be a bug on our side (transparency gets lost, the circumstances are being determined). We'll fix it.

tonyw
504 post(s)
#21-Apr-19 04:12

Thanks Adam.

Dimitri


5,359 post(s)
#12-Apr-19 15:47

I too found many(!) stored favorite wms datasources not working with version 10+ as they did before.

As always, it's a battle of details. You have to look at each server individually to see what is going on, and you have to carefully look at each and every detail of work flow.

Example: What are the Cache data and Save cached data between sessions settings for the data source?

Have you tried pre-10 on exactly the same server? Etc.

What you find with WMS is extremely wide variability in how servers are configured. That includes things like use of transparency vs opacity in what alpha is supposed to mean in whatever format they are using to send tiles. As noted in the TIFF topic discussion, people can and do misuse the tags associated with the fourth channel all the time.

It seems to be that the way such inconsistencies are handled in 10+ works better for most four channel TIFF images. But whenever you have an inconsistency, guessing one way or the other is going to cause some things to look OK and others to not look OK.

We should be able to control transparency and png/tiff/... format without detaching data from servers.

I agree it's great to have controls to support flexibility, although I'd hate to make the data source dialog overly complicated. It would help to have details on what's going on with those servers.

adamw


8,447 post(s)
#19-Apr-19 13:41

Regarding this:

I too found many(!) stored favorite wms datasources not working with version 10+ as they did before. To open the table triggers fetching tiles but to pan and resize a map does not necessarily - a cache thing??

If you come across such a server, drop us the URL and we'll take a look. I doubt this is related to cache (if you are adding a new data source, it cannot have had any cache stored in the target database, it's only if you are opening an existing data source which is also configured to use persistent cache - our default is not to do that and only cache data within a session - that the cache can even get involved - it can then get desynced, yes, especially if the server changes what it's doing).

Also transparency is always off while it seems it had been always on in previous builds. And with some favorite wms datasources I noticed the reversed color effect.

This sounds like issues with channel remapping, eg, TIFF. We really need a server URL to debug it. We'll do our own testing, too, but a specific server exhibiting the issue would help a lot.

We should be able to control transparency and png/tiff/... format without detaching data from servers.

Support for transparency depends on the output format. WMS servers advertise what output formats they support and we select the best one automatically. Our current preferences are roughly this: jpgpng -> png -> tiff -> (rare things like ppm or gif) -> jpg, specifically designed to prefer formats with transparency. We could allow specifying preferred format explicitly in the New Data Source dialog for when the server does a better job at the format that manages to lose to some other format.

KlausDE

6,300 post(s)
#21-Apr-19 10:22

Home again from bonnie Scotland I can start test wms servers.

I guess the "servers not fetching tiles in a map but starting to do so when opening the table" was the effect of an unsuitable scale of the map without desirable log. I create a wms datasource from URL https://www.wms.nrw.de/geobasis/wms_nw_alkis with my preferred coordinate system EPSG:25832, drop it into an empty map. Nothing happens and the system logs 0.000 sec to render. Change scale to 1:1,000 to meet the range of ScaleDenominator (0..5001 according to GetCapabilities) and the GetMap request gets tiles that are displayed in the map.

( BTW this is logged in the Log window only and not in the log file. That reports open/close project and import logs only. Would be nice to have it together with error logs in the file as well to check server sided failure in run time. )

The server offers a LegendURL and type. It would be nice to have access to this, a Format="image/png" in this case. No need to offer it in the GUI to the interactive user IMO but it would be nice to have it reported in mfd_meta besides Text and Description together with the LegendFormat to be able to address it in an IMS.

A "color reversed" background in an other WMS turned out to be just an another color for a background that should be transparent. This will probably be fixed with transparency. The object color was not reversed.

adamw


8,447 post(s)
#23-Apr-19 09:11

Not even trying to fetch tiles at some zooms is indeed quite likely related to the image not being available at those zooms, as reported by the server.

Regarding logging failed web requests into the file - this should probably be something you turn on temporarily for the duration of the current session. Because, in general, that's just too much logging that floods the log and makes it unusable. Plus sometimes you want to log successes, too. Or we could log summaries (first N successes and then every 1000th success, same for failures, something like this). We'll think about it.

We'll consider exposing legend images in WMS.

KlausDE

6,300 post(s)
#12-Apr-19 09:49

Why it is that webmasters configure WMS services so they show nothing until you zoom far in? Good question for the webmaster publishing that server.

Usually because you cannot make use of a lots of data that have to be processed by the server due to the unappropriate scale. That's a very good reason for a webmaster to reduce the load.

GetCapabilities usually return valid bounds and scales. So this could be handles by the client. I'd like to see a note in the log and perhaps we need some flag so that legend details could be replaced by a note for an ordinary user.

Dimitri


5,359 post(s)
#12-Apr-19 15:30

lots of data that have to be processed by the server due to the unappropriate scale.

Well, that's the result of using inefficient server software. A screen full of pixels is a screen full of pixels no matter what the scale, so you'd think the server side could handle the abstraction of showing simplified points, even for a big data set.

But if that's a hassle, they should put a higher layer showing "no data at this zoom" or whatever.

Or, if you zoom out farther than the operator wants to show data, switch to showing OpenStreetMap with red boundaries marking where data is available once you zoom in.

GetCapabilities usually return valid bounds and scales. So this could be handles by the client.

Could be, like showing a Message notification.

The problem when you start all that is the client starts designing the web site instead of the web master who runs the web site. If somebody wants to run an WMS server, they should handle the design of their site, not the client.

KlausDE

6,300 post(s)
#13-Apr-19 16:01

All good points and manifold 9 redraw logic is perfect.

Reading the GetCapability attribute range of scale and offering the result to the administrator of the server is important for the future manifold 9 server, not the client. I got that wrong.

I currently know a number of datasources that offer data at reduced range of scale for the time of a rework of the data. The adminstrator of the ims server often is not the administrator of the wms datasource. If manifold 9 working as IMS server could offer the attribute from the GetCapabilitiers document server message could automatically reflect the state of the remote wms data.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.