Subscribe to this thread
Home - General / All posts - Many types of KML file that M9 cant read
antoniocarlos

527 post(s)
#15-Jan-20 23:42

Hi

Normally I produce KML/KMZ files for clients and friends and have had no problems with this using M8. Now with the current earthquakes here in Puerto Rico I am a consumer of these files. The USGS produces this (enclosed) KML file that M9 or M8 cant read (but Global Mapper 18 can!).

This particular file only contains a link to the data resting on a USGS server that refreshes itself with some frequency so it does not contain any GIS data itself. It would be nice if M9 could link to it. Anyway I thought I would start a thread with this topic a I am sure there are many flavors of KML file that M9 wont read or reads partially.

Cheers

Attachments:
USGS Magnitude 2.5+ Earthquakes, Past Month.kml


How soon?

Dimitri


5,753 post(s)
#16-Jan-20 09:02

Your subject line for the thread is misleading, since you only cite one type of KML file 9 won't read. You don't cite "many types." If you file this as a suggestion, I'd recommend adjusting that to "one type of KML file...". See the tips in the Suggestions page on that.

Onward...

As they say, "anything is possible," but there are two questions to ask....

1. What is your priority for "read all possible forms of KML files"? As with many of the "markup language" formats, like GML, the formal description of KML allows some niche contructions. To put this another way... if Manifold can read 99% of the KML files typically used by GIS people, is spending effort on reading the last 1% or 1/10th of 1% of rarely-met KML forms more important, than, say, a handy tool for georeferencing rasters?

2. Do you really want a dataport that, while appearing to open a local file, actually opens a live connection to any URL at all on Internet (yikes !) and automatically pulls data from that every given number of seconds or minutes?

The nice thing about various web server standards like WMS, etc., is that it is clear you're connecting through the web to something, and you can see the URL that will be used. Those servers are also typically arranged so the system only pulls that data necessary, not downloading everything. They also don't download everything every given number of seconds or minutes.

Take a look at the text inside that KML and it's telling the system to pull data (could be millions of records) every... what? 60 minutes? 60 seconds? Suppose the <refreshInterval> tag value was set to 1? Suppose you switched to another application on your Windows desktop and forgot you left Manifold running with this thing downloading the whole data set every few seconds.... could be a lot of downloading if that went on overnight / days / weeks.

Automatic connections to URLs to do something when you click a file always sound convenient... at first. But one reason the Internet has become such a sewer of phishing crime is that the convenience of just clicking a file and having it do things for you automatically was prioritized ahead of the common sense understanding that some people will use that "just click it" convenience to do things in their interest and against yours. So that's always something to think about if you want to package any URL launching within a file.

So far, when Manifold imports or links to a file format you don't have to worry about Manifold launching a connection or a script from within, say, what looks like a JPG or a shapefile. There's something to be said for that. That something is possible doesn't mean that it is wise or without risk to do it.

Dimitri


5,753 post(s)
#16-Jan-20 09:21

You could read that file as follows:

1. copy the URL from the kml file and paste into your browser bar. The URL is:

https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_month_depth.kml

2. Press enter to download whenever you want. Save it as a favorite if you like on your browser's favorite bar.

3. Launch Manifold and create a read-only data source to the 2.5_month_depth.kml file in your downloads folder. Manifold reads that KML just fine, creating drawings. Drag and drop as desired into your maps.

4. Whenever you want to refresh, press the favorite for that on the browser bar to download the KML file again. Refresh in the map.

Personally, I prefer having control over something downloaded from the web. The above does that.

antoniocarlos

527 post(s)
#16-Jan-20 12:45

Thanks for the reply.

Yep I could have given this thread a better title. Sorry about that. I would change it if I could.

I was hoping to see if others in the forum wanted to submit other flavors of KML file that Manifold reads but not optimally for their purposes. I'll see if I can provide more of these files myself.

For importing, this particular file provides a link to the internet that could be harmful as you say. But would it not be possible/better if Manifold asked if the user wishes to download the linked data accepting the risk? My workaround was to open it in google earth and save the kml file to the desktop.

For linking the file I completely see your point. Didn't think of it. I don't know how to address it technically within Manifold if the philosophy is to avoid automatic connections.

Cheers


How soon?

Dimitri


5,753 post(s)
#16-Jan-20 13:15

It's easy to find KML examples that Manifold "doesn't read" if the criterion is that actions possible within KML file are not supported within Manifold.

KML can be almost anything you want it to be, including embedding full HTML, CSS, javascript, browser stuff. Here's an interesting page, for example. As it evolved at Google, it was developed to support a very web-centric business model (Google's), where they want people welded to using the web.

That's very useful for many purposes, just like browsers are very useful, but it is getting far afield from the idea of interchanging GIS data using file formats.

It's also extremely dangerous for users if you really want to enable all that stuff. For example, it's perfectly possible from a technical perspective to embed a ransomware launcher inside a KML - the entire code of the thing, not just a link to a malware site - and have that fire unknown to the user when they pop open the KML to look at some spatial data.

if Manifold asked if the user wishes to download the linked data accepting the risk?

Sure, of course that would be possible. And it would be possible to sandbox the other dangerous capabilities of fully supporting all flavors of KML. Anything is possible. To support everything in all KML flavors, you end up writing a browser, so expensive even Microsoft is giving up on Edge and turning to using Chrome as a base.

The question is whether that is a higher priority than spending the same effort creating features and capabilities more useful in GIS work. Maybe some very simple things, like references to other .kml files, might be worth doing while other things, like allowing arbitrary HTML and scripting, not. But in all cases it's that priority that should be stated.

I don't have anything against pulling data from the web. I think that's a great idea and it's good Manifold does that with so many web-based data sources (all those many web server standards) both with web servers and also with things like CSV files pulled over the web.

But I think the way to do that with web based KML is to do what's done with web based CSV: allow entering a URL to the file in the Source box in the New Data Source dialog. As with CSV, when somebody puts a URL into that box they know they're connecting to an external file.

Do that and then they could refresh when desired. I'd also consider adding a refresh option, to refresh every given interval, only when the project is opened, etc.

But if the objective is spatial data, I don't think it's good to automatically execute scripts and so on inside of KML files.

adamw


8,882 post(s)
#17-Jan-20 09:08

Dimitri is right in the above post is that KML is way too liberal with indirection levels and references, and is quite hard to support coherently because of that (for anyone but Google who extend KML in ways that suit their own code whenever they want).

That said, we have to work with what we have, so we will likely extend our code to allow downloading data given by direct link in KML, since requests for that have been popping up multiple times.

This will improve things for cases where a KML references an image, or where a KML references a different KML - and that second KML contains no further references.

We'll start with this and then see how many scenarios will be left uncovered. It's not that it is particularly hard to download the entire tree of all resources - it's that if this has to be done, there needs to be some control over the amount of data downloaded, there needs to be caching so that you don't download the same multi-MB file over and over and over, etc, it's that control over this dynamic downloading that makes it complex.

Dimitri


5,753 post(s)
#14-Feb-20 08:23

Quick update: Recursive KML reads have just been added in 9.0.170.4. That's a pretty quick turnaround, about two weeks from suggestion to new feature. :-)

From Changes and Additions:

Reading KML supports downloading data recursively referenced by URLs in the KML. The recursion limit is specified via a link depth option in the New Data Source dialog. For example, if a KML references a different KML that references a JPEG, to download it all, the link depth option has to be set to 2. By default, link depth is assumed to be 0 = download nothing. This is a safety measure to avoid KMLs that silently trigger downloads.

adamw


8,882 post(s)
#14-Feb-20 14:31

...and an immediate reminder - the default is to download nothing, so in order to link the file in the first post *and* have it follow its link, link it through File - Create - New Data Source. The setting for the link depth suggested in the dialog is 1 (because you can see it before you click OK, unlike if you link through File - Link, or programmatically via query or script that is unaware that the setting exists), for the file in the first post that will suffice.

antoniocarlos

527 post(s)
#15-Feb-20 16:37

Hi

I created these files with Global Mapper 18 (not the latest version). They look OK in Google Earth but I am having difficulties reading them in in M9 using the above process. I may be missing something.

Cheers.

Attachments:
JPGTest.kmz
PNGTest.kmz
TIFFTest.kmz


How soon?

Dimitri


5,753 post(s)
#16-Feb-20 06:45

This is a related issue to recursion. It is not the recursion per se, it is how people specify the paths to what they want to fetch in a link and what KML allows or does not allow.

The files you attached are all KMZ files that use a relative path within the <href> tags. They'll work if you first unzip them to create a doc.kml and the accompanying images, and then create a data source on that KML.

What's going on, I suspect, is that the KMZ is being decompressed on the fly into a temp folder, but then the relative path reference might no longer match the path to the images. Or, it could be the decompression of the zipped file (KMZ is just a zipped folder containing the KML and any accessory files) is not unzipping the images to the same folder, etc. If you just use the KML, there is no unzipping and the relative path within the <href> tags continues to work.

Technically, <href> as a tag within KML is supposed to be a hyperlink, so it should start with something like http:// or ftp:// or file:/// (for local files) depending on what you want to use. The use of relative paths is a convenience. To what extent that is an exception to hyperlink protocol I don't know, but the usual rule of thumb is that whatever Google does is the de facto standard anyway, so no point arguing with it. I've filed a suggestion that unzipping a KMZ should track whatever was the original relative path no matter what, just in case that is the issue.

You can force your KMZ files to work in all cases by converting the relative link to an absolute link. Suppose you've put your JPG files into a folder called C:\test ... you should then change every <href> within the KML to an absolute path. For example, change

<href>kml_image_JPGTest_L1_0_0.jpg</href>

to

<href>file:///C:\test\kml_image_JPGTest_L1_0_0.jpg</href>

Put that edited doc.kml inside a zip folder, and rename the zip folder to end in .KMZ and, voila! you have a KMZ that works no matter where you put it on your computer.

Note the above does not arise in recursive KMLs the way they most often are used, to refer to web resources, since those usually are fully qualified hyperlinks, starting with http:// or https://, that is, the hyperlink equivalent of an absolute path. But the issue might arise within web servers where further depth in links might involve a KMZ on the server.

If interested in writing hyperlinks that refer to local files, see this discussion in stackoverflow.

antoniocarlos

527 post(s)
#16-Feb-20 11:42

Thanks so much Dimitri.

This works. I did as you mention on your 2nd paragraph. Your explanation is also helpful for me to understand a previous one here about the problem with trying to code a Universal KML/KMZ importer.

Cheers.


How soon?

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.