Subscribe to this thread
Home - General / All posts - Importing txt files with custom delimiters
Forest
529 post(s)
#02-Sep-17 01:21

For some reason delimited text data is read only after it has been imported. Also there is a trick with the import dialog that needs attention.

I created a text file with the pipe symbol (|) as delimiter. If I select txt as the filter in the import dialog, the txt file is treated is imported as a comment component. If I set the dialog filter to csv, then the txt file is hidden. If the dialog filter is set to csv and then Data.txt is manually entered, then the data imports fine but the results are read only. I can add a geom field but can't run a transform to populate the field. I can't view my points.

Data is attached. Tested with Manifold Viewer 9.0.163.2.

Attachments:
PhotoCoordData.txt

tjhb

7,503 post(s)
#02-Sep-17 04:35

Is it OK to have to rename the file to *.csv?

If that is done, then a straight import seems to work flawlessly, giving the expected table.

It is not read-only. However, to be able to select and edit records, you must first add a BTREE index to the table, using Edit > Schema.

A BTREE can only be added on a field having unique values. If there is not one at the point of import (here you could use [FileName], that feels less than natural), you can add one afterwards. I think the manual has a tutorial on doing that. I will check.

[Added.] The simplest way is covered in Start > Basics > Add an Index to a Table. (Alternatively you could add a UUID. See the related link at the bottom.)

Dimitri


4,271 post(s)
#02-Sep-17 05:23

I created a text file with the pipe symbol (|) as delimiter.

A "csv" file that uses a pipe symbol | instead of a comma , symbol is a "psv" file. Name your file using a .psv extension, as in "myfile.psv".

Manifold Future and Future Viewer understand .psv and import it directly. See the release notes for the Future builds. Choose File - Import and then double-click on the .psv and it imports fine.

There are many topics that touch on making tables read-only by adding an index, such as this one in the Basics chapter. This is also covered in the Tables topic with links to other topics to read.

As the topic notes, "Tables without indexes are inconvenient: we cannot edit them and we cannot make selections in them, a typical situation in most DBMS environments, including Manifold. "

DBMS people get really annoyed when consumer software alters their data to "Do what I, the all-knowing software that is designed to save smartphone users from themselves, think you, the obviously inexperienced user, should want to do instead of what you commanded." Given that Radian, and now Manifold Future, are used for big-time data there is a reluctance to mess with the data on import to provide more the conveniences of a personal information manager than a DBMS.

But, for all the value of such rigor, I personally find it tedious to always have to add an mfd_id field and then an mfd_id_x index (or whatever you prefer to use) to imported tables. I'd like to see a one-click way of doing that or an options setting to have it done for me automatically.

Dimitri


4,271 post(s)
#02-Sep-17 05:49

to have it done for me automatically.

Forgot to add... the above is non-trivial. One reason DBMS people do not want their data altered by the package on import, for example, to add an index, is that with big-time data you need to think through such things very carefully.

It's no big deal with a few thousand records in a personal information manager to toss an extra field or an index into there. But if you've carefully set up your DBMS and the schemas are all carefully designed, you don't want extra fields suddenly materializing and new indexes being created. You have good reasons for what you have done and you don't want a package mindlessly changing that. If nothing else, if you have a billion or more records you might not want the overhead involved in building that index.

So, any automated addition of fields and indexes on import should not be a "one size fits all" thing. I agree there are people who will think "well, I never will be working with a billion records and I prefer the conveniences at what for me is the negligible issue of having an extra field and index appear in the schema." For them an automatic addition of mfd_id and an index is the right thing. For others, more options would be better.

Note, by the way, that to support the very wide variety of sources Manifold accepts you can't just say "OK, create the index automatically only if there are fewer than 100000 records..." because with many data sources you don't know how many records there will be until you read them, and you need a table with a schema to read them. There are ways around all that but it is surprisingly intricate.

It's like the new system of tracking when sources fail to assign coordinate system info and then automatically switching allowed coordinate system commands. It sounds really simple and works very simply at the top level, but underneath it required engineering that involved code for hundreds of sources.

tjhb

7,503 post(s)
#02-Sep-17 06:57

Dimitri, do your posts invalidate mine? Probably yes, but I'm not sure how.

Must we use .psv when pipe seperators are used? Since .csv seems to work fine. It's already wired up.

Any limitations seem to me to be instead about the existence and use of indexes. Is that wrong?

Dimitri


4,271 post(s)
#02-Sep-17 09:00

No, no conflict at all.

If your .csv uses pipes instead of commas, the .csv dataport will recognize pipes as delimiters as well. It wasn't clear if the original file used a .psv extension or a .csv extension. My point was just that if it is already a .psv, no need to rename it to being a .csv. Either will work OK.

Using a plain .txt extension tells Manifold to import it as a comments component (text).

From a purely pedantic viewpoint I can see how some might argue that "Comma Separated Values" as the meaning of .csv should most logically be reserved for files that use commas as delimiters. It's dissatisfying to have a whole menagerie of different three-letter extensions for each possible delimiter, such as .psv for "pipe separated values." Maybe just one, dsv for "delimiter separated values" would do the trick. But all that is just idle speculation as csv is what it is and to the extent psv has traction it is what it is as well.

Forest
529 post(s)
#02-Sep-17 11:58

Thanks Tim, adding the index was the trick.

I don' t use manifold to process billions of records. I use to manipulate data in ways that are difficult to do in other GIS systems. Recently I have been using Manifold Viewer for its convenience. It starts up in about one second whereas QGIS lately takes about 3 minutes. I would not underestimate the number of people who use Manifold for the same reasons.

Forest
529 post(s)
#02-Sep-17 12:36

The next issue is very similar. It appears that I can't symbolise a GPX file without adding an index. A GPX file is generally something that one only want in a map for reference purposes. I view manually adding an index to a data type that I never want to edit as a pain. Here is a screen clip of my track, can you see where I went with the default line symbol?

Attachments:
2017-09-02 21_33_57-[Project1] - Manifold Viewer.jpg

Dimitri


4,271 post(s)
#02-Sep-17 13:25

could you post the GPX file?

Dimitri


4,271 post(s)
#02-Sep-17 13:37

Recently I have been using Manifold Viewer for its convenience

Same here, although I usually use Future. Just this afternoon I needed to drive somewhere and I wanted to take a look at what parking might be like near there. I have several image servers on my favorite data sources so I popped open a map and dropped in a few layers.

You can look at something though Google in a browser, but you only get the Google views. If you want to look at exactly the same thing in Bing, OSM or whatever, you have to open a different browser session and navigate there.

I'm happy to use Google in a browser (I like the driving directions and drive times you can get), but surprisingly often when I just want to scope out a particular location it is more convenient to zoom in with Future or Viewer and just click tabs off and on to see different displays of exactly the same thing, the same view, to see which is the best in that particular place.

Hmmm.... here's an idea... why not allow launching Google Street View from a view within Future? :-) Nothing complicated about launching a browser session with a street view URL loaded with coordinates of what you clicked...

Forest
529 post(s)
#03-Sep-17 01:01

I could not get Manifold Viewer to format the trail yesterday but today it formats the trail without any issues. If I can find the sequence of operations that makes the formatting capability lock up, I will let you know and post the file. Usually, I have several goes at making something work before posting here.

Forest
529 post(s)
#03-Sep-17 01:21

Found it. If you add a gpx data source as read only, that locks down the data so that it can't be symbolised. This is a weird design choice. I cannot imagine wanting to add a drawing and not symbolise it so that I can see the layers. The error message is in the attached jpeg.

Attachments:
2017-09-03 10_19_22-[Project2] - Manifold Viewer.jpg

Dimitri


4,271 post(s)
#03-Sep-17 08:05

Found it. If you add a gpx data source as read only, that locks down the data so that it can't be symbolised.

I've edited this to keep it as simple as possible.

"Read-only" means "no changes" and "no changes" includes "no adding records providing styling info". That's why you get an error message of "can't add record."

How to command what you want:

There is a step-by-step example for that. Follow it and you can have your cake and eat it too: Apply the advice in the Persistent Styling section of this example topic.

What the above does is leave the original table in read-only form, protecting the table/data source from any changes. It creates a visualization that is read/write to which you can apply styling as you like. Save the project and the styling persists in the project, with zero changes to the original, read-only source.

adamw


7,215 post(s)
#04-Sep-17 06:57

Like Dimitri says, read-only data source means no changes to data at all, and formatting is part of data.

To add a bit, you can have:

  • both table and drawing on a read-write data source - then you can edit everything: records / geoms, coordinate system, formatting, etc,
  • both table and drawing on a read-only data source - can edit nothing,
  • a table on a read-only data source (ie, a GPX file linked as read-only) and a drawing referring to it on a different, read-write data source (ie, a MAP file) - then you can edit data that belongs to the drawing: formatting, but not data that belongs to the table: records / geoms, coordinate system.

There are also cases where a read-write data source may have read-only components. This happens, for example, when you connect to a database as a user with limited permissions.

Dimitri


4,271 post(s)
#04-Sep-17 12:03

To follow up on PSV vs CSV - the dataport is identical. In either case it parses the contents of the inbound file and if the delimiter is a pipe it uses that, and if the delimiter is a comma it uses that. Explicitly accepting files that end in a .psv three letter extension is just a convenience, so if anybody is concerned that a .psv cannot be used, etc., there is no need to worry.

Forest
529 post(s)
#05-Sep-17 02:53

Hi Adam,

I was expecting the third case, where the data is read only but the drawing can be symbolised.

As soon as two layers of the same type are placed in a map, one of the layers will need to be symbolised. I looked up the link in Dimitri's post for persistent styling and saw temporary styling was also an option. However I am not sure that if I import a gpx file as read-only that temporary styling is available.

I can understand three use cases:

- read only data and editable symbology;

- read only data and read only data driven symbology (eg. where some standard symbology is used); and

- read only data with read only layer based symbology.

I just wonder if symbology requires a separate read only flag from the data.

Often I want the data to be read only to maintain an original record but need to symbolise it in some form to be able to see it in a map. Read only symbology only applies if I am working on a large project or in a team. In this case, I need to be able to indicate that the symbology has been set to a standard and must not be changed. In other systems, I achieve a similar result by importing a style definition. I wonder if it will be possible to apply save styles to read only datasets.

Dimitri


4,271 post(s)
#05-Sep-17 06:49

I was expecting the third case, where the data is read only but the drawing can be symbolised.

No problem. If you do it as adamw suggests in the third case that is exactly what you get.

However I am not sure that if I import a gpx file as read-only that temporary styling is available.

Yes, it is available. That's the third case. The example of persistent styling also is the third case.

I just wonder if symbology requires a separate read only flag from the data.

We can only go so far without knowing precisely the workflow and files you are using. But it is important to remember a key distinction: "read-only" means just that. There is no "read-only except for these things I want to write." If you can write to the data in any way it is *not* read-only.

Respecting the one and only true meaning of "read-only" is very important because it is very important to be able to mark data as read-only and be absolutely sure it is not changed in any way. That will not get in the way of what you want to do, for example:

Often I want the data to be read only to maintain an original record but need to symbolise it in some form to be able to see it in a map.

That's the third case. Use that and you have the above.

In this case, I need to be able to indicate that the symbology has been set to a standard and must not be changed.

No problem. At some point, the author of the data has to have read/write form to have created it in the first place or to apply standardized symbology. The author does that and saves it in a format (.map, of course...) that is then marked read-write. Other users can open it and view it but not change the symbology that is in the data set.

I wonder if it will be possible to apply save styles to read only datasets.

If it is read-only you can't save anything to it. If you could, it wouldn't be read-only, right? If you want to edit a read-only data set the way to do that is to use your administrative privileges to make it read/write, edit it as you like (for example to save a style into the data set), and then change it to being read-only so others cannot alter it.

The other approach is the third way. Remember, if a file really is read-only you can't save anything to it. But, you can link it into a project where the project is read/write. In that case the approach to a styled version of the data is to open the project. That's fine for many purposes but it is not branding the data with the specific style that you want because the style info is kept in a separate project file.

Forest
529 post(s)
#07-Sep-17 03:02

I can't edit the styles used in a drawing based on a linked read-only gpx file. I get the can't add record error message that I have attached previously. I think that this is a bug. If it is not, it is an unhelpful error message. I understand the logic of what Adam says and since I am using Manifold Viewer, I am getting an enormous amount of practice importing/linking and creating new data sources and think that I am getting very familiar with the interface. I cannot add a temporary style to a read-only drawing.

Dimitri


4,271 post(s)
#07-Sep-17 06:21

I can't edit the styles used in a drawing based on a linked read-only gpx file. I get the can't add record error message that I have attached previously. I think that this is a bug.

Sigh... I'm not exactly sure how many times I can write the same thing in different words if you ignore what I write. I repeat for... what?... the fourth time?... You cannot modify a read-only data source. That is not a bug, it is a critically important function of the software. When someone declares a data source read-only he or she must have 100% assurance it will not be modified in any way. I hope you can see the wisdom and the non-negotiable necessity in that.

Now, that you can store styling information within the data source or within the project also is not a bug. That is a feature of great power, but like many features of great power it will apply, with great power, exactly what you tell it to do. If you tell it to do something that's contrary to something else you told it to do, it will complain.

For example, saving styling information means adding a record. I trust that given your interest in styling you've already carefully reviewed the example topics on manipulating the mfd_meta table and that you've watched the video on that, right? By doing that, you know that styling is saved in records in a table, like everything else, and also by reading those topics and watching the videos you have learned what a tremendously powerful and useful thing it is that styling properties are saved in tables. It's just fantastically useful, not something to throw away.

So, since you know such stuff is saved in tables, why the surprise that when you try to save it in a table in a read-only data source the software tells you that it cannot create a record?

Before we talk error messages, consider that Manifold lets you work with about 500 formats and data sources. It does indeed come to that when you count all the formats and all the data sources and then multiply by the very many variations some of them have. Many of those are highly sophisticated data sources that provide zillions of options.

Back to the error message: If Manifold was a smartphone app that did just one, itty-bitty thing, then of course it would be possible to provide a very simple error message. But given that when you try to modify a data source there are many ways you can instruct the system to do something contrary to what you have previously instructed it to do, it gets much more complicated to report that you've done something wrong without telling a lie. You want a helpful error message? Me too and everybody else, so we all agree with you. Helpful error messages almost always require telling the truth.

In this case, there are many reasons why you might not be able to modify a data source *other* than it being read-only. For example, the data source might be perfectly read-write for other people but not for you, or it might be read-write for certain fields but not some fields, or it might be perfectly read-write for existing fields but not allow modification of the schema to add new fields, etc, etc. Those are all legitimate things that people who administer data routinely want to apply. So there is no, one-size fits all, "Sorry, can't modify a read-only source" error message. The least common denominator is to report what actually failed "can't add record" and leave it at that.

Now, you may think "Oh, I don't want to use any of those sophisticated things, I just want to do this one thing and I want an error message crafted for me specifically in this case." I wrote that as if the request is daft, but, really, it is not daft at all. It's a perfectly fair request, and one thing that gets software a reputation for, despite technical sophistication, being easy to use is the accumulation of thousands and thousands of such small things. So sure, Engineering could step right in and write special code that examined data coming in from linked read-only GPX files, consider the various classes of things people might do with such files and then write a series of special case error messages to pop up. The only question is how happy all the other users would be about the application of such effort instead of the same amount of engineering time being spent on matters of broader interest.

For example, I would recommend not altering such error messages but instead provide infrastructure that made it easier to take read-only sources from outside the project and to echo them as read/write entities within the project. I told you how to do that but it appears you either did not read what I suggested or that it was too onerous to accomplish. I fully agree that if something is too challenging people will often blow it off. But the solution to that is not to pile on with more elaborate error messages, it is to make the task easier.

So, for now, if you want to assign styling you do have choices:

1. Use a read/write data source that allows storage of styling within the format. If you look at the new GPKG screenshot in the Radian Gallery page, that shows an example of styling that is applied to and stored within the very same .gpkg file downloaded from the New Zealand government. It is convenient to be able to give someone a .gpkg file and when they open it in Manifold they see the styling you assigned.

2. If you want tricky (ahem, "sophisticated") controls over who gets to specify styling with other data guaranteed kept untouched, use a sophisticated data source like a DBMS (PostgreSQL comes to mind) where you can, as administrator, have fine control over who gets to change what tables and even fields. You can then do very intricate administrative things like have standardized styling that cannot be changed by users even if they are allowed to add, edit and delete objects.

3. Leave the data in the read-only file but then arrange to have styling kept in the project, like in the links I recommended. The easiest way to do this is to on a one-time basis make the gpx read/write so you can create a geom in there, so it is possible to link a drawing to it in a very simple way, and then make it read-onlyso nobody can mess with the data after that.

I've offered to show you how to do this if you post the GPX. I'm mildly surprised that if this is a big deal you haven't done that.

By the way, on a philosophical note, to add value despite the limitations of GPX you surely can use Manifold. But when you store all that extra value stuff in a .map project you are undertaking the classic addition of "hair" that people do when, say, they want to get around the limitations of shapefiles by adding all sorts of accessory files, such as .prj or .lyr files. All those then must continue onward as an ensemble or you lose the value-added they provide. If, really, GPX has limits you don't like it probably is time to consider a more sophisticated format, such as GPKG, that does not have such limits.

Dimitri


4,271 post(s)
#07-Sep-17 15:23

Let me take a stab at this without even seeing the GPX.

1. Read the Example: Create a Drawing from a Geocoded Table topic. I'll use the Cities geocoded table from that example as a stand-in for your GPX table. Suppose your GPX table is called Dots.

2. Suppose that table is in a read only format that you link into the project.

3. Copy the table and paste it into the main body of the project. Rename the pasted table DotsGeom. You now have a table called DotsGeom that for each record has a Name, Longitude and Latitude. It is fully read/write. Edit the schema to add a field called Geom of type geom.

4. Right click onto the DotsGeom table and choose New Drawing. Check the box "create spatial index" and assign Latitude / Longitude as the coordinate system. Press Create Drawing.

5. Create a query called UpdateDotsGeom that contains the following SQL:

-- delete old records

 

DELETE FROM [DotsGeom];

 

-- insert new records

 

INSERT INTO [DotsGeom] (

  [Name][Latitude][Longitude][Geom]

)

SELECT

  [Name][Latitude][Longitude]

  GeomMakePoint(VectorMakeX2([Longitude][Latitude]))

FROM [Dots];

Now, if the Dots table is in a linked data source called GPX you'll change that last line so it reads

FROM [GPX]::[Dots];

Whenever you want the latest and greatest from your linked, read-only Dots table, run the UpdateDotsGeom query. The DotsGeom Drawing will automatically show what's new. You can Style the DotsGeom Drawing however you like because it is a read/write thing in the project, not in the read-only linked data source.

When you run the query it begins by deleting all the old records from the DotsGeom table so there are no collisions from the INSERT INTO. Then it inserts new records from the table, computing on the fly the Geom from the latitudes and longitudes. Because an index was created when we created the drawing, there is an index on Geom that the drawing can use to display. We don't need to worry about the mfd_id field because that happens automatically.

Where did the above query come from? It is an adaptation of what adamw posted in an earlier thread when you asked about an MDB. The GeomMakePoint usage comes from what you get when you push the Edit Query button in the example cited at the top of this post, instead of just running the Compose Point template... it is the SQL that the Compose Point template uses. If I can't remember how something goes together I just do a similar Transform template, push the Edit Query button to see what it writes and that reminds me.

In the above workflow I use dialogs to do some of the infrastructure work. For example, I copy the read-only Dots table and paste into the project to get a read/write table that has the same field names and data types, and then I use the Schema dialog to add a Geom field. I use the New Drawing dialog to create the drawing while adding a spatial index and assigning the coordinate system as Latitude / Longitude so I don't have to remember how to do all that in SQL. It's nice to be able to mix simplistic use of dialogs with a snippet of SQL to reduce the learning curve.

Hope that helps! (would make a fine video, I think...).

Forest
529 post(s)
#08-Sep-17 04:52

Hi again,

Please remember that Manifold is not my only frame of reference. I was a heavy Arc user and now use QGIS for cartography, with Manifold used mainly for heavy lifting.

QGIS allows themes to be created on read only datasources and these themes are stored in the project file. in QGIS, one can also add extra fields to a table to store formatting information. I often add extra fields to control label rotation and offset so that I can independently customise each label. In this case, if the data file or source is read only, then the formatting would be not changeable. All this is fairly transparent in QGIS.

Looking at manifold future, when read only is checked, it makes all the contained components read only including the system data tables that are in the manifold project. It seems that with a gpx file, that even if the data is not checked read-only on import, that the data is read only in Manifold Viewer. As gpx files are subject to custom extensions and are also subject to device related issues that are nothing to do with the definition of the gpx format, I would be suspicious of applications that can add new features to gpx files. So in future, I will not worry about making gpx data read only.

I am not trying to pull anyones chain. I am being very clear about how a QGIS user upgrading to Manifold could come unstuck. I suggest that a small change be made to the wording associated with the read only checkbox in the new data source dialog to indicate that not only are the external data files read only but any formatting associated with the data source is also locked (Open Data and Styles as Read Only? or "Set Components to Read Only"?). The "Can't Add Record" should not appear as the edit styles menu should be greyed out for read only components. It is a case of having to unlearn slightly difference conceptualisations of the same process and as you will know that slight differences between similar systems often cause more grief that more obvious difference.

I can imagine that once people start linking maps files together and creating libraries of styled data in manifold map format this will be a great feature. An example GPX file is attached if you want to see what I work with.

Attachments:
2017-02-18 08.59.30 Day.gpx

Dimitri


4,271 post(s)
#08-Sep-17 10:37

Please remember that Manifold is not my only frame of reference.

When operating a sophisticated, powerful product the frame of reference you *must* use is that product, and not some other product that works differently.

Example: Love Gimp? Great. Me too. Try to operate the latest Adobe Creative Cloud software by applying Gimp commands thinking "this should be the same as GIMP" and it won't work. You *must* operate Adobe using Adobe's instructions, not Gimp's.

Example: Love SQLite? Super. It has a lot to offer. Trying to operate Oracle Exadata Cloud by applying what you know of SQLite, without careful, serious study of Oracle documentation, and that won't work. You *must* operate Oracle on the basis of careful study of Oracle.

In the above examples there is nothing wrong with any of the four software packages cited. Gimp, Adobe, SQLite and Oracle are all fine for the purposes for which they are intended. But they are different from each other in many characteristics big and small. You get nothing but frustration and you will fail to gain the power each of those packages offers if you try to work it as if it were some other package.

Let's get back to specifics:

Why do you use Manifold for heavy lifting and not Q? You do that because Manifold can do things that Q cannot and you get that capability because Manifold is different from Q.

You can do what you want in Manifold, and usually with greater power, flexibility and reliability than Q, but you have to learn how to do what you want in Manifold.

I suggest that a small change be made to the wording associated with the read only checkbox in the new data source dialog to indicate that not only are the external data files read only but any formatting associated with the data source is also locked (Open Data and Styles as Read Only? or "Set Components to Read Only"?). The "Can't Add Record" should not appear as the edit styles menu should be greyed out for read only components.

I'm always in favor of better captions but they have to accurately reflect how the product works. The above does not.

Checking the read-only box doesn't mean that "any formatting associated with the data source" is also locked. That's not how it works. There are many ways of associating "unlocked" formatting with a read-only data source. For example, you can store the formatting in the main part of the project and not within the read-only data source. I gave an example of one way to do that.

The read-only checkbox is simple: the data source you create will be read-only. Nothing in that data source can be changed. Everything in that data source is read-only. If that is not what you want, don't check the read-only box. People have to be able to trust 100% that if they command a data source is read-only, then absolutely without exception that command will be obeyed.

Looking at manifold future, when read only is checked, it makes all the contained components read only including the system data tables that are in the manifold project.

No. Creating a read-only data source makes what is in that read-only data source read-only. It does not make anything else in the manifold project read-only, it only applies to the data source that you created and commanded be read-only.

This is easy to see: a read-only data source has a lock on the database cylinder icon. Expand that data source and everything within is read-only.

QGIS allows themes to be created on read only datasources and these themes are stored in the project file. in QGIS, one can also add extra fields to a table to store formatting information. I often add extra fields to control label rotation and offset so that I can independently customise each label. In this case, if the data file or source is read only, then the formatting would be not changeable. All this is fairly transparent in QGIS.

I don't get what you mean by the above as it contradicts itself. For example, you say Q can add extra fields to a table to store formatting information for read only sources. But then you say "if the source is read only then the formatting would not be changeable."

I think what you mean is that if a source is read-only, Q can store formatting info in a project file outside of the read-only source. Great. So can Manifold. You can see from the videos and topics I mentioned that sort of thing is extraordinarily transparent in Manifold, apparently more so than Q.

Manifold works differently than Q for very good reasons, many of which revolve around serving the sophisticated demands made by people who want to take advantage of powerful, serious, modern methods for getting the most out of their data. Learning how to use all that gets you more power, greater ease and more flexibility, not less. Once you learn how to use a more powerful tool it usually is much easier to do what you want, and to do it quicker as well.

Thanks for attaching the GPX, but it is not the original data. Somebody has already been playing with it, to add geom fields and such. Do you have the original data to post?

Dimitri


4,271 post(s)
#08-Sep-17 12:14

Thanks for attaching the GPX, but it is not the original data. Somebody has already been playing with it, to add geom fields and such. Do you have the original data to post?

Ah, sorry... my mistake. The data seems to be original. I should have taken a closer look.

Having sample data helps a lot. This is simpler than you think it might be, because the dataport for GPX automatically adds a geom, so you can just use the read-only table as it comes in from the GPX with no need for any clever SQL or queries, etc.

The solution to what you want is easy:

1. File - Create New Data Source using the .gpx. Check the read-only box.

2. A read-only data source appears in the project. In the main part of the project right-click and choose New Drawing.

3. Change the Coord system to Latitude / Longitude. Press Create Drawing.

4. Right click on the new drawing and in the Properties change the table it uses from [Drawing Table] to refer to the table for the drawing you want to use in the data source, for example,

[Data Source]::[2017-02-18 08.59.30 Day TrackPoints]

5. Done. You can now style Drawing however you like. You have a drawing in the project that takes its data from a read-only table in the data source.

In the photos I show the original, read-only drawing open with Drawing next to it, styled to use yellow fill color.

I also show the Properties for drawing (after styling, so it includes style properties) so you can see the property for Table.

It's hard to believe we've gone around and around on this for two weeks. :-)

Attachments:
gpx01.png
gpx02.png

Forest
529 post(s)
#09-Sep-17 02:03

Hi Dimitri,

Thanks for persisting. Please note that I cannot add a new drawing that is bound to the above mentioned GPX file in the 9.0.163.2 Manifold Viewer when the data source is added as read only. I suspect that this is a limitation of the viewer.

I also cannot add a raster layer to the map that created when a gpx data source is added as read only (as expected).

I don't really care as I know that I don't have to use the read only checkbox. This is just to say that I have followed your instructions exactly (as I usually do) and got a different result. What you have described is the behavior that I expected and had actually tested before I made my previous post.

Dimitri


4,271 post(s)
#09-Sep-17 09:15

Please note that I cannot add a new drawing that is bound to the above mentioned GPX file in the 9.0.163.2 Manifold Viewer when the data source is added as read only. I suspect that this is a limitation of the viewer.

It is not a limitation of Viewer. My guess is that you have missed some detail... Could it be that you did not right click into the main part of the project (that is, outside the read-only data source) to create the new drawing?

Above is an image showing 9.0.163.2 Manifold Viewer with a read/write, styled drawing that takes its data from the read-only (note the lock icon) GPX data source that is your file.

Perhaps if you write everything that you are doing, step by step, that will clear it up. This is a very short workflow so that will not take long.

Attachments:
viewer.png

Dimitri


4,271 post(s)
#09-Sep-17 09:24

I also cannot add a raster layer to the map that created when a gpx data source is added as read only (as expected).

Yes, of course. It seems worth repeating: When you command a data source to be read-only that means it really will be, 100% guaranteed, read-only. "Read-only" means no changes. That little "lock" icon on the data source icon is there to remind you that the data source is read-only. No changes to anything in that hierarchy. You cannot add or delete to it.

You can, of course, in one second create a map outside of the data source you have commanded to be read-only, and then whatever layers you want to that, including raster layers, as seen below:

Here are pictures of a map that has a raster image as the background layer with the styled drawing created from the read-only data source...

...and the layer taken from within the read-only data source. Both are in Viewer.

Attachments:
viewer2.png
viewer3.png

Forest
529 post(s)
#09-Sep-17 23:27

I have always worked in the regions and have dealt with GIS specialists in head office and I have learned that you need to talk more than otherwise necessary in this case to ensure that there is proper understanding. Some of what I have said is just to let you know that I am on the same page as you. I have heard you say similar things about bug reports.

In my last testing run something happened and Manifold 9.0.163.2 lost its ability to import data. I could not add a new data source, link or import data. I could not open a sample radian studio map file either (books.map). I was using the 32 bit version and when I closed that and used the 64 bit version, the situation was the same. I suspect that for some reason that creating a new project failed. Rebooting did not help. I have attached a screen shot showing 9.0.163.2 in the foreground and 9.0.163.3 in the background. I have just downloaded 163.3 and usually delete non-latest builds as soon as they are superseded to make sure I do not supply outdated issue reports. I was using 163.2 and one minute it was fine and the next it was basically down.

If I follow your instructions to the letter with 9.0.163.3, then I cannot create a new drawing that links to a read only data source by right clicking in the main part of the project pane. The read only data source is not listed in the 'Based On' drop down list. See attached screen print. Even if I first left-click on the system data table in the main part of the project screen before right-clicking (make sure that something outside the locked folder tree is currently selected), I cannot create a drawing that references the read-only gpx file.

Attachments:
2017-09-10 08_04_44-[Project1] - Manifold Viewer.jpg
2017-09-10 08_16_14-Program Manager.jpg

tjhb

7,503 post(s)
#10-Sep-17 00:20

The screen shot shows that you are not following Dimitri's instructions, so in a way that's helpful.

1. File - Create New Data Source using the .gpx. Check the read-only box.

2. A read-only data source appears in the project. In the main part of the project right-click and choose New Drawing.

3. Change the Coord system to Latitude / Longitude. Press Create Drawing.

4. Right click on the new drawing and in the Properties change the table it uses from [Drawing Table] to refer to the table for the drawing you want to use in the data source, for example,

[Data Source]::[2017-02-18 08.59.30 Day TrackPoints]

5. Done. You can now style Drawing however you like. You have a drawing in the project that takes its data from a read-only table in the data source.

At step 3, you're trying to make a drawing from the read-only dataset directly. You can't, because tables from read-only databases do not appear in the 'Based on' list. (Should they? Could go either way on that.)

But what Dimitri says is just to press Create Drawing. It doesn't really matter for now what the table is, in this case it is just pro forma. The drawing will have no content for the moment, until...

At step 4, you adjust the drawing's property to the name of the target table within the read-only database.


By the way, you can create the drawing referencing the table in the read-only database in one step, using SQL. In this case that might be clearer than using the GUI.

To see how to do it in SQL, first do it in the GUI following Dimitri's instructions, then Copy the resulting drawing in the Project pane, open a new Command window (Ctrl-~), and Paste into the code pane (top left). That will write out the necessary SQL for you.

In fact, if you want you can apply formatting to the drawing after creating it, then do the copy and paste, and you'll get SQL for the formatting as well. Very handy, and a great learning tool.

Dimitri


4,271 post(s)
#10-Sep-17 05:53

You can't, because tables from read-only databases do not appear in the 'Based on' list. (Should they? Could go either way on that.)

I suppose in theory you could go either way but in practice there is a lot of utility in having the box work on the selected context, without including all contexts in the project. Keep in mind that if a project has a data source linked to some archival collection of nested and subnested data sources you could have a zillion tables in that hierarchy. You don't want all of them appearing in the Based on box, so then you'd need even more elaborate rules to choose which are shown and which are not.

The way it is now is the 'based on' list applies to the context data source, analogous to how most dialogs work. For example, when you are working on a table, launch Transform and then choose from a pull-down list of Fields in the Transform dialog it shows you a list of fields in that table, not in all tables in the project.

A project that has several external data sources has besides those data sources the context of the project file itself, that is, the local context. If you right-click in the main part of the project to get a context menu and then choose New Drawing, that New Drawing dialog is launched in that local context and it shows you Based on options for that local context.

If you right click onto a table within an external data source (now your context is that data source, not the local context), and that data source is connected via a dataport that allows creating new drawings in the data source, then you can click on New Drawing and the Based on list will show you tables within the external data source but it won't show you tables in the local project context and it won't show you tables in some other data source.

All that is plain and simple, just the way pretty much everything works in Windows. For example, if you open two different File Explorer windows and in one you navigate to C:\Program Files and in the other you navigate to C:\Users you expect the C:\Users window to show you files and folders in the Users folder context. You don't expect it to show you a much longer list that shows everything in the computer with items from other folders using an expanded notation like [C]:[Program Files]::[...

When dialogs show those items directly relevant in the context of interest they help people be productive: File Explorer would be useless if instead of only showing those files in the context chosen it tried to show everything all of the time. That the "Based on" box shows tables in the context of interest is similarly useful, and especially so if you are working within data sources that are nested many levels deep.

Note, by the way, that the New Drawing dialog / Based on box is not just a dumb reaction like "whatever is in the data source is OK." It has intelligence about what is possible in a data source based on the capabilities provided by the dataport. Create a read/write data source on a gpx file [do *not* check the read-only box] and you cannot create a new drawing there because gpx does not support that. But, create a read/write data source on a gpkg file and you *can* create a new drawing because GPKG is a far more capable format.

As Tim points out, those who want to do absolutely anything can use SQL. They can use expanded notation to refer to any table anywhere at any time.

tjhb

7,503 post(s)
#10-Sep-17 06:44

You are right Dimitri. After reading your reasons I think it can only go one way, only one way can be intuitive and also manageable.

If lots and lots of users had trouble with this (?), maybe there could be an extra option in the "Based on" list, something like <none>, to be used to create a disembodied drawing. I don't know if that would prevent confusion or cause more.

Dimitri


4,271 post(s)
#10-Sep-17 14:51

something happened and Manifold 9.0.163.2 lost its ability to import data.

A good rule of thumb is that software does not suddenly change functionality with no cause whatsoever. There's always a cause, and most of the time that cause is some change the user has made to a system.

While by an overwhelming margin user changes are to blame for "didn't touch it! it just changed by itself!" phenomena, other common causes are changes made by automatic updates to software such as in anti-virus software, changes in permissions, Windows or other software updates, or the installation or de-installation of any software. Those can be really hard to detect.

In rare cases, malware infections can cause perfectly good software to stop functioning or to operate with bizarre effects.

Ways to debug such issues include 1) jumping back to a prior restore point when all was good, 2) a meticulous consideration of every last danged detail that was changed between when all was good and any problem happened.

Example: "163.2 no longer can open any files, create a new data source, link or import data." "Did you make any changes?" "No, absolutely none." "OK, I see you report using 163.3. Did you install 163.3?" "Yes. ...Does that count as a change?"

Yes, it does count as a change. Somebody who installed 163.3 might have gotten confused while copying and pasting files when moving a portable installation around and managed to have overwritten some of the files in a 163.2 installation with 163.3 files.

You may think nobody would be daft enough to do that on purpose, but hey, accidents are not a matter of being daft or not daft. Sometimes they are just a matter of not noticing that what was an intended Ctrl-C for copy was done as a Ctrl-V for paste because that finger was just one one key over from where you thought it was.

The worst thing about such accidents for debugging strange situations is that because you didn't notice in the first place that finger was on a V instead of a C key, now you might be 100% certain no change or error was done. It's those changes you don't remember making or didn't notice making that are the hardest to find.

I was using 163.2 and one minute it was fine and the next it was basically down.

You're lucky the time span is so short. The shorter the time span the better chances to discover what you or what something else did to nuke 163.2.

Another approach is if 163.3 works fine, not to worry about what nuked the 163.2 installation. It might have just been a random accident, like confusing Ctrl-V to paste with Ctrl-C to copy that will never be found and won't be repeated.

Forest
529 post(s)
#11-Sep-17 04:19

Thanks Tim,

I could not see what I was doing wrong. I had to enter the full string into the properties i:e [Data Source]::[2017-02-18 08.59.30 Day TrackPoints].

Cheers

Forest
529 post(s)
#11-Sep-17 04:26

Hi Dimitri,

I had the issue with 162.2 ceasing to work before I loaded 162.3. I loaded 163.3 just in case the issue had been fixed. Perhaps the behaviour will never be seen again but I am not in a position to determine if the event actually is significant so I report it anyway. Usually I do things a few times to ensure that the result is repeatable before writing anything here. 163.3 looks good.

adamw


7,215 post(s)
#10-Sep-17 09:06

The discussion moved on a bit, but I just want to add a small note:

I suggest that a small change be made to the wording associated with the read only checkbox in the new data source dialog to indicate that not only are the external data files read only but any formatting associated with the data source is also locked (Open Data and Styles as Read Only? or "Set Components to Read Only"?). The "Can't Add Record" should not appear as the edit styles menu should be greyed out for read only components. It is a case of having to unlearn slightly difference conceptualisations of the same process and as you will know that slight differences between similar systems often cause more grief that more obvious difference.

We agree that when formatting cannot be changed because, for example, the data source containing the component is read-only, then the dialog should not allow you to change it in the first place and the controls should themselves be read-only. The "Can't add record" error will not appear.

We are also considering adding a readout to the formatting dialog indicating why the controls are read-only with a button to create a new component in the data source that is the parent of the read-only data source, which would have writable formatting.

Ideally, in your case you would (a) link a GPX file (same as now), (b) open the drawing (same as now), (c) go to change formatting and see that the controls are read-only (an addition to what we have now), (d) see something like "<name of GPX data source> is read-only. Click here to create a writable copy of the drawing in <MAP file>." at the top or bottom of the dialog, click to create a copy of the drawing (an addition to what we have now), and set the formatting there.

We will also allow copying and pasting just the drawing from the GPX file into the MAP file and have the pasted drawing automatically adjust and point to the table that remained in the GPX file.

Dimitri


4,271 post(s)
#10-Sep-17 13:22

Ideally, in your case you would (a) link a GPX file (same as now), (b) open the drawing (same as now), (c) go to change formatting and see that the controls are read-only (an addition to what we have now), (d) see something like "<name of GPX data source> is read-only. Click here to create a writable copy of the drawing in <MAP file>." at the top or bottom of the dialog, click to create a copy of the drawing (an addition to what we have now), and set the formatting there.

It's much simpler than that. Just disable Edit - Style for read-only drawings. That's what the other commands do.

For example, if you right click onto a table within a read-only data source there is no option to create a new drawing. All the New ... commands in the context menu are disabled, because the data source is read only. That's true of the File - Create menu as well. If the context data source is read-only, all the Create - New ... commands are disabled. It could be the same for Style.

Besides that, I'm all for one-click means to echo in the context of the .map a read/write version of a read-only data source drawing or table. This is useful for things like custom styling of image server images.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.