Subscribe to this thread
Home - General / All posts - Mad idea - version control for queries, scripts, schema etc
rk
621 post(s)
#15-Mar-18 07:29

I'm mad, because yesterday when I took a 2h break from coding for hobby, Win10 had to restart without my consent. I had M9 session unsaved for several hours - I know, but the changes I make to data (quite big) are mostly temporary, I cannot bother to save them until I have reached desired state. Code however, is different. Every change is precious.

I have used git when my scripts are files.

I'm also forced to have many .map files ("branches") active and if something comes up I go and help to make some change or another in that particular project. I try to copy the changes to our template ("master") but often forget (or change is incomplete, or smth).

I have wished for a way to compare projects. Not data, but queries, scripts and table schemas.

I have thought of exporting .mml-s (get rid of data first) and comparing them with diff - I suspect, it wont work nicely.

Exporting mfd_meta would give snapshots of queries and scripts but not schema. I think I tried it once. It could have saved me yesterday. Comparing diffs of mfd_meta.csv's won't still be as pleasant as comparing diffs of single scripts.

Thanks for indulging me.

adamw


10,447 post(s)
#15-Mar-18 07:49

I think this is good material for an add-in script:

* Create a new text file.

* Take all components in the active MAP file, sort them by name.

* Go through tables, write their schemas into the file. Use some separators.

* Go through queries, write their code into the file.

* Go through scripts, write their code into the file.

This can then be put into version control.

adamw


10,447 post(s)
#15-Mar-18 08:24

Example:

Download the attached archive. Unzip it, place the CS file under ~\shared. Either restart 9 or do Tools - Add-ins - Rescan (you don't have to close anything, can continue working with the MAP file you have opened). You should now have a command named Save Code somewhere under Tools - Add-ins. Invoke it and it will create a text report of the currently opened MAP file (as in the above post) and place it under 'c:\data' folder - you might want to edit the path, it is at the bottom of the file. This should work with version control pretty well.

You can extend what gets reported and how, obviously.

Attachments:
Save Code.zip

tjhb
10,094 post(s)
#15-Mar-18 08:26

God's work as always.

mdsumner


4,260 post(s)
#15-Mar-18 13:39

{like button pressed}


https://github.com/mdsumner

rk
621 post(s)
#15-Mar-18 09:39

Oh, my!

Thank you.

rk
621 post(s)
#15-Mar-18 12:31

Before:

Sneaky little windowses.

Wicked. Tricksy. False.

Took my precious changes.

55 minutes later (sic!):

Every change is sacred

Every change is great

If change's not commited,

Git(*) gets quite irate.

(*)The Boss

rk
621 post(s)
#18-Mar-18 01:01

I adapted the code. The result is here on github.

  • clone and build it

or

  • simply copy Dump_Code.cs under ~\shared and copy Newtonsoft.Json.dll(*) under ~\bin64

(*) I expected more use of this JSON library. Right now it is used only in one place. Should be easy to get rid of this. Read more here.

Attachments:
Dump_Code.zip

adamw


10,447 post(s)
#20-Mar-18 15:25

Nice. :-)

The answer to "what else should be escaped?" in PropertySubClause - single quote. We will expose a function to escape string values for use in SQL, what we use also converts unprintable Unicode characters into '\uXXXX'.

You can parse JSON using Application.CreatePropertySetParse.

rk
621 post(s)
#22-Mar-18 19:51

New version uses CreatePropertySetParse and escapes single quote. Simply copying Dump_Code.cs under ~\shared is enough.

Attachments:
Dump_Code.zip

ColinD

2,081 post(s)
#15-Mar-18 07:53

Win10 had to restart without my consent

Happened to me yesterday as well. Very rude. Fortunately I had saved the project.


Aussie Nature Shots

KlausDE

6,410 post(s)
#15-Mar-18 09:57

... and on restart the explore ui was frozen at least long enough for me for a cool reboot with power-off.

BTW do we have any events (and an example like the code of this first Mfd9 Add-In) for our own add-ins?


Do you really want to ruin economy only to save the planet?

adamw


10,447 post(s)
#15-Mar-18 14:22

No events yet.

You can start a timer and check component versions to detect changes to data. This is likely going to stay that way (I guess we can create helper classes to do this and fire events on detected changes).

We will add events for UI actions - a notification for "active window has changed" and similar things. We have several ideas for handling user input (we don't really like click events, they are too primitive and require more code than necessary).

dchall8
1,008 post(s)
#15-Mar-18 14:45

Win10 had to restart without my consent.

And people in the office ask why I insist on Win 7 for my OS. With Manifold 9 it saves so fast, I could do it every 10 minutes or so. But on M8 my project takes anywhere from 8 to 40 minutes to save. M9 is going to be great.

tjhb
10,094 post(s)
#15-Mar-18 17:33

Never use Save, always Save As.

(It's always a good time to nag about this, if it might save even one person from losing a day or a week of work.)

StanNWT
196 post(s)
#15-Mar-18 19:08

As a 14 year user of ArcGIS, I can attest to "Save As". Complex documents can get so easily corrupted, that having date based documents and version numbers for versions even within days, has some efficacy as I've learned in my 20+ years of working in GIS. ArcGIS MXD files get corrupted so easily. That's not the case with Manifold projects, that I've seen so far but the reality of forced system reboots because of what MS does with Windows updates, or forced corporate reboots that happens, using "Save As" regularly makes good sense, I do it with a software programs.

Manifold_file_v1_Mar15_2018.map

As an example nomenclature.

Each iterative major change give it a new version number, when the day changes change the date reference. Also if you export PDF maps, make sure the Manifold name version and date reference is in the filename of the exported maps, that way you can more easily chase down versions of exported maps to their original project files. I can't tell you how many times I've had someone in my organization want a stylized bit of this version, a stylized bit of another version all fused together into the newest map. You get the idea.

danb

2,064 post(s)
#15-Mar-18 19:16

I always use save then z7/zip the map file


Landsystems Ltd ... Know your land | www.landsystems.co.nz

Mike Pelletier

2,122 post(s)
#15-Mar-18 19:47

How about a checkbox in the save as window to append the date to the filename?

danb

2,064 post(s)
#15-Mar-18 21:01

Or a checkbox to automatically create a mxb in the same location (as a background process separate from the GUI) if the save is successful.


Landsystems Ltd ... Know your land | www.landsystems.co.nz

adamw


10,447 post(s)
#16-Mar-18 07:14

Just do File - Export Project, MXB then instead of File - Save As. You can open MXB directly.

adamw


10,447 post(s)
#16-Mar-18 07:22

Not sure about the checkbox. The original intent seems to be to write a new file no matter what, and any option that tries to help with that will conflict with normal saves: the user would be frequently turning the checkbox on and off - might as well just have two different commands (Save / Save New Copy or whatever).

Maybe something like this instead: convert the message box that pops up when you are trying to overwrite an existing file from 'Overwrite? Yes / No' to 'File X already exists. Overwrite / Save as X1 / Cancel'.

tjhb
10,094 post(s)
#16-Mar-18 07:34

Absolutely ideal Adam.

Mike Pelletier

2,122 post(s)
#16-Mar-18 14:20

Yup, that is the ticket.

dchall8
1,008 post(s)
#16-Mar-18 15:00

Save As new file name is a Manifold 8 thing, because sometimes M8 hangs in the middle of a save but not during a Save As. Is the hang still an issue in M9?

And is there some confusion over the Save and Save As selections? The only suggestion was to append the date to the file name during the Save As. My file names have the date and version number. If I'm going to change the version number anyway, then incrementing the date by one day isn't much of an additional burden.

adamw


10,447 post(s)
#20-Mar-18 15:31

Neither 8 nor 9 should hang during a save, but 9 should behave better because 8 has to reopen the file after the save (which has to refresh linked and shared components, which both takes time and can fail, and can take a long time to fail due to timeouts being long - you normally want timeouts to be reasonably long, but this backfires here) and 9 doesn't have to do it.

9 might currently look like it has hung during a big save, because there is a phase during the save which is not responsible to cancel. We will improve this.

dchall8
1,008 post(s)
#16-Mar-18 15:04

I believe it was you who posted this tip many years ago. I've been happily doing it ever since. I have not done it with M9, yet, but I don't have any important files in M9. I'm hoping M9 will not have the same problem as M8.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.