Subscribe to this thread
Home - General / All posts - Using Manifold for non-GIS tasks
Forest
625 post(s)
#30-Jul-20 02:37

I am currently doing some computer vision work and am using Manifold for non-GIS work. I just want to say that it works extremely well. I was using it yesterday to read json files and Manifold pulls them in and converts them to useful tabular format better than any of the other tools that I use. Manifold also has always been in my toolkit for cleaning up messy attribute tables. It is more heavy duty than ms-access and has a lot of additional functionality. Just a shout-out to the team. Thanks for your good work!

dale

630 post(s)
#31-Jul-20 10:31

Hey Forest,

You able to give some examples? Sounds interesting.

dchall8
1,008 post(s)
#02-Aug-20 18:28

Using the appraisal district maps I found literally thousands of miscoded fields in the data which resulted in significant errors in school district funding as well as emergency services. While the maps illustrated the errors, correcting them was entirely done in the data tables. I copied the data to Excel or to paper (sigh) so the clerks could make the changes.

Forest
625 post(s)
#03-Aug-20 05:03

Hi Dale,

This what I am doing.

I am looking into computer vision for plant species identification. I have a high spec’d Alienware laptop and have installed everything that might be useful, and I know how to use a wide variety of applications. Manifold tends to fill the cracks or help me investigate datasets that I cannot view well in other environments.

Computer vision uses convolution as the first step to image identification. Manifold calls these kernel operations. With Manifold, I can play around with kernel operations and see which features they are picking up.

The actual issue that prompted the post was a competition on Kaggle for using machine vision to guess the genus of an unknown species. Here is a link to the repo. https://www.kaggle.com/c/inaturalist-challenge-at-fgvc-2017/data

The data for the competition is accessed via json files which have the URL for the sample input photos. So, I reached for NotePad++ and opened up the big json file. It took a long time and then I found that the json file had been linearized, which means that all the line breaks had been taken out and all the data was on one very long line. No problem, I would just add the json formatting tool to NotePad++, but the security settings will not let me install plug-ins. I was not happy but had to find an alternative. I tried Manifold and it very quickly imported the data and put the data into tabular format which was very useful.

There is not much GIS in what I am doing, however there is a great deal of relational database work. I developed my own version of the Altas of Living Australia (before they appeared) in ms-Access. Now I want to be able to leverage those older systems but want to use Postgres as a backend as it scales better and works seamlessly with a lot of open source technologies which I want to use. Using Manifold, I can import tables from both Ms-Access and from Postgres and write queries that use data from both backends. I cannot do this in pgAdmin or in dBeaver, which is the other database management system that I use. In Manifold, I can also drag data from gpkg files into postgress and I am hoping that I can drag non-spatial tables from ms-access into postgress too. I have not tested all the tasks in front of me.

For every task, there should be at least two software applications that can do the task. I work hard to eliminate potential single points of failure. Manifold is often the Swiss Army knife that covers the provision of a second means of doing key tasks. And as mentioned, it is often the primary tool for importing data and exploring it.

tonyw
736 post(s)
#31-Jul-20 16:55

I just want to say that it works extremely well.

I've been thinking similarly, that Manifold's ability to handle extremely large tables very rapidly along with SQL, relations, transformation templates, and filters would be great for handling and analyzing large databases. I don't have such a project yet so currently I'm in the state of having a solution looking for a problem to solve.

hugh
200 post(s)
#31-Jul-20 23:23

where I use Manifold frequently like this is working with the tables in US Census file geodatabases like the ones here. This is one for Florida 1.58 GB in image open and ready for table work:

Attachments:
gdb.jpg

hugh
200 post(s)
#24-Aug-20 06:31

I should add that these US gov't databases are only available in Esri file geodatabase format and before M9 were not accessible by users unable to afford expensive software to access them.

Dimitri


7,413 post(s)
#02-Aug-20 10:22

I also use Manifold a lot for non-GIS tasks. Some examples:

1. Personal information manager - Anytime I need to keep track of something I keep it as a table in Manifold. It's easy even for just a handful of notes.

Suppose I'm planning a trip somewhere... there's a lot of information like reservations, addresses, URLs to hotels, local restaurants, and so on. It's nice to have all that at my fingertips in a project. Manifold .map format is so durable that I like having it in a format that is reliable. This is also convenient because many times what I'm keeping track of has some sort of location element to it, mixed in with many records that aren't spatial.

Sometimes there's nothing spatial about it at all. For example, I do research on various topics of interest, both personal and business, and such research often involves fairly extensive sets of online and offline resources. Need to buy a new dishwasher and researching vendor and product options? Keep track of all the URLs, links to reviews, prices and options, links to photos, etc., in a project.

There are different passwords and logins for different sites to keep track of as well. I don't like having to rely on various different browser systems for keeping track of bookmarks and credentials, when after all, that's all just database stuff. I like the reliability of having it in a project, I like being able to sort or select by characteristics of interest. That's useful for things like using projects as personal inventory managers.

Even a small business can end up with very many gadgets and other assets that all have their own IP addresses, URLs, passwords, and so on, like routers, web cams, different computers. Keep track of all that plus how the gadgets are configured, like what processors, memory, motherboard, and so on, in a Manifold project.

I also like being able to grab text from a website and to save it, in case that site disappears. When you consider that an individual cell in a Manifold table can store 2 GB of text you can actually paste an entire book into a cell. I think as Manifold adds additional, rich data types (imagine saving an PDF into a cell, or an image, or an ebook, or a video...) such uses will only expand. Nothing I will do using Manifold as a personal information manager is going to slow it down in any way, so I don't even have to think about saving something in case I might need it in the future.

2. Scratchpad - Right now I have five instances of Manifold running. I often have an instance or two running that I use as scratchpads for text. Whatever I'm writing, I like the ease of copy/pasting (right click and copy to get the whole thing...) into cells in Manifold tables. It's way easier and more sensible to organize, review, update, delete, and manage saved snippets than using the wretched "Quick Parts" system in Outlook or similar things in Word. It's easier for me to just use what I know in Manifold than to have to learn and remember different ways of doing the same thing in each of 20 different applications I might use.

3. CAD or Illustrations - When I need to draw something, I'll use Manifold. I'll sketch up drawings for workmen on various projects, I'll plan renovations in gardens or structures, or I'll help friends with plans for houses and such. Use spherical Mercator for your projection and you can do a surprisingly large amount of CAD, checking dimensions and such. For a recent project where a friend is building a vacation home I overlaid plans of his existing home on various ideas for the vacation home so he could get a sense how big the various rooms will be in comparison to spaces he already knows very well. We could move around and rotate the plans for existing rooms to match up similar spaces so he could get a feel for how big a corridor might be, what it might feel like to move around the place.

For raster editing, I prefer Photoshop, but for simple vector illustrations I find Manifold easier than, say, Illustrator or Inkscape. A friend needed a "do not park here" placard for his garage, with a picture of a tow truck and a phrase, so I did that as a layout and sent him a PDF. There are million tools you can use for such stuff, but once you get Manifold commands into your muscle memory it's very easy to cobble up such a layout.

4. Scraping from the web - Manifold is a great tool for cleaning up stuff you scrape from the web. It's often the case that interesting info is published in tabular form on web pages, but to just copy it you get a whole lot of tags and other junk you don't really need, but which skilled use of Manifold transforms in tables can not just clean up, but use as a way of extracting data into more useful tabular form. It's also convenient to scrape data of interest from things like Leaflet sites. People publish data on Leaflet sites all the time without realizing that when they do that they've published the original source data as well, which is trivial to get. But then you have to manipulate it (easy with transforms) to clean it and get it into sensible tabular form with clean lat/lon fields.

mdsumner


4,260 post(s)
#03-Aug-20 00:08

I have a very similar list of regular uses, been using Manifold like this since 2002


https://github.com/mdsumner

lionel

995 post(s)
#21-Aug-20 09:50

Many small business use excel for bill since excel easely support create code in a gui way and let us create custom Form and custom view.

I find manifold less suitable than excel but if M9 is suitable for you ...no matter

The first time i see this video the first question was which tool was use to project 3D to 3 plan area ( 2 here) ?

the image appear inside manifold but perhaps is only an import from third tool ? ( can i have the name ! tool or/and guy ? )

Does Manifold 9 already support orthogonal projection of 3D design ( branch implemented not available)

regard's

single image is equal to milions of words

Attachments:
gis_orthogonal_projection.png


Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

lionel

995 post(s)
#21-Aug-20 10:28

when we transfer data from Web to excel / word or excel to word there is many sub menu users choices to define how the converter should work ( with or without style , only raster ....).

I can copy a selection in excel and paste in word.... It is a different world in manifold since manifold is not aware that there is data in memory buffer that can be use ( Memory Buffer share Active X Local "OLE DCOM" ) . I shure there was a manifold version where i can copy paste and manifold ask if it should resize the already create image ! Paste is not available in manifold after copy image that is inside manifold project !! Here capture screen show that manifold is aware of object only inside manifold context not Windows OS context ( the is no <buffer> choice only <new table> ......) . I use in my daily work a software and copy paste image work ( paint -> software) but the final image has not the same viewable rendering !!

I have to test a copy of georegister image in Qgis to see if during paste in manifold all data and metadata ( projection, ...) stay/is preserve in manifold image !!!

I think things work in Microsoft because microsoft implement both excel and word ... it is like Adobe Photoshop and illustrator copy and paste is manage between Suite ( adobe or microsoft )

Attachments:
image_from_memory.png
tableEcxel_to_Word.png


Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

adamw


10,447 post(s)
#22-Aug-20 16:48

This is an interesting thread. :-)

I used 9 to do the following non-GIS tasks:

* find duplicate images in a huge set of folders with photos (import all - sort by image size - in each group of images with the same size, subtract images from each other and check if the result is all zeros),

* explore voting patterns for elections (take various CSV and XLS files from government portals - massage the tables to make them useful: tons of things here, the first thing that hits you is hundreds of fields with multiple repeating groups serving the same purpose, sigh - create drawings with point plots using different measures for X and Y - aggregate data in plots by using Join dialog to transfer point counts / sums to a grid),

* test database connections / view or edit data in database backups (all the time),

* scan IMDB data for shows to watch, applying my personal ratings for what I already watched or tried to watch (IMDB data are available as TSV files, this project has just started and likely won't ever finish as ideally I would like to be able to capture updated data + re-run the queries + put out top N recommendations using a single command),

etc.

One other constant use of 9 is mass-converting files between various formats. We have command-line utilities to drive 9 that were designed to perform testing, they can also be used to perform conversions (with some knowledge of the internals applied). :-)

gpBike1001
48 post(s)
#22-Aug-20 17:18

Would some "How To" videos on some of these non-GIS applications of M9 be possible? I'd be really keen to see exactly how you structure the find duplicate image files problem/solution you mention.

adamw


10,447 post(s)
#23-Aug-20 09:58

I did that using a script, so I don't know if a video would be all that educational. But the thing that made me use 9 for removing duplicate photos in the first place was that file comparison utilities were not enough because duplicates had different tags (ie, some photos were rotated +/-90 degrees during viewing on the camera) and while I am sure there are specialized utilities that actually compare pixels, 9 was right there and had plenty of functions to do that already (make sure the two images are of the same size - INNER JOIN the tiles - aggregate TileValueMax(TileAbs(tile1 - tile2)), although back then we didn't have TileValueMax and it had to be written using a script as well).

But I get the idea, maybe we could have a video or two for uses not necessarily related to GIS. Why not. :-)

lionel

995 post(s)
#23-Aug-20 00:41

manifold is an ETL ( so data should be well format in database) and if you master programming and API ( for search tool like SQL , regexp ...) ..... shure you can do a lot ( master Visual studio help a lot ....)

I ask myself if Microsoft OS or Manifold or Excel support import file from not "file://" protocol but http ? ( TODO) .

the issue s not about where to find data but howto format them in database from html.if data is available in a database format then there is no work to format them ...import them in manifold and apply SQL And script (easy to say not easy to do when we don't use SQL and vbscript a lot) . I like the idea to grab some data "hidden" inside html . i wish to do this on rotten tomatoes who hide "rent/watch" text link to fandangonow , hulu , amazon , apple, netflix .... . I am more a webdesigner, re use adapt js/php script webmaster than a gis/script guy ...

The web is your database but some website add some javascript or don't show data embedding in html code to prevent you to grab their own data ( i like charles .. it is a software name !!) . You have to study all the html structure case to be able to grab all the data without missing information ( script can stop or insert data in the wrong cell ) !!!

We can control headless browser using script with selenium web driver and phantomjs ( javascript) or python or any API and language you confident with .

The video about uber use data is not from html but i really like how SQL is use to "extract" data information !! ( make me think of SQL inception :-)


Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

lionel

995 post(s)
#23-Aug-20 00:58

for compare files i like https://winmerge.org/?lang=en and the old https://www.whereisit-soft.com/

scraping from web page is a more difficult because of

-not wellformed html ( apply more on xml with dtd..xsd i think )

-html /browser version

- whatwg/W3C .


Book about Science , cosmological model , Interstellar travels

Boyle surface fr ,en

dchall8
1,008 post(s)
#23-Aug-20 23:12

Mass converting files??? That was my bread and butter when I was working. With help from the good folks out here in the Interwebs I patched together a command line script which runs a tif2pdf converter to take many tif images with the same file name and create one concatenated pdf file per file name/document. I've probably converted 300,000 images into 43,000 documents (averaging 7 pages each). Are you suggesting I could do some of that inside Manifold?? File names look like this

0201546.001

0201546.002

0201546.003

0201547.001

0201547.002

0201547.003

0201547.004

indicating the document name and various page numbers inside a folder. The result is

0201546.pdf (a 3-page document)

0201547.pdf (a 4-page document)

Oh, oh, oh! What about this? I get txt files which are space delimited tables of varying data types (numbers, text, dates, and times). The column spaces never change, but many of the rows are redundant(ish), so I've used a macro in Excel for years to read the text file, delete duplicate rows, and turn it into sortable/searchable information. That seems doable with SQL.

adamw


10,447 post(s)
#24-Aug-20 11:57

We don't currently have a dataport that would produce a PDF, but if we did, yes, you'd be able to mass-convert TIF files to PDF using a script.

Mass-importing / cleaning / re-exporting TXT files with space delimited values sounds doable right now.

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.