Subscribe to this thread
Home - General / All posts - Using Manifold for non-GIS tasks
Forest
617 post(s)
#30-Jul-20 02:37

I am currently doing some computer vision work and am using Manifold for non-GIS work. I just want to say that it works extremely well. I was using it yesterday to read json files and Manifold pulls them in and converts them to useful tabular format better than any of the other tools that I use. Manifold also has always been in my toolkit for cleaning up messy attribute tables. It is more heavy duty than ms-access and has a lot of additional functionality. Just a shout-out to the team. Thanks for your good work!

dale

558 post(s)
#31-Jul-20 10:31

Hey Forest,

You able to give some examples? Sounds interesting.

dchall8
760 post(s)
#02-Aug-20 18:28

Using the appraisal district maps I found literally thousands of miscoded fields in the data which resulted in significant errors in school district funding as well as emergency services. While the maps illustrated the errors, correcting them was entirely done in the data tables. I copied the data to Excel or to paper (sigh) so the clerks could make the changes.

Forest
617 post(s)
#03-Aug-20 05:03

Hi Dale,

This what I am doing.

I am looking into computer vision for plant species identification. I have a high spec’d Alienware laptop and have installed everything that might be useful, and I know how to use a wide variety of applications. Manifold tends to fill the cracks or help me investigate datasets that I cannot view well in other environments.

Computer vision uses convolution as the first step to image identification. Manifold calls these kernel operations. With Manifold, I can play around with kernel operations and see which features they are picking up.

The actual issue that prompted the post was a competition on Kaggle for using machine vision to guess the genus of an unknown species. Here is a link to the repo. https://www.kaggle.com/c/inaturalist-challenge-at-fgvc-2017/data

The data for the competition is accessed via json files which have the URL for the sample input photos. So, I reached for NotePad++ and opened up the big json file. It took a long time and then I found that the json file had been linearized, which means that all the line breaks had been taken out and all the data was on one very long line. No problem, I would just add the json formatting tool to NotePad++, but the security settings will not let me install plug-ins. I was not happy but had to find an alternative. I tried Manifold and it very quickly imported the data and put the data into tabular format which was very useful.

There is not much GIS in what I am doing, however there is a great deal of relational database work. I developed my own version of the Altas of Living Australia (before they appeared) in ms-Access. Now I want to be able to leverage those older systems but want to use Postgres as a backend as it scales better and works seamlessly with a lot of open source technologies which I want to use. Using Manifold, I can import tables from both Ms-Access and from Postgres and write queries that use data from both backends. I cannot do this in pgAdmin or in dBeaver, which is the other database management system that I use. In Manifold, I can also drag data from gpkg files into postgress and I am hoping that I can drag non-spatial tables from ms-access into postgress too. I have not tested all the tasks in front of me.

For every task, there should be at least two software applications that can do the task. I work hard to eliminate potential single points of failure. Manifold is often the Swiss Army knife that covers the provision of a second means of doing key tasks. And as mentioned, it is often the primary tool for importing data and exploring it.

tonyw
595 post(s)
#31-Jul-20 16:55

I just want to say that it works extremely well.

I've been thinking similarly, that Manifold's ability to handle extremely large tables very rapidly along with SQL, relations, transformation templates, and filters would be great for handling and analyzing large databases. I don't have such a project yet so currently I'm in the state of having a solution looking for a problem to solve.

hugh
177 post(s)
#31-Jul-20 23:23

where I use Manifold frequently like this is working with the tables in US Census file geodatabases like the ones here. This is one for Florida 1.58 GB in image open and ready for table work:

Attachments:
gdb.jpg

Dimitri

6,104 post(s)
online
#02-Aug-20 10:22

I also use Manifold a lot for non-GIS tasks. Some examples:

1. Personal information manager - Anytime I need to keep track of something I keep it as a table in Manifold. It's easy even for just a handful of notes.

Suppose I'm planning a trip somewhere... there's a lot of information like reservations, addresses, URLs to hotels, local restaurants, and so on. It's nice to have all that at my fingertips in a project. Manifold .map format is so durable that I like having it in a format that is reliable. This is also convenient because many times what I'm keeping track of has some sort of location element to it, mixed in with many records that aren't spatial.

Sometimes there's nothing spatial about it at all. For example, I do research on various topics of interest, both personal and business, and such research often involves fairly extensive sets of online and offline resources. Need to buy a new dishwasher and researching vendor and product options? Keep track of all the URLs, links to reviews, prices and options, links to photos, etc., in a project.

There are different passwords and logins for different sites to keep track of as well. I don't like having to rely on various different browser systems for keeping track of bookmarks and credentials, when after all, that's all just database stuff. I like the reliability of having it in a project, I like being able to sort or select by characteristics of interest. That's useful for things like using projects as personal inventory managers.

Even a small business can end up with very many gadgets and other assets that all have their own IP addresses, URLs, passwords, and so on, like routers, web cams, different computers. Keep track of all that plus how the gadgets are configured, like what processors, memory, motherboard, and so on, in a Manifold project.

I also like being able to grab text from a website and to save it, in case that site disappears. When you consider that an individual cell in a Manifold table can store 2 GB of text you can actually paste an entire book into a cell. I think as Manifold adds additional, rich data types (imagine saving an PDF into a cell, or an image, or an ebook, or a video...) such uses will only expand. Nothing I will do using Manifold as a personal information manager is going to slow it down in any way, so I don't even have to think about saving something in case I might need it in the future.

2. Scratchpad - Right now I have five instances of Manifold running. I often have an instance or two running that I use as scratchpads for text. Whatever I'm writing, I like the ease of copy/pasting (right click and copy to get the whole thing...) into cells in Manifold tables. It's way easier and more sensible to organize, review, update, delete, and manage saved snippets than using the wretched "Quick Parts" system in Outlook or similar things in Word. It's easier for me to just use what I know in Manifold than to have to learn and remember different ways of doing the same thing in each of 20 different applications I might use.

3. CAD or Illustrations - When I need to draw something, I'll use Manifold. I'll sketch up drawings for workmen on various projects, I'll plan renovations in gardens or structures, or I'll help friends with plans for houses and such. Use spherical Mercator for your projection and you can do a surprisingly large amount of CAD, checking dimensions and such. For a recent project where a friend is building a vacation home I overlaid plans of his existing home on various ideas for the vacation home so he could get a sense how big the various rooms will be in comparison to spaces he already knows very well. We could move around and rotate the plans for existing rooms to match up similar spaces so he could get a feel for how big a corridor might be, what it might feel like to move around the place.

For raster editing, I prefer Photoshop, but for simple vector illustrations I find Manifold easier than, say, Illustrator or Inkscape. A friend needed a "do not park here" placard for his garage, with a picture of a tow truck and a phrase, so I did that as a layout and sent him a PDF. There are million tools you can use for such stuff, but once you get Manifold commands into your muscle memory it's very easy to cobble up such a layout.

4. Scraping from the web - Manifold is a great tool for cleaning up stuff you scrape from the web. It's often the case that interesting info is published in tabular form on web pages, but to just copy it you get a whole lot of tags and other junk you don't really need, but which skilled use of Manifold transforms in tables can not just clean up, but use as a way of extracting data into more useful tabular form. It's also convenient to scrape data of interest from things like Leaflet sites. People publish data on Leaflet sites all the time without realizing that when they do that they've published the original source data as well, which is trivial to get. But then you have to manipulate it (easy with transforms) to clean it and get it into sensible tabular form with clean lat/lon fields.

mdsumner


4,218 post(s)
#03-Aug-20 00:08

I have a very similar list of regular uses, been using Manifold like this since 2002


https://github.com/mdsumner

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.