Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.169.1
adamw


8,696 post(s)
online
#21-May-19 17:49

9.0.169.1

Here is a new build.

manifold-9.0.169.1-x64.zip

SHA256: a6755f93e5e847b8e803fdc642ff22e4de05a30c0ad6c5830d3c8362bc2a4f76

manifold-viewer-9.0.169.1-x64.zip

SHA256: 66ff2f1934e110d333c8dad3b3b363cadd5380a2ae35bd65f0508ba752e79cce

adamw


8,696 post(s)
online
#21-May-19 17:49

Changes

The Layers pane allows grouping layers for maps, frames for layouts, and fields for tables.

Each layer group has a leading layer, layers following it can be grouped with it using the Group button in the toolbar. Grouping indents child layers one step to the right. Groups can be nested. Ungrouping is done using the Ungroup button in the toolbar. Selecting a leading layer in a group selects all layers in that group, including layers in nested groups. This then allows quickly turning on / off, etc, the entire group. Adding, removing, reordering layers auto-adjusts groups.

The Layers pane supports the following shortcuts: Ctrl-Right = Group, Ctrl-Left = Ungroup.

(In the current build, layer groups only appear in the Layers pane. Future builds will allow using layer groups from component windows as well.)

Code for ECW / JPEG2000 is moved into a separate DLL and is not loaded until it is needed.

Code for external Unicode collates is moved into a separate DLL and is not loaded until it is needed.

Data for external Unicode collates is loaded from a single shared location (~\shared\icudtl.dat) in both 32-bit and 64-bit modes.

Code for V8 is moved into a separate DLL and is not loaded until it is needed.

Viewer installs exclude the majority of code for V8 (which we do not need because Viewer disallows running scripts).

(The changes above decrease the size of installs and conserve resources at runtime. More importantly, they significantly decrease the startup time, usually cutting it in half or so, which is very noticeable.)

V8 scripts run faster, V8 objects exposed by Manifold do less locking and use less memory. (We are working on the API documentation for the object model for V8. It is somewhat similar to the object model for .NET, but tailored specifically for use from Javascript.)

Installation packages register the ODBC driver without running code in Manifold binaries. (This helps solve the most popular problem with uninstalls - that of the 32-bit executable blocked due to a false positive in a third-party antivirus tool and thus not being able to run as part of the uninstall.)

The Help - About dialog shows the install state of the ODBC driver (32-bit or 64-bit, regular or cutting edge). Possible report types:

  • 'Not installed' - the ODBC driver is not installed
  • 'Installed' - the ODBC driver is installed into the current location from which the application has been launched
  • 'Installed with mismatching options' - the ODBC driver is installed into the current location, but the install is either partial or damaged or out-of-date, the driver has to be reinstalled to work properly
  • 'Installed in different location' - the ODBC driver is installed but into a different location (eg, portable install in a different folder),

The dialog for Viewer does not show the install state of the ODBC driver, because Viewer does not include the ODBC driver.

The Help - About dialog allows installing (repairing) or uninstalling the ODBC driver from within the dialog, as long as the user has administrative privileges. Command-line options to install or uninstall the ODBC driver are no longer supported.

GPGPU code requires CUDA devices of compute capability 3.0+ (Kepler or higher, cards with Kepler started appearing in 2012). All GPGPU modules have been adjusted to use Kepler features.

GPGPU code can use much more memory, allowing for bigger tiles / longer sequences of operations (not specific to the new requirement for Kepler or higher).

All types of Kriging - regular / median Polish / regression - compute model parameters on GPGPU, if available. (This is significantly faster than doing the same on CPU.)

The schema field dialog shows and allows specifying type restrictions for geometry fields (area / line / point) for databases that support it.

The schema field dialog hides bounds and Z option for geometry fields that are not native to the database (GEOMMFD and GEOMWKB, both stored as binary data opaque to the database).

The schema index dialog disallows creating spatial indexes on geometry fields that are not native to the database.

Linking ECW / JPEG2K / SID allow editing the coordinate system, style and other metadata for the linked images.

Exporting an image to PNG or WEBP produces color (with or without alpha), grayscale or palette data depending on style info.

Exporting a big image to ERDAS IMG or TIFF skips producing data for intermediate level 1 (produces intermediate levels starting from level 2) to save space.

Reading GML, GeoJSON, KML and similar data preserves all coordinates in multi-type geometry values. Geometry values declared as multi-type but with all parts having the same underlying type (all areas, all lines, or all points) preserve that type. Geometry values that mix parts of different types are converted to either areas (if there are no lines and the value is a mix of areas and point) or lines (all other cases).

End of list.

tjhb

8,883 post(s)
#21-May-19 23:46

Quick comments on the grouping features. You will obviously have many things already worked out, or implemented but not finalized, but just in case.

I notice we can have groups within groups. That is cool and will be useful for logical organization of data.

I like the idea of using a leading layer, rather than something analogous to a folder, which would have no contents, and no other function besides being a named container. A leading layer can have contents, or none. If it has no contents then it is exactly as good as a folder, a named container.

If a leading layer does have contents... Then my initial instinct is that I will tend to use a leading layer as an AOI drawing for the layers in its group. That prompts two thoughts. (a) It would sometimes be useful to have a command (maybe from the leading layer tab) to insert an area into a leading layer (if it is a drawing) that describes the combined Rect of all the layers in its group (and any subgroups). (b) It might be interesting to be able to use an area in a leading layer as a mask for the group, in effect to set a maximum Rect for all member layers. But that quickly gets complicated: what if there are multiple areas in the leading layer, or the area(s) are irregular or multi-part (allow irregular masking? would be that be too slow?); would there be any strange effects, or major effects on speed, if member layers had different projections from the leading layer? I am unsure whether (b) or even (a) are worthwhile ideas at all.

Simpler: it would be good to be able to drag components into and out of a layer with the mouse, both in the Layers pane and in a component window. Also, to reorder layers, groups, and layers within groups, just by dragging, again in both places.

I imagine there will be right-click options on layer tabs in component windows to allow grouping and ungrouping; and for a leading layer, to hide/unhide its group; and ways to show the names of member layers (and member groups, and so on) without affected their visibility. It will be cool to see how the design for these things works out.

Layer groups are going to make dealing with large datasets (and composite datasets) much easier, especially for projects that evolve over time, and for showing to/sharing with different people including other users, non-users, and groups.

Mike Pelletier


1,629 post(s)
#22-May-19 00:40

The ability to drag and drop in/out of layers would be nice and more use of right click would be welcome. It will be interesting to see/hear how grouping will be brought to maturity.

For maps it would be nice have a group of vector layers stylized for use with an aerial and a group of vector layers stylized for use with a topographic background. Grouping would allow a quick switch between the two.

I noticed that layers linked in from ESRI's SDE cannot be stylized. Not sure if this is the case with other databases but I hope this limitation will go away.

Very happy to see redefining projections for linked ECW is now possible. Thanks!

adamw


8,696 post(s)
online
#22-May-19 07:48

I noticed that layers linked in from ESRI's SDE cannot be stylized.

Which database is it? SQL Server?

Mike Pelletier


1,629 post(s)
#22-May-19 15:29

SQL Server Express 2017.

adamw


8,696 post(s)
online
#22-May-19 17:12

We checked against Oracle and we'll check against SQL Server as well, but everything should work fine as long as you have permissions to write to MFD_META in the (database) schema of the SDE table that is being edited (and permissions to create new tables in that database schema, if it does not have MFD_META yet).

Long and short, try connecting to the database as the owner of the database schema that SDE is installed in, and seeing if that solves the issue.

Also, if there's anything reported about the failure in the log window, that could perhaps shed some light onto what's going on as well.

Mike Pelletier


1,629 post(s)
#22-May-19 19:29

Ah, so we have to connect with write permissions to stylize the layer. That is a definite issue since we may want to keep in read-only for a variety of reasons. I see that a work around is to copy the linked drawing, paste into the project, and rename. A hassle, but doable.

When 9 connects to an SDE database or other databases does it actually add a MFD_META table within the database and leave it there after it is disconnected? I'm leary of manifold getting blamed for causing trouble in Arcmap.

Mike Pelletier


1,629 post(s)
#22-May-19 20:42

Okay I see that connecting with write permissions indeed does add several tables to the database. Any remote chance this could negatively affecting the performance in Arcmap?

Can you explain why SDE in mfd 9 is displayed within 4 different folders: dbo.DEFAULT (contains the default drawings of the tables with geometry), System Data (mfd tables that are added), System Data (SDE) (tables without geometry), and tables within the main folder (seemingly many repetitive tables).

If I want to store within the database my stylized version of a drawing what folder makes most sense to put it or does it matter?

It would be nice if right clicking on a table from a linked database and then choosing "create drawing" that the resulting dialogue gave the option of where to store the new drawing instead of being limited to the folder within the right click happened. Similarly, when doing the same outside a folder, it would be nice if we had the option of specifying an existing table to use. Perhaps there are good reasons that is not an option?

Mike Pelletier


1,629 post(s)
#22-May-19 23:56

Also wondering if there is a way to link to a single table in a database rather than the whole database. Not a big deal but linking to a large database comes with some time lag. I see that linking the database, then creating a drawing from one of the tables in the linked database, and then deleting the linked database breaks the connection.

adamw


8,696 post(s)
online
#23-May-19 08:38

There is no way to link a single table, you always link a data source and then reference a table inside that data source. If it takes too long to connect to the data source, we'd rather address that. We did several improvements in this area already, including for SQL Server, can absolutely do more.

Moreover, if you have a drawing referencing a table in a linked data source, when you open that drawing and the data source is not connected yet, the drawing asks the data source to connect, but it does not wait until the data source enumerates all tables / views / whatnot. The drawing only waits until the data source retrieves enough data to become operable and proceeds straight to the table it wants. The drawing will then render data from the table and the data source will finish its discovery in background.

adamw


8,696 post(s)
online
#23-May-19 08:26

When you change components in some way, eg, format them using the Style pane, we have to save what you did somewhere in the database so that we can remember it next time you connect. We save such data in MFD_META tables. We only create an MFD_META table when we need to save some data - we do not create anything in anticipation of future changes, we consider that to be too intrusive. MFD_META tables are per-database schema. When we connect to a database, we determine which database schemas we can access and present a virtual MFD_META table which can be used to access all data from the individual MFD_META tables stored in the database.

I'll check what other tables we create, but MFD_META is the main one, most other MFD_xxx tables are virtual ones, synthesized from database data dynamically. There should be no issues whatsoever from having an additional MFD_META table in the database with the performance in Arc products -- this is just one more extra table opaque to Arc code, no triggers, no relation to any of their data, no nothing. Every database with SDE installed necessarily contains tens of tables that have nothing to do with Arc data, they are safely ignored as they should be, that's why SDE contains its own system tables listing what other tables contain SDE data, everything not listed is not SDE and ignored.

On folders - System Data with MFD_xxx contains virtual tables (by default, you can move tables in and out of folders freely, so that's just a starting point), System Data (SDE) contains tables used by SDE to navigate the database, dbo.DEFAULT likely lists all SDE layers in what SDE calls 'DEFAULT', but I'll check for sure.

We might extend the New Drawing dialog to allow creating a drawing in a data source different from that of the table, yes. Maybe it would be better to go the other way around though - start creating a new drawing in the data source that you want it in (so, the MAP file) and then select the table from a child data source. I realize that right-clicking a table before the dialog is faster than selecting it in a list, but not changing the target data source during the dialog has big pros - if the dialog allows changing the target data source, we have to protect from trying to create components in read-only data sources, have to adjust specified data (eg, if you started creating a table in data source X which supports RTREE indexes, specified such an index, then switch to data source Y which does not support these indexes, we have to delete the index you specified, perhaps warning you), best to avoid all that.

That said, why don't you create a drawing on the database? Does it not work for some reason? Maybe you want to understand what gets written to the database during this first - it's just some records in MFD_META.

The folder of the created component is irrelevant - that's just one more property that is there for convenience in the UI. You can move components freely between folders and this won't change how they work.

Mike Pelletier


1,629 post(s)
#23-May-19 16:46

Thanks for all the great info in your posts. Very helpful stuff. Consider adding some of this to the documentation.

As for why storing a newly created drawing in a local project versus the database, I think it just boils down to whether you want others to access it and/or you think you'll want to access it again from another project. Perhaps another factor is how easy it is to take the .map project mobile and then reconnect it later.

So if there many folks linking to a table in a remote database, is there a separate MFD_META table created in the remote database for each user?

There are likely good reasons for this, but if the linked drawing is stored in the .map project, why not store the required MFD_META info within the .map project's MFD_META table?

adamw


8,696 post(s)
online
#24-May-19 08:15

How many different MFD_META tables there will be in a database depends on how the database is organized.

If there are multiple users with each user having its own database schema with the intent being that each user can only see / change their own data and nobody else's, then each user will have their own MFD_META.

If in addition to that, there is a shared database schema which all users can access in addition to their own private schema, then there will be an additional MFD_META stored in that schema as well. When a specific user connects to the database, we will combine data from MFD_META in the shared schema and data from MFD_META in the user's private schema, and the user will be able to see / work with both tables and Manifold components created on top of these tables in both schemas.

If there are multiple users but they all work in the same shared database schema, then there will be a single shared MFD_META and everyone will see / work with the same tables and Manifold components.

On why not store MFD_META info for the linked drawing stored in the MAP (meaning the .MAP file) in the same MAP - that's what we do. If you have a table on SQL Server, then create a drawing based on that table and put that drawing into a MAP, then when you format that drawing, the style info gets written into the MAP. However, some info is attached to the table and not to the drawing - like coordinate system info. That info is stored in the MFD_META of the data source that contains the table.

adamw


8,696 post(s)
online
#23-May-19 10:11

A follow-up:

We don't indeed put any other service tables onto the database besides MFD_META, we store everything as properties. Manifold 8 Enterprise storages store data in tables named MFD_ROOT, they are per-database schema as well, and 9 will read data from those tables, too, but it won't create new ones.

If you are worried about making accidental changes to the database, you can connect to the database using a limited account which can only write to MFD_META and cannot write anywhere else - this will be enough to browse through everything, style everything, adjust coordinate systems, even create scripts / drawings / etc, because none of that goes beyond MFD_META, it's all just properties. If you do this, someone has to create MFD_META first, because permissions to write to the table and permissions to create new tables are different (perhaps first give the user account full permissions, connect, create a new folder to create MFD_META, disconnect, limit permissions to 'can read everything, but can only write in MFD_META', reconnect).

xxx.DEFAULT folder for SDE data contains drawings for the default version. SDE data is versioned, versions have names and are organized into a hierarchy. We expose the entire hierarchy by showing a separate drawing / table for each version of each SDE layer, and organizing all that into a folder tree.

Eg, if you have an SDE layer named CITIES with versions DEFAULT, VER_1 (based on DEFAULT), VER_1_1 (based on VER_1), and VER_2 (based on DEFAULT) we will show this, assuming the schema is named DBO:

  • a folder DBO.DEFAULT, with table DBO.CITIES.DEFAULT and a drawing for it,
  • a folder DBO.DEFAULT\DBO.VER_1, with table DBO.CITIES.VER_1 and a drawing for it,
  • a folder DBO.DEFAULT\DBO.VER_1\DBO.VER_1_1, with table DBO.CITIES.VER_1_1 and a drawing for it,
  • a folder DBO.DEFAULT\DBO.VER_2, with table DBO.CITIES.VER_2 and a drawing for it,

and a system folder System Data (SDE) with table DBO.CITIES (the base table) and various tables named DBO.Axxx / DBO.Dxxx / etc, which store data specific to versions.

You can open DBO.CITIES.<version> and view / edit data for the specific version. You can also open DBO.CITIES, the base table, and view / edit data in there, but in this case you have to know what you are doing because you are working with raw data organized according to the rules of SDE and making changes that break these rules can render the layer unworkable, same for working with DBO.Axxx / DBO.Dxxx / etc, -- that's why we hide all these tables in System Data (SDE).

Hope this helps.

adamw


8,696 post(s)
online
#22-May-19 07:43

(a) It would sometimes be useful to have a command (maybe from the leading layer tab) to insert an area into a leading layer (if it is a drawing) that describes the combined Rect of all the layers in its group (and any subgroups). (b) It might be interesting to be able to use an area in a leading layer as a mask for the group, in effect to set a maximum Rect for all member layers.

Maybe we could do this:

For (a), a command to zoom to the extent of layer + all child layers. And maybe a command to add a rect covering the current window area - or, perhaps better, a tool to insert a rect and a snapping mode that would snap to screen edges.

Why do you want (b)? If it is just a visual effect, that's one thing (can be done and seems not terribly difficult especially if the clipping area is just a rect aligned to the XY axes). But if that's a real clip, as in, you want to delete objects outside of the rect / reduce objects with parts outside of the rect, that looks more like a transform - which, yes, can take the rect to clip with from the active map window.

tjhb

8,883 post(s)
#22-May-19 09:59

I like your (a) much better than mine. It would be a multi-purpose tool, new uses all the time. For your second sentence ("And maybe...") I personally like its first part better (before "or, perhaps..."). Adding a rect for the current window area seems BTW somewhat analogous to creating a Location--and equally simple which is good.

Re (b), I was thinking of a visual effect, yes. Like setting a clipping mask or (for uppermost leading layer) an artboard in Illustrator. Non-destructive, can be repeatedly adjusted (for free besides re-rendering), easy to repurpose (same data, different map, different rect).

Dimitri


5,552 post(s)
#22-May-19 08:27

Agree 100%.

I find myself already using groups to deal with tables that have endless attributes not of interest. For example, I like the Natural Earth free data but man, oh man, do they have so many attributes. The table illustration in the Contents - Layers topic is a good example, where you might not really need the name of a country in Arabic and Chinese if you are working in English.

So, already when I work with attribute tables that contain way more attributes than I want to see, I group them and turn off those attributes not of interest with a click (well... double-click...).

By the way, that table illustration in the topic is a good example of how the work done in recent builds in Unicode and collations has paid off. It's really nice to see the different languages, like Arabic or Greek, used within Natural Earth appear as they should in a table. Import that shapefile into a different system, even one as shapefile-expert as QGIS, and you get the usual gobbledygook symbol salad.

Here's the table, to save folks from finding it in the topic...

Just for comparison's sake, here's the same table in QGIS 3.6.3 ...

Attachments:
natural_earth_table.png
qgis_table.png

rk
335 post(s)
#22-May-19 10:39

Surprisingly QGIS 3.6.3 does not care that there exists sidecar *.cpg file with UTF-8 inside.

One can repair (assign) the encoding easily enough under Properties...

Attachments:
qgis_utf-8.PNG

Dimitri


5,552 post(s)
#22-May-19 12:28

I'm also surprised it doesn't read the sidecar file. It's an odd oversight, given that Q is generally great at shapefiles. I suppose that's a result of shapefiles being the native format for Q for so many years, until the recent switch to GPKG. I don't know if Q now has a native shapefile importer or if it uses GDAL/OGR, which does not read the .cpg. (Can be confirmed by using GDAL to import the shapefile into Manifold... which results in symbol salad in the table).

But I wouldn't say assigning the right encoding is done "easily enough" for most people. Granted, as with any big application you have to RTFM to learn to work it correctly, but picking the right encoding from a very long list of possibilities is tedious and error-prone (few will know to look inside the .cpg). That's how encodings get set wrongly, and then such errors propagate when the data is saved into different formats, different DBMS's and so forth. And you need more than encodings, that is, you need to handle collations as well.

For those unfamiliar with Q, below is a shot of a small part of the list. The right choice is UTF-8.

Attachments:
qgis_data_source_encoding.png

Maps4planners12 post(s)
#23-May-19 16:35

Is there a limit on the Union Areas transform (I'm using Manifold 64 bit)

I imported 430,000 points and created a 200 metre buffer from each. I wanted to remove the overlaps so I used the Union Areas transform on the buffers. However, It just wouldn't complete. In fact it didn't look like it was starting. I had to cancel it (which didn't work) and then force the program to close in Task Manager.

In terms of computing power, I should be fine. I'm running a Threadripper with lots of fast RAM and an SSD

Any ideas - or am I approaching it incorrectly?!

adamw


8,696 post(s)
online
#24-May-19 08:23

The Union Areas and several other important geometry transforms have limits that carry over from 8, yes. The transform will probably complete, just slowly.

We are planning to significantly improve the code behind these transforms to increase limits / performance.

In the meantime, consider joining areas in parts - first 150,000 buffers, then second 150,000 buffers, then the rest, then join three resulting areas, something like that. Or try waiting for the transform on full data to complete, eg, leaving it working overnight -- because it might complete (although that's not guaranteed, it might run into a limit late). We realize this is annoying, hence plans for reworks.

Maps4planners12 post(s)
#30-May-19 12:51

Thanks - that makes sense. In the past using some much older GIS programs I've had success batching the unions like that and it processes quite quickly. I look forward to the reworked versions!!

KlausDE

6,352 post(s)
#26-May-19 09:02

German ui file for version 9.0.169.1

Attachments:
ui.de.txt

Dimitri


5,552 post(s)
#27-May-19 18:13

Thank you, Klaus!

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.