Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.171.1
adamw


8,971 post(s)
#24-Mar-20 17:33

9.0.171.1

Here is a new build.

manifold-9.0.171.1-x64.zip

SHA256: cd537e288437d01c7293ff4d66c55f1921b253d3945ff10da5b6afdba7904901

manifold-viewer-9.0.171.1-x64.zip

SHA256: a44c0aefdf7672dda8d2ab14ba5475cf62f631daa7839a014754ab11b7d64068

adamw


8,971 post(s)
#24-Mar-20 17:43

Reprojecting Images

The Reproject Component dialog for images includes an option to specify reprojection method. Available methods are:

  • 'standard (fast)' - the traditional reprojection method used in 8 and used in 9 for display, the default,
  • 'direct sub-pixel (more accurate)' - an improved method which takes longer, but produces a better quality image.

The 'direct sub-pixel' reprojection image allows specifying the number of pixel subdivisions. The default number of subdivisions is 1, this is quite enough for most cases. Increasing the number of subdivisions improves image quality (with diminishing returns) at the cost of processing time.

New query functions:

  • CoordConvertTileSet - takes a coordinate converter, an image in the source coordinate system, a rectangle in the target coordinate system and a number of pixel subdivisions, and returns a table with image tiles projected to the target coordinate system.
  • CoordConvertTileSetPar - a parallel variant of CoordConvertTileSet.

(We might rename these two functions in the near future and reuse those names to build similar functions for the standard reprojection method.)

GPGPU

The GPGPU interface is no longer tied to a particular version of CUDA and can use CUDA devices starting with compute capability 2.0 (Fermi).

The GPGPU modules are loaded from the GPGPU.DAT file included into the installs of 64-bit versions of Manifold and Manifold Viewer. The file contains multiple versions of each module, the GPGPU interface automatically loads modules of the latest version that can be used on the CUDA device present on the system. If the system has multiple CUDA devices, the GPGPU interface loads modules of the latest version that can be used on all devices, to enforce consistency and avoid recompilations. Currently, GPGPU.DAT includes modules for compute capability 2.0 (Fermi), 3.0 (Kepler) and 5.0 (Maxwell). Performance differences between versions are mostly small, but positive, with later versions performing very slightly better than earlier ones.

Join

There is a new Join dialog to transfer data between tables, invoked via the Edit - Join command for a table window.

The current component is shown on the left and cannot be changed. The component to join data from is shown on the right. It can be a table or a query in the data source of the current component.

The line below components specifies the join criteria. The only join method available for tables is 'equal', with a field chosen from each table.

The dialog tries to guess the field to use as a key for both tables. For the target table, the dialog tries to use a field with a BTREE / BTREENULL index (but not MFD_ID), with a type preference (numbers > text > everything else) and a name preference ('..._id' = '... id' > '...id' > everything else). For the joined table, the dialog uses a similar logic, but first tries to use a field with the same name as in the target table.

The bulk of the dialog is a list of fields in the target table. Fields that participate in unique indexes (BTREE / BTREENULL, this also includes the implied system index on MFD_ID in MAP files), fields used as a join criteria and computed fields cannot be written to. Other fields can be specified to be populated from data in the joined table, showing the transfer method in the middle column and the source field in the last column. To set a field to be populated from the joined table, edit (double-click or focus and press Enter or F2) the empty space for either the transfer method or the source field. Editing the transfer method lists all transfer methods which can be used with the target field and with any of the joined fields. Editing the source field lists either all fields which can be used with the selected transfer method or all fields which can be used with any transfer method if there is no transfer method selected yet. Editing both the transfer method and the source field allows selecting '(none)', excluding the field from the join.

If the join criteria is such that for each record in the target table, there is at most one record in the joined table, the list of transfer methods is limited to 'copy' (used when the type of the target field exactly coincides with the type of the joined field) and 'convert' (used when the type of the target field is different from the type of the joined field, but there is a conversion available).

If the join criteria is such that for each record in the target table, there might be multiple records in the joined table, the list of transfer methods includes all transfer methods available for aggregated fields in the Transform pane (numeric aggregates, spatial aggregates, string aggregates, etc).

If the target table allows adding new fields, the dialog enables the Add button in the toolbar above the list. Clicking the Add button shows a drop-down menu of fields from the joined table. Selecting a field from the menu adds a new field to the list of fields in the dialog, sets the transfer method for the added field to 'copy' and the source field to the selected field. The name of the new field can be edited. The type of the new field is not shown but is set to coincide with that of the source field. New fields are highlighted using the bluish 'computed' color and have a distinctive icon on the record header. Unwanted new fields can be selected and deleted using the Delete button in the toolbar.

The list of fields can be filtered by name. The filter matches the name of both the target field and the name of the joined field, if it is specified.

To perform a join, click Join Component at the bottom of the dialog. Joining copies data over to the target table. Any changes to the data in the joined table made *after* the join will not automatically appear in the the target table. However, you can save the join operation using the 'Save update query' option - this will create a query which will repeat the join whenever this is convenient. You can have any number of saved joins. You can also customize what these joins do by editing their code.

The Edit - Join command is disabled for queries, readonly tables (tables that miss a unique index and so cannot be updated, or tables on readonly data sources) and system tables (MFD_META / MFD_ROOT / MFD_SRID and such).

(Note 1: You can join data from a table or query in a different data source, too. Create a query in the data source of the target table and set query text to 'TABLE [datasource]::[table]', citing the component on the foreign data source. This query works as an alias for the table on the foreign data source. In particular, it will be visible in the Join dialog for the target table.)

(Note 2: We are going to create a quick cheatsheet of how to use the dialog, invokable by F1. For now, we only have a placeholder page, but it is going to grow. We might create cheatsheets for other dialogs in the future as well.)

Other

The Stack Horizontal / Stack Vertical commands for a layout window support stacking frames of varying width / height (previously, the frames were stacked evenly using the width / height of the active frame).

The new Align Center Horizontal / Align Center Vertical commands for a layout window align layout frames to common center.

(Fix) Switching the cursor mode in a layout window no longer fails to update the cursor mode icon on the toolbar.

The various favorites dialogs require selecting new favorites to add them to the list of current favorites. Adding new favorites to the list of current favorites does not remove them from the dialog.

ECW support is updated to the latest version of ECW SDK, incorporating fixes to reading ECW and JPEG2K files.

WEBP support is updated, incorporating fixes to reading and writing WEBP files.

V8 engine is updated, incorporating improvements to Javascript compilation and execution, including reductions in the amount of used memory.

(Fix) Reading PBF no longer sometimes skips blocks of lines and areas (mostly happens on large files).

Reading PBF splits PBF data tags into multiple tables based on the tag name: tags starting with 'addr' / tags starting with 'building' / tags starting with 'source' / all other tags. (This change allows importing whole-world PBFs without running into the current limit of 2 billion records per table. After the limit is removed, we will read tags into the single table again.)

Reading PBF is optimized to perform significantly faster, particularly on huge files. (We want to allow importing whole-world files. We removed the roadblock with the tags above, removed two or three similar roadblocks elsewhere, and are now working on performance. Reading a whole-world PBF used to take multiple days - only to run into a limit on the number of records per table and fail. Now the process no longer runs into the limit, but it still takes about a full day to complete, because the amounts of intermediate data processed are huge - over a terabyte, with complex organization. We are going to improve the time it takes to handle all this. We are learning a lot from dealing with vector data sets of this size as well.)

(Fix) Attempting to export an XLS file no longer sometimes fails in the creation of the initial XLS file.

Reading KML recognizes more variants of geometry data (all types in the semi-official Google samples).

Reading KML recognizes more variants of URL links (NetworkLink, etc).

End of list.

hugh
173 post(s)
#24-Mar-20 21:39

Just tried new transfer data between tables join dialog-- works easier and clearer than table>relations in 8 and so much faster than update SQL workaround we have had to do for this in 9 -- thank you!

wvayens54 post(s)
#24-Mar-20 22:37

Very nice on the join implementation....this has been one of the stumbling blocks for me, moving from 8 to 9, even with all the advice I've received here on other ways to accomplish this....for me, this is great!

Mike Pelletier


1,698 post(s)
#24-Mar-20 23:21

Also very happy to have the new join dialogue. On thing that I will miss from 8 though is the ability to setup a relation between 2 tables and then as you fix data in one you can see instantly if the correction made a match. Very gratifying as your slogging through data, especially when the link allows corrections on a map. Any chance that kind of behavior is doable?

Also, like the new ECW support. It recognizes my coordinate system by default.

adamw


8,971 post(s)
#25-Mar-20 07:49

We understand where you are coming from and we do have something planned for dynamic matches as well.

As a brief recap, relations in 8 are dynamic (the system fetches data from the joined table on every read) whereas joins in 9 are semi-dynamic (the data is transferred once, and if you want to capture changes made after the transfer, you have to repeat the operation - or run a query generated automatically specifically for that purpose). The reason for the change is that the second approach is just better, it scales to much bigger data, allows to put much more interesting computations into the join, including custom ones, works for data stored in databases outside of MAP file, etc.

That said, we agree that while re-fetching data (by re-scanning a table) on every read is costly, etc, etc, sometimes you still want to do it in some limited way - like in your scenario with editing where you want to see if the value you just entered is correct. What we are planning to do is allow such dynamic matching happen in the Record pane, for the edited record. You will tell the system that when you edit a record in table X, you want to also see matching records from table Y. When a record from table X enters the Record pane, the system will find matching records from table Y and show them. As you edit the field values of the original record, the system will update the list of matching records.

There will be no new fields created and the matching will be limited to the edited record, so if you would like to use the data from other tables to, say, style the objects in the original table, you'll still have to perform a regular table-wide join, which would be semi-dynamic. But in return everything will work with tables of any size (given appropriate indexes), and the UI for the dynamic per-record matching will be able to be different from the UI for table-wide matching and be more useful. Eg, for a 1-N joins we will be able to show all N records instead of a single aggregate record, we will be able to match records from multiple tables, and so on.

Mike Pelletier


1,698 post(s)
#25-Mar-20 13:52

This sounds really excellent Adam, you guys are on top of it. Showing 1-N joins with a smooth UI will be a big jump forward.

The Record Pane is blank today when working with tables. Can that change such that as your working through edits in the table, the Record Pane updates automatically and shows the joined info? Not sure, but it sounded like what you were describing all happened in the Record Pane.

adamw


8,971 post(s)
#26-Mar-20 15:56

Yes, that's what I meant - we'll extend the Record pane to work for tables in addition to maps / layouts.

hugh
173 post(s)
#27-Mar-20 02:32

just as an aside, working with census data over many years I have used M8 relations constantly without ever not flattening the correctly matched fields after. After the work of finding and joining to a drawing table a few of the thousands of fields in a source table I would never want to leave them dynamically dangling.

apo98 post(s)
#25-Mar-20 09:16

The new join approach is really good. Yes it is static at this point but I really like the new command window. Great add for any quick join and I'll stick to SQL procedures I set for the more dynamic needs.

Dimitri


5,892 post(s)
online
#27-Mar-20 14:23

It's also very convenient (and flexible) to launch the update query the Join dialog can create. See examples in the new Edit - Join topic.

dchall8
688 post(s)
#28-Mar-20 17:29

Sorry I don't have my historical data to work with anymore, so I can't test this. It looks like you have to opt in, field by field, one at a time, from the joined table to the original table. I had 33 (and growing) fields in the data I used. Is there a way to bring in all the fields or Select All and then itemize those you don't want/need?

Dimitri


5,892 post(s)
online
#28-Mar-20 17:52

Yes, to use the dialog you add fields from the joined table to the original table using the Add button. It's one at a time to keep it simple, since that way you can specify what join method you want for each field. It goes very quickly.

I guess one could come up with a mass dialog to join 33 fields at a time. Maybe an option in the pull down list when adding a new field for (all). You could then select the ones you don't want and delete them.

adamw


8,971 post(s)
#03-Apr-20 08:04

A status update: we are planning to issue the next build early next week.

It is really nice that at least the IT infrastructure evolved to the point where teams can keep working and collaborating productively even in the face of physical lock-downs. Let's hope that the current pandemic situation is over soon.

Manifold User Community Use Agreement Copyright (C) 2007-2019 Manifold Software Limited. All rights reserved.