Subscribe to this thread
Home - Cutting Edge / All posts - Manifold System 9.0.163.11
adamw


7,307 post(s)
#30-Nov-17 18:22

9.0.163.11

manifold-future-9.0.163.11-x64.zip

SHA256: d51642649e948ff481906d9205afdfc405977a2a4a102f44b8bfd693f818e4cb

manifold-viewer-9.0.163.11-x64.zip

SHA256: a650a0aee5e5d51eeecd3ef55aa17630c1c0ae0edd4c817d3318d0b7e18fe833

adamw


7,307 post(s)
#30-Nov-17 18:22

Changes

(Fix) Tables in MAP file no longer fail to accept more than 2 billion variable-length values. Attempting to insert too many values was producing a crash. Tables created before the fix will no longer crash, but will continue to reject values above the limit (there is no way to accept values without re-creating the table). Tables created after the fix will not be subject to the limit.

(Fix) Working with metadata for system tables on remote databases automatically appends the default schema to the name of the table on both reads and writes.

(Fix) Dataports for remote databases no longer sometimes fail to create, rename or delete folders.

Web dataports report sizes for returned tiles that do not match the size advertised by the server. (This allows creating another data source with the actual tile size for the server that misreports it.)

There is a new MXB dataport for compressed MAP files. MXB files allow storing all types of data available for a MAP file can, but they do not store data that can be re-created dynamically (such as indexes or values for computed fields), and they compress all data they do store. MXB files serve two main purposes: they provide reliable backups which are resistant to changes in MAP file structure, and they provide means to exchange data in compact form.

The new File - Export Project command allows exporting the opened MAP file as an MXB file. Linking an MXB file automatically uncompresses it into a MAP file with the same name and then opens the produced MAP file. Making changes to the MAP file opened that way will not write changes back to the MXB. Opening a linked MXB file that has already been uncompressed into a MAP file will not overwrite the MAP file even if it has been changed after it was uncompressed.

Compressing and uncompressing an MXB file tracks progress and can be canceled.

Right-clicking a data source in the Project pane allows exporting it as either MAP or MXB.

The File - Open command allows opening either MAP or MXB.

End of list.

adamw


7,307 post(s)
#30-Nov-17 18:33

A short comparison of file sizes for an example vector data set:

  • Manifold 8 MAP, compression turned on (the default): 102.0 MB.
  • Future MAP, no compression + indexes: 430.3 MB.
  • Future MXB, compression: 103.5 MB.

Same for an example raster data set:

  • Manifold 8 MAP, compression turned on: 74.4 MB.
  • Future MAP, no compression + indexes: 213.2 MB.
  • Future MXB, compression: 47.3 MB.

In the latter case, MXB wins over Manifold 8 MAP mostly because MXB does not store pyramid data.

Dimitri


4,332 post(s)
#01-Dec-17 07:40

Important:

1. The new table structure introduced for .map projects in 11 cannot be read by builds prior to 11. That means .map projects created by 11 are not backwards compatible to prior builds of Radian, Future or Viewer.

2. Tables created in projects prior to 11 remain in "pre-11" form even when opened by 11 and have the limitations of pre-11 builds, namely a limitation of 2 billion variable length values. If you create a project in, say, .10, save it, and then open it in .11, all the tables in that project remain in .10 form. If you just modify one of the tables and then save the project, the table remains in pre-11 form because the table itself, edited though it may be, was still originally created before 11.

3. Any new tables created in the project by .11 or greater will be in 11 form. Given point 2 above, you could have a project with tables created in .10. Open the project in .11. Copy one of the tables and paste it. You've just created a new table in .11. That new table that you've pasted is in .11 form. All of the other tables, including the original you copied, remain in .10 form.

We strongly recommend that anybody working with 11 and newer take a moment to migrate their projects into 11 form.

The easiest way to do this is to open a project in 11 and then choose File - Export Project and export it to .map format. That creates an 11 .map in which all of the tables are in 11 form. Doing this avoids the messy situation of copy/pasting tables within a project where some can be in 11 form and others not.

If you prefer, you could also export the project to MXB format. When you open the MXB all of the tables also will automatically be in 11 form. Since MXB is new to 11, exporting projects to MXB is one way to ensure that users of those projects (for example, people who have downloaded Viewer) are forced to move to 11 or more recent Future and Future Viewer builds. We probably will change all of the projects made available as downloads on the Manifold web site into MXB form for just that reason.

We realize all the above is an inconvenience. But that's life in the fast lane with an open beta process. The modification to tables to go to 11 and further removes a limit that was in prior releases but very rarely encountered. It is the root cause of the crash reported by StanNWT (thank you Stan!).

That there is a limit to what can be stored in a table in a .map file is OK, since all products have some limits. That exceeding that limit caused a crash is a bug, plain and simple, since exceeding limits should pop open a routine message and not cause a crash.

I think in some ways it would have been perfectly acceptable to simply alter the product to prevent a crash and to pop open an error message when the limit was exceeded. Instead, we decided to do that but also in addition to remove the limit. Or, more accurately, to condense all such limits down to a single limit of 2 billion records per table within a .map file.

2 billion records at the present time is the limit for the number of records in any one table stored within a .map file. It's not a project limit: you could have 200 tables, each with up to 2 billion records, stored in a .map file.

It's also not a Manifold limit. If you have an Oracle or PostgreSQL database with 10 billion records Manifold is happy to work with them. If you do a transform that takes a table with 10 billion records in Oracle and you want to add 50 billion records resulting from that transform to another table in Oracle or in PostgreSQL or whatever, no problem. You can do that as well. It's just that for now the limit to the number of records in a single table stored within a .map file is 2 billion.

Eventually, that will be dialed up to some larger value but for the next few months or so 2 billion records per table seems to be enough. :-)

Dimitri


4,332 post(s)
#01-Dec-17 08:25

I have to add.... there is way more than meets the eye to this new File - Export to .map or .mxb stuff. For example, you can right click onto a data source and choose Export and then .map... this creates a .map project from the data source. If you just want to save it in more compact, Manifold form, export it to .mxb.

Suppose you have an Access .mdb file with a few hundred tables, queries, etc, in it. Launch 32-bit Future and create a new data source called, say, "MyMDB".

Right click onto MyMDB in the project pane and choose Export, and then export to .map.

Launch Future in 64 bit mode and open that .map... you now have what used to be an .mdb as a Manifold Future project!

I gave the example above using an .mdb, but you could just as easily create a data source from an ESRI .gdb, GPKG, PostgreSQL data source, etc., and automatically save it as a .map. It's an easy way of converting lots of data within complicated arrangements into a .map project.

There may be one or two bugs to be worked out depending on the data source. GDB and GPKG sometimes doesn't export right depending on the types of indexes used in tables, etc. We'll get those sorted out quickly as such things can indicate something overlooked in the dataport, so working those issues is a very profitable process. But the key idea of being able to right click on a data source and, like magic, create a .map out of it is definitely a good thing.

artlembo


2,916 post(s)
online
#01-Dec-17 21:51

this is a great idea. Currently, I get NRCS soil databases (with the dozens of tables), copy all of them, and then paste them inside of MF. Now, we can simply export them.

I will have to look at the object model, but I wonder if we can use VBScript to do the same, and never have to even open up MF - just cycle through 48 .map files.

adamw


7,307 post(s)
#02-Dec-17 09:35

Yes, you can export a database using a script.

This exports the current MAP file:

'VBScript

Sub Main

  Set app = Manifold.Application

  Set db = app.GetDatabaseRoot

  db.ExportFile "ignored""c:\\data\\test.mxb"

  app.Log "Exported to MXB"

  app.OpenLog

End Sub

The component name passed to ExportFile is ignored, because MXB exports a whole database, same for MAP. We will allow passing an empty string to support exporting individual components should we need that in the future.

lionel
129 post(s)
#03-Dec-17 11:50

when we import the mxb , MF create a map file in the same locatin of the mxb file .

What is strange is that when i slide a mpa file on MF the project file is open and compnent appear inside MF GUI .

If i do the same for mxb nothing occur !!

2) slide V8 map project occur silently without alert of format change structure ! the question is ask when we quit the project ( here without change anyhting in the prpoject) .

3) slide the map on manifold gui ( map create by manifold when slide the mxb) is imported and work like any map create by MF .

4) i really like that in MF the import let us choose by extension file compare to manifold 8 where we choose by functionnalities structure. So now we don't have sub menu when do file import ( drawing , table , raster , ) . .

5) in microsoft when hide the application the icon is a one red fill triangle with a blavk m and manifold 8 is the same but with a yellow triangle at the bottom behind the red triangle

6) i test import exceltable.xlsx ( File -> Import ) and paste the path long url with file in the end

in manifold 8 ( select excel extension) show me GUI windows Import table title to let me select source and column source

in manifold 8 ( select access extension) Can't etablish connect to data source error binding data

source

in MF ( select nothing by defaut default= "All file") nothing occur without a message why the file is not import. Do nothing and alert nothing

in MF ( after s e le c t find excel item in the L O N G Li.s..t ) nothing occur without a message why the file is not import. Do nothing and alert nothing .. Strange ther eis a bug !! click on exce elfile show me a pop up that file can be open only in read mode because the sessin for import this file in manifold 8 is not ended !! after finish this session and try to import the same file nothing occur . What is new is that Microsoft OS show me a new option for import the file : "Show previous version "

So my excel file import don't work in MF .....until now . The file contain 1 value in a cell

if seem history on excel file is not implemented by MICROSOFT ? or only available for

professionnel

7) best ll be that manifold detect the extensionS of al lthe items selected to let us import in a batch mode all the selections for us !! or that slide one file inside manifold project behave like File import or link using a windows pop up that ask how we want manifold behave/compute the data source ( url , file ) .

( NB for import more than 4 image save and re edit the post )

Attachments:
M11_import_excelhistory_notimplemented.png
M11_import_externalData.png
M11_import_showHISTORYversion.png
MF_11_icon_simpliest.png
MF_11_import.png

lionel
129 post(s)
#03-Dec-17 12:09

stange that MS OS show us the request like Webserver to open a loca lfile using desktop application Attributes:0013=32&UNC=%5C%5Clocalhost%5CC$%5CUsers%5Clionel%5CProgramFiles%5Cmanifold-future-9.0.163.11-x64%5Cbin64%5Cexceltable.xlsx&IsLocal:000b=-1&Name=exceltable.xlsx&

Does Manifold Future is a uwp application so we can automate ( tedst purpose) using Selenium Appium with microsoft driver ?

adamw


7,307 post(s)
#04-Dec-17 06:52

Replying to both posts.

We will allow opening an MXB via drag-and-drop from Explorer.

The icons for Manifold 8 and Radian / Future are different, yes.

If an Excel file fails to import into Radian / Future, check the log window, it should contain the error messages. We don't pop up error messages in a dialog because there might be many errors and all of them are recoverable (the import just moves on to the next file). The most common reason for errors with importing Excel files is lack of 64-bit Access Database Engine.

We don't have an UWP application. Currently, we are not planning to have one, although that might change.

artlembo


2,916 post(s)
online
#11-Dec-17 02:42

BTW, I took all the SSURGO soils in New York State, which was originally a 20GB file, and the mxb is now 3GB - almost 7x!

And, the 6GB NYC taxi file is now 723MB. - almost 10x

These sizes are now easy to transport on a usb drive.

Two questions:

1. Once the file is opened and uncompressed, is there any limitations to it? BTW, it opened very fast.

2. I've been working off of an external USB 4TB drive (Western Digital). It is still rather fast. For convenience, do you have any thoughts about the performance on a SanDisk 64GB thumb drive?

tjhb

7,545 post(s)
#11-Dec-17 04:42

Question 2 is not clear Art.

artlembo


2,916 post(s)
online
#11-Dec-17 04:48

Sorry. Having Viewer, portable postgres, and a thumb drive is so enticing :-)

You can take it anywhere and get work done.

I was just wondering if people ran into trouble doing that.

adamw


7,307 post(s)
#11-Dec-17 10:03

Uncompressing an MXB file creates a MAP file. The created MAP file is a regular MAP file and does not have any additional limitations imposed just because it was created by uncompressing an MXB.

USB drives work pretty well, especially in modern operating systems which contain optimizations specific for the device type (cache reads differently, distribute writes to reduce wear, etc).

tjhb

7,545 post(s)
#01-Dec-17 21:45

(Fix) Tables in MAP file no longer fail to accept more than 2 billion variable-length values.

Does "2 billion" mean 2^64 - 1, or 2 billion decimal (2 x 10^9 US, or 2 x 10^12 UK--which)?

Or something else. Loose lips sink ships.

[Added] Another way: you could state the actual data type used for the index or pointer.

adamw


7,307 post(s)
#02-Dec-17 09:18

2 x 10^9, the limit is 32-bit: the number of records in a table is a signed 32-bit value (because one of the key structures is using it that way).

We will remove this limit, we just want to couple this with optimizations in MAP file structures.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.