Subscribe to this thread
Home - General / All posts - GPKG: what am I doing wrong?
tjhb

8,760 post(s)
online
#15-Apr-19 03:25

I am trying to File > Import a GPKG file in 9.0.168.0 or 9.0.168.11.

I have the standard Manifold sqlite-spatialite dll extensions installed.

Nothing happens, with either build. (I know, but yes, nothing.)

I'm almost certain that I have previously had success, at least with 9.0.168.0.

Dimitri


5,452 post(s)
#15-Apr-19 07:50

I've attached a sample gpkg file (zipped). Here's what I did, using .11:

1. Unzip the 64-bit extensions package into the bin64 folder of the portable extension.

2. Launch manifold.exe from that folder.

3. Import the unzipped sample.gkpg file. That produces a mexico drawing and a bronze image.

Guesses, what might be an issue:

a. need to unblock the dll extensions package after downloading it?

b. not in the bin64 folder or otherwise in the PATH?

c. haven't launched a fresh Manifold session? [just unzipping it into the path won't affect a running session]?

d. downloaded dlls quarantined by some third party virus/security package?

Attachments:
gpkg.zip

tjhb

8,760 post(s)
online
#15-Apr-19 22:00

Thanks Dimitri! It was (a).

Feeling a bit foolish--I thought Microsoft had stopped setting that flag (under Windows 10). I was wrong.

I've a good clean-up, using this old script, and applied the registry patch to prevent further trouble.

tjhb

8,760 post(s)
online
#15-Apr-19 22:20

...I spoke too soon. Still no luck. Packages (and all other files) unblocked, new copies extracted, hard links deleted and recreated. Rebooted.

Still getting errors like

2019-04-16 09:10:35 -- Startup complete (0.761 sec)

2019-04-16 09:10:35 -- Create: (New Project) (0.016 sec)

2019-04-16 09:10:40 *** Failed to load library: D:\manifold-9.0.168.11-x64\bin64\extension-dlls\sqlite-spatialite-x64\sqlite3.dll

2019-04-16 09:10:40 *** Failed to load library: sqlite3.dll

2019-04-16 09:10:40 *** (import) Cannot connect to database.

2019-04-16 09:10:40 *** Cannot run command.

Puzzled. I am going to think it may be your (d)--that would be Windows Defender or Smart Screen.

I am going to restore an OS image to see if that corrects it.

tjhb

8,760 post(s)
online
#15-Apr-19 23:41

That didn't help.

Fresh download of database packages. Not blocked. Freshly extracted.

Nothing. Same errors logged.

IronPython .dlls have been loading correctly throughout.

More puzzled. I won't keep bothering everyone.

Dimitri


5,452 post(s)
#16-Apr-19 07:59

Try taking the .dlls and putting them within bin64, not in a sub-folder. (just a wild guess)

Permissions? Some sort of group policy?

tjhb

8,760 post(s)
online
#16-Apr-19 09:21

I did wonder about permissions, so I went back to a freshly reformatted working drive. Then reverted to a Windows 10 image made prior to installation of Manifold 8 or 9 (leaving them uninstalled), added Visual C++ 2017 runtimes, unzipped Viewer and added the extension .dlls directly to its path (not in subfolders).

Still no luck! Attempting to import a .gpkg just does nothing. (No logged errors now because it is Viewer.)

On two other machines, with the same basic drive and folder structure (as this machine when it is set up normally), no problem at all, always works.

I will keep trying, probably next with a fresh blank Windows 10 install, just runtimes, Viewer and .dlls. If that doesn't work I'll post full installation notes.

Thanks for your patience Dimitri.

tjhb

8,760 post(s)
online
#17-Apr-19 08:43

The test with a fresh Windows 10 install also failed.

tjhb

8,760 post(s)
online
#17-Apr-19 09:07

I have a hypothesis to test tomorrow.

Plus notes to send as a bug report to Tech.

tjhb

8,760 post(s)
online
#19-Apr-19 01:41

Manifold's database extension libraries, posted on the Product downloads page of the website, are either corrupted or outdated.

At least for GPKG (I have not tested any other format), they do not work with any remotely recent version of Manifold 9.

I get exactly the same error as detailed above on a very different computer, running Manifold 9.0.164.0 (prerelease, an old backup image useful in this case). No result, same error message.

What does make it work? Installing GDAL (I use GISInternals), and adding that to the user PATH.

I strongly suspect that that is what makes the import work for Dimitri as well. I.e. that he also has GDAL installed and accessible in user PATH. If he did not, he would get no import and the same error messages as I do.

To make a long story short: don't use the current Manifold database extension downloads, at least not for GPKG.

The warning on the source page is correct:

The above packages of DLLs are not supported and are infrequently updated. They provide an example of what DLLs are usually required and where they should be placed.

But that is not enough. The download should be either updated, or removed.

How many hours have I wasted on this? I'm with the alligator. I alligate. I am alligating right now.

Attachments:
grrrrrrr.png

tjhb

8,760 post(s)
online
#19-Apr-19 02:10

P.s. not at all at Dimitri, who has been extremely helpful, and is probably as surprised by this as I am.

oeaulong

192 post(s)
#19-Apr-19 18:38

I tried this and failed to reproduce the issue. Dimitri's GPKG opened for me. I do have QGIS 2.4 installed. Will try upgrading to see if this breaks it. I will try and go over this thread again and spin the noodle on it.

Good luck T.

Dimitri


5,452 post(s)
#19-Apr-19 08:18

At least for GPKG (I have not tested any other format), they do not work with any remotely recent version of Manifold 9.

Puzzling, but I cannot replicate the above. Here is what I did:

1. I located an old machine not recently used, which had old GDAL and QGIS installed.

a. renamed GDAL intallation folder from OSGeo4W64 to -OSGeo4W64

b. renamed QGIS installation folder from ~\QGIS 2.18 to ~\-QGIS 2.18

2. Verified QGIS no longer launches.

3. Launch Manifold and verified GDAL no longer works.

4. Download from the Product Downloads page and put the sqlite-spatialite-x64 folder in the ~\bin64 folder.

5. Launch Manifold and verify gpkg works.

It's puzzling. I have another machine, a 32-bit notebook, which has never had either GDAL or Q installed. I'll try with that after lunch.

By the way, I tried with Q installed/not installed to eliminate the possible path that if the sqlite dlls from the downloads page don't work, it is having a Q installation that is installing alternate sqlite dlls in the PATH.

Just to be sure, I searched the disk drive for ALL instances of spatialite.dll and verified that they existed only within the ~\bin64 folder where I placed the latest download, or within the paths where I modified the names so any PATH variable references will not work (I also checked the PATH and other environment variables).

It is interesting to note that the OSGeo4W64 and also the QGIS 2.18 spatialite.dll files are date stamped 6/23/2016 1:27 AM, exactly the same as the date stamp on the spatialite.dll file provided by the product downloads zip.

The logical conclusion is that if somehow those .dlls are being used instead of the one downloaded, they are still the same thing.

It could be something else going on, a mirror image of the problem: consider the hypothesis that the dll being used is not the one you think, but a dll that comes earlier in the PATH.

Case A: In your statement of that hypothesis, you state the possibility that maybe the dll Manifold provides is wrong, but my test case was working because without realizing it I was launching a good dll installed by GDAL that came ahead in the PATH before the "bad" dll I got from the Manifold site.

Case B: But it could be the mirror issue... suppose the reason you had problems was because you had an old GDAL/QGIS/something else that placed a too-old spatialite.dll in the PATH before the dll downloaded from Manifold. In that case, instead of running the dll provided by Manifold you were running a too-old or otherwise problematic dll installed by something else. When you installed GDAL, it could be you placed a newer, OK spatialite.dll in the path ahead of the problematic dll and then suddenly it started working.

Like I say, I see no spatialite.dll files in the PATH ahead of the one I just downloaded from the Manifold site. I checked that by looking at environment variables and also by searching the disk for any occurrence of spatialite.dll. I might have missed something, but the above seems to indicate the spatialite files downloaded from Manifold are OK. I'm baffled why they work for me but not for you.

PS: Great photo today on the site. Some I love, some I don't but this one I like. I think the theme is crunching with many teeth at once lets you bite off a bigger chunk of data. :-)

adamw


8,579 post(s)
#19-Apr-19 10:53

I followed Dimitri and also tried to reproduce the issue, and could not. (Unfortunately, else we'd fix it.)

I used a clean test machine (well, Windows 10 with all updates and with VCRedist already installed, no GDAL or anything like that and no custom PATHs to anything). Downloaded the archive for the 168.11 portable from the Release 9 Downloads page, unpacked it. Downloaded the archive for the 64-bit client DLLs from the same page (extension-dlls-x64.zip), unpacked it into ~\bin64 of the portable install. Launched the 64-bit EXE in the portable install, imported an example GPKG, everything worked. The log contains no errors:

#

2019-...  -- Manifold System 9.0.168.11 Beta

2019-...  -- Starting up

2019-...     Log file: ...\Local\Manifold\v9.0\20190419.log

2019-...  -- Startup complete (0.731 sec)

2019-...  -- Create: (New Project) (0.010 sec)

2019-...  -- Load library: ...-dlls-x64\sqlite-spatialite-x64\sqlite3.dll

2019-...  -- Load library: ...-dlls-x64\sqlite-spatialite-x64\spatialite.dll

2019-...  -- Import: ...t132869\water_lake_superior_basin.gpkg (0.421 sec)

The problem on your configuration evidently is that SQLITE3.DLL fails to load. It's hard to understand why that happens.

The SHA256 hash of the 64-bit SQLITE3.DLL that loaded in my test configuration is:

d004551352eff866df5401281657576ae292eafc6cedeecb2674a97a3df5e8a5

The DLL contains just two dependencies - one of them on Windows (KERNEL32.DLL, you cannot not have that and SQLITE3 does not use anything extraordinary, so the problem is likely somewhere else), and another on the CRT (MSVCR100.DLL). Could it be that you updated VCREDIST / rolled back the system which changed VCREDIST recently? Or maybe something added an entry into PATH and some folder with a custom version of MSVCR100.DLL started winning searches for it (and that version is incompatible with what SQLITE3.DLL that we have expects)?

tjhb

8,760 post(s)
online
#20-Apr-19 00:10

Thanks Adam! Your observations about dependencies got me 90% of the way, experiment did the rest.

The SQLite libraries provided by Manifold for download require the Microsoft Visual C++ 2010 redistributables (latest versions here).

It is not enough to have only later versions of the C++ redistributables installed, which is the default situation after a clean install of Windows 10 (and possibly Manifold 9).

On the other hand, the SQLite libraries installed by GDAL version 2.2.x (recommended for use with Manifold 9) are built against either the 2013 runtimes or the 2015 runtimes (I use the 2013 version), and do not require the 2010 versions as well.

That explains why installing GDAL and placing its binaries in the PATH allows GPKG imports through Manifold 9, even when the Manifold-supplied libraries cannot be used because the 2010 runtimes are missing. (I expect the same is true a fortiori for recent versions of QGIS.)

Solved!

Many thanks to everyone who gave this annoying little problem their attention. Very grateful.

(Maybe it would be worth adding a note to the downloads page? If confirmed.)

adamw


8,579 post(s)
#20-Apr-19 08:18

Thanks. We'll try to update the DLLs for database clients to latest versions, then check their CRT requirements and document anything that needs special attention.

Dimitri


5,452 post(s)
#20-Apr-19 08:38

It is not enough to have only later versions of the C++ redistributables installed, which is the default situation after a clean install of Windows 10 (and possibly Manifold 9).

But... if the above is the case, how is it that adamw's post reported success starting from a clean install of Windows 10 with VCRedist already installed? I base that question on the guess that it is unlikely the VCRedist he had installed was the 2010 redistributable.

Another example: I launched .11 from a portable installation on a zip drive connected to a 32-bit notebook computer that has never had GDAL or QGIS installed. The redistributable installed was later than 2010. I downloaded and unzipped the 32 bit dlls into the ~\bin folder. Worked fine for GPKG.

Here's another possibility: maybe if you have Release 8 installed on the same machine that might have an older VCRedist?

I agree it's something to be tracked down, dll's updated to the latest versions, etc. There was another thread on bundling all the database dlls into a "fat" version of the portable installation, maybe even by default.

artlembo

3,113 post(s)
#19-Apr-19 14:32

can you try connecting to the .sqlite database with QGIS, just to make sure that you can get a valid connection?

Also, maybe open a Python script and connect by:

import sqlite3

db=sqlite3.connect(':memory:')

b=sqlite3.connect('data/mydb')

just trying to rule out some things...

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.