Subscribe to this thread
Home - General / All posts - Invalid data item - cannot open .map file
rk
228 post(s)
#12-Jan-18 13:45

Build 9.0.164.2

I have broken my .map file. I get 'Invalid data item' when trying to open the file.

I saved a .map several times. Then I saved it with different name. Both files does not open anymore.

Dimitri


4,422 post(s)
#12-Jan-18 13:47

Interesting... Is the .map open in another Manifold session? Is it linked into a project that is open in another Manifold session?

rk
228 post(s)
#12-Jan-18 13:49

No, I closed all Manifold sessions

adamw


7,444 post(s)
#12-Jan-18 15:03

Was the file big (hitting the limit of free space on the temp drive)? Were the fault-tolerant saves on (they are on by default, you can turn them off when you link a MAP file as a data source)? If the fault-tolerant saves were on, is the .SAVEDATA file there in the folder of the .MAP file? What was the last action prior to the save and how long did it take to complete (trying to gauge the amount of changes)?

Ideally, we need a way to reproduce this to understand what is going on. Lacking that, we might be able to tell something from the files (need .MAP plus .SAVEQUEUE if it is there).

rk
228 post(s)
#12-Jan-18 15:12

I did a lot of changes. Mostly deleting data from big tables. Also changing queries and arranging into folders.I had problem deleting from particular table. I had to drop the table.

The file is 6GB, but should be much smaller after deletes I made.

The first location was on network drive, the Save As location was local drive. Everything seemed to work until I closed the files and tried to reopen.

Temp drive free space is 6TB.

I contacted tech and uploaded the file.

Dimitri


4,422 post(s)
#12-Jan-18 15:18

Were the fault-tolerant saves on (they are on by default, you can turn them off when you link a MAP file as a data source)? If the fault-tolerant saves were on, is the .SAVEDATA file there in the folder of the .MAP file?

How about the above?

rk
228 post(s)
#16-Jan-18 11:19

Fault-tolerant saves was on == "SourceSaveQueued" property not set.

No .SAVEDATA files left.

adamw


7,444 post(s)
#12-Jan-18 16:10

The first location was on network drive, ...

Eh. That's most likely the root of the issue.

(Working with files over the network is very dangerous. Working with databases over the network is safe, working with web servers is safe, working with files in live mode where the file is kept opened for a long period of time / is saved incrementally is not. This is true for all files, and particularly for big ones. The main issue is that the network sometimes says 'ok, I wrote what you wanted where you wanted' but that 'ok' is actually susceptible to the network connection failing some time later and leaving the write incomplete. The application frequently has no means to even know about the failure because it closed the relevant handles thinking that everything went fine. Yet the file on the server is in a zombie state with the writes that only completed partially. This is why Microsoft, for example, required that nobody works with MDB files over network shares - people still did, but they were always on their own, that could fail horribly and it did fail horribly losing data plenty of times. We have been considering just copying MAP files accessed over the network locally and copying them in full back after they are saved, but there are obviously huge performance / space repercussions.)

tjhb

7,601 post(s)
#12-Jan-18 21:30

Would it be possible for each change written to a .map file (remote or not) to then update a unique version value in the file (a UUID, say)? On the next transaction, the expected version ID would first be read back from the .map file, and in case of a mismatch (indicating that the previous write had failed), the user would be alerted, could do a Save As instead, and/or the previous transaction could first be repeated. Then a new unique version value would be written... and so on.

I know this is naïve--if it would make a crucial difference then you would already have done it. Maybe the unique version value would almost always seem to have been written, even when it had not, because of OS-level caching. Also it ignores difficulties with (eventual) concurrent multi-user editing.

adamw


7,444 post(s)
#13-Jan-18 07:56

This is more difficult, because the UUID might end up in the file while the write it confirms might not. It takes effort to enforce the order of writes.

There are techniques to detect failures and techniques to protect from failures (ie, by allowing to rollback to the previous state) in the presence of unreliable carrier, yes. Eventually, we might prevent this kind of damage from the network.

KlausDE
6,075 post(s)
#13-Jan-18 11:12

For me a more handy access to the temporary file that Mfd8 created during saving would be sufficient. Isn't Mfd8 rereading the saved file and thus checks its integrity before it closes. The failure could automatically offer a rollback to the temporary file in the local folder.

tjhb

7,601 post(s)
#13-Jan-18 12:55

But Manifold 9 does not do this. It writes to the open project file (database). Thus the amazing speed.

rk
228 post(s)
#16-Jan-18 11:58

It happened again on another computer with Build 9.0.164.1 64bit. This time .map files were on local disk (D:\). Only some linked dwg datasources were on network.

TEMP free space 600GB (D:\)

pagefile free space 45GB (C:\)

I have more details. We are also trying to reproduce it.

Dimitri


4,422 post(s)
#16-Jan-18 15:24

Only some linked dwg datasources were on network.

OK, so you are working with files over a network. As Adam noted, working with files over a network is very dangerous. Have you analyzed your network? There are some great tools out there.

What happens when you make copies of the network files and put them onto your local machine, so 100% of what you do is local?

rk
228 post(s)
#16-Jan-18 19:06

working with files over a network is very dangerous.

Noted. In my mind, I placed saving .map fileover a networkand reading .dwg over a network (and even saving .mapcahe) in different categories. Can local .map be corrupted by failing write of .mapcache on network?

Have you analyzed your network? There are some great tools out there.

No, I have not. Can anyone suggest a tool and/or some howto reading?

tjhb

7,601 post(s)
#16-Jan-18 20:30

Can local .map be corrupted by failing write of .mapcache on network?

What a great question. (Following this with interest.)

Another question: assuming a live local network connection is to be used, is it more reliable to use WiFi rather than Ethernet? I think wireless protocols have (even) better error detection and recovery--but I don't really know.

adamw


7,444 post(s)
#17-Jan-18 07:05

Another question: assuming a live local network connection is to be used, is it more reliable to use WiFi rather than Ethernet? I think wireless protocols have (even) better error detection and recovery--but I don't really know.

It depends on the driver, but generally it doesn't even matter what type of the network it is, because Windows itself sometimes disconnects and reconnects network resources in order to solve some other issue (prevent some cache from degrading which is hard to get to in other ways, etc).

If the user just performed a save over the network, some of the writes may not make it to the destination. The bigger the file and the larger the amount of saved changes, the worse it is.

adamw


7,444 post(s)
#17-Jan-18 07:01

Can local .map be corrupted by failing write of .mapcache on network?

Directly, no. Indirectly, if a structure working with .MAPCACHE will get malformed data (from an earlier failed write), it might return wrong data to the caller and that could potentially escalate into damage to opened .MAP files, yes. The structure would have to get pretty unlucky in that wrong data would have to coincide with some of the correct data, for example, but it is possible. (I wouldn't bet on this happening in your case, because it seems like a lot of things have to coincide for the issue to be in .MAPCACHE accessed over the network, and you have been able to run into it too fast from the previous report. Let's hope it is something local - maybe some bug in the DWG dataport, by the way - because that's vastly easier to reproduce and fix. :-) )

Looking into the network beyond the very basics is largely fruitless, because Windows itself frequently installs patches that disconnect / reboot the network regularly in order to avoid other issues.

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.