Subscribe to this thread
Home - General / All posts - Message: Image cannot be rendered because it stores no data for intermediate levels
dchall8
522 post(s)
#13-Nov-18 17:40

This is going slow and is kind of herky jerky in practice. Here's what I'm doing:

  • Imported 125 DEM files using a script
  • Created new Map
  • Dragged DEM images from project into Map. The tab for each DEM image has a red stop sign icon.

  • Go to View>Messages to get the dialog window with the message (Image cannot be rendered because it stores no data for intermediate levels.) The dialog gives two clickable options to either Build Temporary Index or to Dismiss. The message dialog shows almost three of the messages and cannot be resized.

  • Click the Build Temporary Index (BTI) button and the red icon disappears from the tab.

  • Click the second BTI button - red icon disappears

  • Click the third BTI button - red icon disappears

  • Scroll down to continue clicking the BTI buttons. When I scroll down the dialog does not scroll to the 4th file. It jumps down several files. There is no indication in the window as to which of the files has a new temporary index, so at this point I'm just guessing which button to push (herky jerky).

    Am I doing this process wrong, or is there a way to click all these buttons at one time? If I have to do this process one image at a time, then if I could resize the Message dialog window I could have better control. Or if the individual messages disappeared from the window or changed color as I clicked the BTI buttons that would make this go much faster.

    Eventually I want to merge all these DEM images into one for easier analysis. The final image will be hillshaded and used to trace the remains of ancient native American and European walking trails from community to community.

    Attachments:
    DEM Message Dialog Window.jpg

  • tjhb

    8,410 post(s)
    #13-Nov-18 17:46

    First statement: these images seem to be linked, not imported. Can you show the script?

    dchall8
    522 post(s)
    #13-Nov-18 17:57

    That very well may be. The script was adapted from another script used to import/link to LiDAR files. Here it is.

    ' VBScript

    Sub Main

    Set app = Manifold.Application

    Set db = app.GetDatabaseRoot()

    Set fso = CreateObject("Scripting.FileSystemObject")

    Set textFile = fso.OpenTextFile("c:\DEMsForCarter.txt")

    i = 1

    Do Until textFile.AtEndOfStream

    app.Log "Processing file " & i

    app.OpenLog

    filePath = textFile.ReadLine

    fileName = fso.GetFileName(filePath)

    tableName = Left(fileName, InStr(1, fileName, ".")-1)

    newPath = CreateBackslashPath(filePath)

    text = "CREATE DATASOURCE ["+tableName+"] ("

    text = text + "PROPERTY 'Source' '{ \""Source\"": \"""+newPath+"\"", \""SourceCacheExternal\"": \""false\"" }',"

    text = text + " PROPERTY 'Type' 'dem' );"

    db.Run(text)

    i = i + 1

    Loop

    textFile.Close

    app.OpenLog

    End Sub

    Function CreateBackslashPath(filePath)

    newPath = ""

    for i = 1 To Len(filePath)

    char1 = Mid(filePath, i, 1)

    newPath = newPath + char1

    If char1 = "\" Then

    newPath = newPath + "\\\"

    End If

    Next

    CreateBackslashPath = newPath

    End Function

    tjhb

    8,410 post(s)
    #13-Nov-18 19:59

    The script was adapted from another script used to import/link to LiDAR files.

    I think the script is adapted from otm_shank's code here.

    Try changing the link properties to use an external cache file (*.mapcache) for each datasource (one change, in bold):

    ...

    text = "CREATE DATASOURCE ["+tableName+"] ("

    text = text + "PROPERTY 'Source' '{ \""Source\"": \"""+newPath+"\"", \""SourceCacheExternal\"": \""true\"" }',"

    text = text + " PROPERTY 'Type' 'dem' );"

    ...

    This gives Manifold somewhere to store the data for intermediate levels. It will now build them automatically, and an RTREE index for the intermediate tiles.

    dchall8
    522 post(s)
    #13-Nov-18 21:21

    I think the script is adapted from otm_shank's code here.

    Absolutely yes. Several people were involved in getting that script right with otm_shank being the major contributor.

    So you're suggesting the only change is to change false to true in that line of the script? That script runs, but is it supposed to import or is it a link? Normally for an import I get a folder icon container, but for this the container looks like a tall drum. Also the Manifold file size is very small.

    When I add the images to a Map, the max number of images is 4 before the program blinks off with the suggestions as before.

    The good news is there is no longer a red message icon.

    dchall8
    522 post(s)
    #13-Nov-18 21:56

    There's data problem causing Manifold to close. I started bringing them in from the bottom of the list and they're going in fine until I get to a certain point. I ran into this before with this data set, so I'm checking my notes.

    tjhb

    8,410 post(s)
    #13-Nov-18 21:57

    That script runs, but is it supposed to import or is it a link?

    The script creates child datasources. This is the same as using File > Create > New Data Source in the GUI, and functionally equivalent to using File > Link.

    If you want the equivalent of File > Import, you need a different script.

    When I add the images to a Map, the max number of images is 4 before the program blinks off with the suggestions as before.

    I don't follow how exactly this fits in. Can you give exact steps? Prefereably starting from "Open Manifold 9.0.***.*. Create new project."

    dchall8
    522 post(s)
    #13-Nov-18 22:38

    There's at least one corrupted DEM file that causes the program to halt. I'm bringing them into the Map 10 at a time until it stops. Then I go 1 at a time to find the bad file. When I open a bad one in Notepad++, there is definitely a problem visually with the data. Normally the data are in one long stream. The broken ones have large gaps in the stream.

    tjhb

    8,410 post(s)
    #14-Nov-18 02:23

    If you can find a corrupted file that breaks the linking process "ungraciously" then don't delete it, as I suspect Tech might like to have it.

    adamw


    8,259 post(s)
    #14-Nov-18 08:05

    Echoing what Tim says above: if you can find a file that produces issues (causes the program to halt as per your post), please contact tech support. This way we can find what the problem is and fix it.

    dchall8
    522 post(s)
    #13-Nov-18 17:55

    Now to follow that up, after clicking through maybe 20 of those BTI dialogs, M9 stopped working giving me the option to check online for a solution, close the program, or debug the program. I closed the program and reopened the file. The Map is intact (I had saved along the way). The red icons are back, so they truly are temporary.

    adamw


    8,259 post(s)
    #14-Nov-18 08:14

    Whenever 9 stops working, you can help us immensely to understand why by capturing a crash dump.

    To capture a crash dump, do NOT stop the Manifold process, that is, do not click any buttons in the dialog with the error message just yet. Instead, launch Task Manager, locate the Manifold process in the list of processes (in Windows 10, the list of processes is in the Details tab of Task Manager), then right-click the process and select Create Dump File. The system will take a moment to write the dump file and will then tell you where it put it. Zip it up and send it to us.

    Having a dump file does help us a lot, frequently we can see the issue right away, and the sooner we see what the issue is, the sooner we can fix it.

    dchall8
    522 post(s)
    #14-Nov-18 14:46

    Knowing which file causes the problem makes this easy. The zipped dump file size is just over 1GB. What's the best way to get that to you?

    I could also zip up and send a problem DEM file along with a good one. Those zip to 12MB which can go by email. Then you can write your own dump file

    adamw


    8,259 post(s)
    #14-Nov-18 15:00

    Email tech support, say that you want to report a potential bug and have accompanying data (full instructions) and they will provide you with FTP space to upload the data to. Reference this thread as well, for completeness. Do send the problem DEM file as well if you can, that also helps - ideally send both the dump and the DEM, the dump will come in handy if the crash will prove to be only tangentially related to the DEM, and the DEM will make sure that we find all issues with the file if there are more issues with it in addition to the one shown in the dump.

    Thanks a lot in advance!

    dchall8
    522 post(s)
    #14-Nov-18 15:41

    Done. Hope it helps.

    adamw


    8,259 post(s)
    #14-Nov-18 08:25

    Setting aside the topic of persistent vs temporary indexes with intermediate image levels (your script explicitly tells the system to not use persistent cache and this is why the system asks you to build it for each image), I do not quite understand this:

    Scroll down to continue clicking the BTI buttons. When I scroll down the dialog does not scroll to the 4th file. It jumps down several files. There is no indication in the window as to which of the files has a new temporary index, so at this point I'm just guessing which button to push (herky jerky).

    You have a lot of images, so the dialog is not big enough to display messages regarding the lack of intermediate levels for all of them, so you have to scroll the list of messages to see them all, fine, I understand this. But what happens next - are you saying that when you try to scroll the list of messages, it jumps to unexpected locations instead of scrolling normally? What exactly happens?

    I don't quite understand the last sentence either - each message has the layer name on it. When you perform any action suggested for the message, the message should disappear from the list. What exactly happens that you are not sure which button means what?

    Last, it might be a good idea to have a catch-all button to build intermediate levels for all images that don't have it. Same for building temporary spatial indexes for drawings that don't have them, etc. We'll think about adding this.

    dchall8
    522 post(s)
    #14-Nov-18 15:14

    There is an unfortunate naming convention involved with so many files written by machine. There are 2,000 files in the folder from which I have 125 in my target area. File names are like this, 2999_14_1_c_1 followed by ...c_2, etc. up to c_4 and then it goes to d_1 most of the time, but not always. All you have to do is blink and you lose your place looking at these things. Every part of that name label changes around the county.

    Now I'm repeating the BTI process and watching carefully as I write this. (Clickity click click...) OMG, they do disappear. Ok, this is trivially easy, now that I see what's happening. There is a lot of clicking, but it is all in the exact same spot, so they clean up just as fast as I can click on them.

    Scrolling does scroll, but with the unfortunate naming convention, it is hard to tell how far it has scrolled.

    StanNWT89 post(s)
    #14-Nov-18 16:15

    Not that this is likely to help, but your issue with file naming conventions in a long list of files made me think of the utility I use do deal with a lot of filenames. It's a piece of shareware called "Advanced Renamer", you can replace any combination of a part of a filename, prefix or suffix using wildcards. It's quite useful. If you think that such a program would help convert your existing filenames to ones that are more easy to sort through, give it a go? Perhaps do it on a copy of the folder and it's contents first, if you have sufficient storage for such a test?

    dchall8
    522 post(s)
    #14-Nov-18 17:52

    That's a good idea. I have one called Bulk Rename Utility that I've been using since the 90s I think. I use it every week for one purpose and don't really think about other places it might be useful. This is definitely one of those. Renaming might help me diagnose errors I believe are in the names of some of these files.

    Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.