Subscribe to this thread
Home - General / All posts - Invalid object reference in Future
dchall8
1,008 post(s)
#17-Oct-17 21:54

I'm sure this isn't a crash, but the effect is the same. I get an Invalid object reference warning box. When I click OK, it returns immediately. Here's what I was doing...

I'm using Windows 7 Pro on a top of the line (3 years ago) Dell Optiplex 9020 i7 with no external GPU. I have a 9GB file of LiDAR point clouds. I want to export one of these point clouds so I can open it in Manifold 8 along with altitude information. I tried the .mfd file and it would not even start the export. I tried exporting to a .shp file. It took 20 minutes and when I opened the file in Manifold, the XYZ data fields were missing. I tried to export a .kml file. After about 15 minutes it stopped with a No memory dialog. I closed three Firefox windows to release some memory. I clicked the OK button on the No memory warning and went back to File > Export to see what my other options were. The Export window opened black. I red-x'd the window and tried again. Got the black window again and red-x'd again. After a few seconds the No Memory warning came up along with the Invalid object reference. I can't do anything except continue clicking the OK button. Memory usage from Windows Task Manager is 558,108 K for Manifold System 9.0. Ending the task in Task Manager closed it immediately.

So if there is a question here, how do I export LiDAR points such that I can use them in Manifold 8? I want to make a topographical map.

dchall8
1,008 post(s)
#17-Oct-17 22:17

It's in the middle of doing it again. This time instead of exporting from the open drawing I tried to export from the open table. After failing to export to .mdb I tried .mml and got the No memory warning. After that Future is unresponsive to the Export command.

Attachments:
Win Task Mgr Following failed Futures Export to .mml.jpg

tjhb
10,094 post(s)
#17-Oct-17 23:38

I want to export one of these point clouds so I can open it in Manifold 8 along with altitude information.

Can this be done? If so, what file formats would support it?

I would say: no, because none.

Radian/Manifold Future are far, far more capable than Manifold 8, especially regarding large data sets (and speed of course).

I think you are trying to do something that can't be done.

Cartography on massive point clouds will come in Manifold Future, but is not there yet.

For now, you could use Global Mapper, somewhat clunky for cartography but perhaps worth a try.

Dimitri


7,413 post(s)
#18-Oct-17 07:39

I get it that you are frustrated but you have to do two things for people to help: the first is to provide important info, and the second is to acknowledge that when software lets you command vast tasks it is on you to use that power wisely.

First, info:

I have a 9GB file of LiDAR point clouds.

C'mon, you should know better by now than to write such inadequate descriptions. If you work with LiDAR you know that what storage format LiDAR uses makes a big difference. Is it LAS (not compressed) or LAZ (compressed)? 9GB of compressed LiDAR can expand into 900 GB of uncompressed points, although more usually it wouldn't be more than what? 100 to 200 GB uncompressed? Still, point clouds can be vastly larger than the compressed size of a LAZ file might indicate.

I want to export one of these point clouds

Whether that is realistic given the hardware you are working with depends on the uncompressed size of the point cloud compared to the resources you have:

top of the line (3 years ago) Dell Optiplex 9020 i7

We can't tell from the above whether you have the resources to work with 9 GB, let alone with 200 GB. There is a big difference between "top of the line" and "top of the line with 200 GB free on my C: drive but only 250 MB free on the SSD drive I use for pagefiles, temp files, etc., storage."

Release 8 has an extensive Performance Tips topic that discusses obvious matters like "if you want to work with 100 GB projects you will need more than 100 GB of free space on disk...". The Radian documentation has a much simpler Performance Tips topic based on the observation that people who need such advice don't ever read documentation in the first place. I personally think that is harsh and that there should be explicit advice in Performance Tips, or a "memory usage" topic like 8 has, that explicitly warns people if they want to work with 100 GB data they need much more than that in free space for things like temp storage, etc. It's easy to forget that sometimes so a reminder would help.

But however you play it, if you work with big data you'll need lots of free space that goes well beyond the size of the data. Radian is far better about that than 8 (amazing how well a 32-bit Radian installation performs, for example), but it cannot fit a size 20 foot into a size 8 silver slipper.

Onward...

So far, we do not know the size of data with which you are working or the resources on the machine, but all the same with the information we have we can touch on some of the key issues:

Release 8 is not a good choice to work with 9 GB of raster data. It will be slow at that, which is why Radian technology was invented. Depending on how you choose to export it, the data could get even bigger.

I tried the .mfd file and it would not even start the export.

.mfd is not Release 8 .map format. It is older technology that uses .mfd/.mdb and thus is limited to 2 GB in the mdb. I guess that is why the export was declined.

In a perfect world Radian would have error messages that explained the limitations of various formats if people tried to misuse them. For now, given it started life as a DBMS tool and not a consumer tool it has prioritized adding more formats and greater connectivity for people who know how to use those formats and connectivity than to educate people who may be unfamiliar with given formats and data sources.

I think such education is a great thing and is absolutely vital in a consumer product and very helpful in a professional product as well, so I'm not in any way putting down the value of error messages like "Sorry, .mfd/.mdb format is limited to 2 GB. You are trying to export much more than that" ... I'm just pointing out that given a choice between prioritizing those general education sorts of messages and adding yet more tools for people who want power, well, Radian has gone the power route for now.

By the way, such general education messages are not remotely "free." Given that all sorts of processes can result in the data streams that one might command as an export it is often the case that the system does not know how big the export will be until it does it. So, there's none of this "hmmm... my user wants to export 20 GB into a format that can only handle 2 GB... better not give him the option of doing that and instead issue an error message..." Systems which thoughtfully check the size of the export in such situations so they can show you only valid formats as options accomplish that by doing the export, in effect, twice: first to discover what the size will be and then second to actually export. That's a fine strategy for consumer software that works with small data but it is crazy expensive in time and capacity for software that can work with hundreds of GB of data and often does. There, too, Radian is not going to crush performance and load up on resource use in order to serve a perceived priority that guarding against a poor choice of format is more important. It could be that balance will change in the future, say, perhaps by guarding against totally obviously wrong choices of format, but for now it seems better to prioritize performance and broader power.

I tried exporting to a .shp file.

Shapefiles are limited to 2GB. Exporting a 9 GB vector data set to shapefiles is not realistic. That might be a good example of a totally, obviously wrong choice of format where with little overhead Radian could advise users "This format cannot handle the size data you want to export"

I tried to export a .kml file. After about 15 minutes it stopped with a No memory dialog.

KML is a spectacularly inefficient format because it converts everything into text. Consider the representation of a binary number: four bytes (four "characters") is enough to represent 1020983.020930349, which takes 17 characters as the numeric text string. It takes even more characters if you spell it out as words "One million twenty thousand nine hundred three and...." (I won't even go into the words for the decimal fraction...). Representing point clouds in KML is almost as bad as spelling out each point's location in words. Not quite (I exaggerate to make my point) but almost.

So, 9 GB as LAZ storage of a point cloud can easily be 100 GB as real, uncompressed binary vector data and 3 or 5 TB as KML.

Google doesn't claim to load KML beyond 10 MB in Earth or 100MB as fusion tables, so it is not so easy to get straight answers as to what the maximum size of data the KML standard allows. However, since you tried the export I'll toss that ball back to you... when you researched what the maximum size of a KML file can be, what did you find?

My guess is that what happened, and it is only a guess, is that your data is way larger than 9 GB, and that the task of trying to export to various formats, especially KML, is far larger than the free resources you have on your machine. Radian tried anyway to do what you commanded and that as it kept hitting limits it kept popping up error messages to let you know. If you had the patience to click on all those messages at some point they'd stop as Radian will have exhausted all possibilities to do what was commanded.

I would be the first to agree that Radian should have a "Forget it, I want to abandon this whole thing" option that makes it easier to bail on such cases. In terms of priorities, that, too, is not easy to do because some sorts of data sources and exports to give you that option require what end up being performance-killing (or at least damaging) belt and suspenders measures. So that's a tradeoff to make.

My advice would be to use a format suitable for really big data, perhaps, storage in a DBMS, to which both Radian and Release 8 can connect. Release 8 is still a poor choice for very large point clouds, so maybe it would be even better to see if the task could be done entirely in Radian.

dchall8
1,008 post(s)
#18-Oct-17 15:05

In the future of Future, this will all be handled within Manifold 9. My current attempts at an export/import workaround are just experiments with varying levels of success. Had I asked the question about exporting from Future and importing to 8 without having already tried some things, the first question would have been, "What have you tried?" It would be more to the point to focus on the idea that Future went into some sort of do-loop and did not recover cleanly. Y'all have gone on and on about how Radian is crash proof. Well, if this is not a crash, then it is the functional equivalent in that the program stopped being useful. It could not be shut down and had to be closed through Windows Task Manager. That's how I normally deal with crashed programs.

I'm trying to be helpful by providing as much information about how I caused the problem and what resources I have. The 9GB file is the .map file itself. Future file sizes seem to be a factor of 10 larger after importing a Manifold 8 file, so I don't know if 9 GB is large or not. The file contains two imported .las files (680 MB each) and a link to a third. In addition there are 9 tables of points extracted from several other .las files. These are subsets of the larger file. In most cases I'm only interested in the returns classified as buildings. In this case that caused the problem, it is the points on the ground I'm interested in. This is the bulk of the .las file. There's probably a way to determine how many points are in each file, but I don't see it in Future. In just looking at the project pane, I don't consider this to be a large project. The fact that it is a 9 GB file was a big surprise to me. Memory usage while running Future, according to the image I attached, seems fine with plenty of overhead for other programs to run. Even while Future was in trouble there was plenty of memory available.

I don't normally dabble in the realm of the largest file possible for any certain format, so I've never thought to investigate that. Is export file size relevant to the issue? Future seemed to choke after the file export failed and the warning was closed.

So, given that I'm trying to do something that the software is not ready to do yet (or maybe ever), what can I do to help discover why Future stopped working?

Attachments:
LiDAR point cloud in Future.jpg

Dimitri


7,413 post(s)
#18-Oct-17 21:58

what can I do to help discover why Future stopped working?

From your prior post it seems that Future did not stop working, it was trying to do what you told it to do and raising error messages. I'd bet that if you had the patience to click through many error messages when you clicked on the last one Future would still be there, perfectly alive and ready for your next command.

In contrast, crashes are easy: don't do anything else until you create a crash dump (there are threads in this forum that discuss how) and send that in to support as a bug report.

The situation is much more difficult in cases where Future hasn't crashed, but is still going through the motions of what it was commanded to do. The way Engineering has to deal with that is to carefully set up a reproducible situation to see what is going on.

Sometimes the "going though the motions" bit may reveal a bug in Future and sometimes not. If you can repeat the situation with smaller data, so you don't have to press "OK" to a dialog zillions of times to find out if Future really does surface at the end, ready to go, that would be a good thing.

Well, if this is not a crash, then it is the functional equivalent in that the program stopped being useful.

I'd respectfully disagree, in that people can command a program to do things that puts the program, while it is doing what was commanded, in a state that is not useful. You have to drill into it to see what is going on.

To improve the quality of a product it is necessary to focus, with precise terminology, on cases where the product itself is at fault. That's one reason why it is important not to use the word "crash" for things that are not crashes. It's not that people freak out over the word, it is just that using the wrong word makes it harder in what is already a complex situation to know what really is going on.

Let me try an example to show what I mean: you can command a program like Radian to connect to a web data source in a way that tells Radian to repeatedly try to fetch data in a series of interactions that each can take a long time to time out depending on what that data source is like. Between using Windows facilities and what Radian has been commanded to do it can seem like Radian has locked up, when instead it really is patiently trying to do what it has been ordered.

I can see the point of view of someone who says, "well, fine, but in that case I want a Cancel command within Radian that no matter what it is doing will stop what I commanded." That's a great notion but built into that is the unstated desire, "oh, and I don't ever want to pay any performance price or lose capabilities to have such a command."

Sometimes, with some connections or some tasks, you don't have that option. A good example is how in Release 8 there are some algorithms that are recursive, that cycle down through levels of recursion and don't surface until they are done. They don't come up from many levels deep every few seconds to see if the user commanded a cancel, either. If they did, they'd run much slower. So in cases like that you don't get a super cancel because the overall judgement is that faster algorithms are better than having a super cancel.

A good bit of Radian is like that as well. Start messing with databases and there isn't any "cancel" available because once you get into mucking about with the database you better take what you are doing to completion or you risk leaving it in disordered, partially done state.

In other cases Radian might not have prioritized the engineering effort required to do a cancel because there are bigger fish to fry. It all depends. It is not the most elegant way of issuing a super-cancel but if there are better things to invest engineering into the idea there is always a brute force use of task manager as a cancel has a certain practicality about it.

My reaction to what you described is simple: If Future was raising dialogs for you it obviously was still alive. Crashed software doesn't do that. If those dialogs were annoying and you preferred a super cancel, I'm with you there. To determine whether it makes sense to implement a super cancel for the specific thing you were doing, Engineering has to know exactly what you were doing and the full context.

If you were getting repeated error messages because as Radian was trying to work with temp files or cache you had flatlined on available disk space, well, that's not going to be a front burner item. It's easier just to advise people to not launch such processes when they have insufficient free space.

On the other hand if it was just a really stupid way of Radian to announce it can't do what you want or, even worse, a bug, well, then it becomes a front burner issue to be dealt with in the next build.

You can help that process by filing a bug report (the web site has tips on that). They have to know everything, all the details, to zero in on any bug. So, things like whether a file has been imported or is linked are important, the sizes of those files, the sizes of the .map file, free space on disk and how that is arranged and all that is key.

Some other notes...

Y'all have gone on and on about how Radian is crash proof.

In reality, Radian went through the entire beta process and months of release before anyone found a way to crash it. That was fixed right away, but it did mean the web site had to be changed to instead of saying Radian never crashes that "Radian is almost impossible to crash." :-)

Future file sizes seem to be a factor of 10 larger after importing a Manifold 8 file

That depends on the content of the file and if you have compression turned on. If you have compression turned on in 8 (everybody does) you are not comparing apples to apples. Zip a Radian file to get a better comparison. And then try opening a multi-billion point project file in Release 8 and see if it opens in under a second... I make that point because nobody cares if a file is five or even ten times larger if that makes a difference between it opening instantly or opening in a half hour. Disk space is almost free. Your time is very valuable, so be happy spending disk space to do in under a second what otherwise takes an hour.

There's probably a way to determine how many points are in each file, but I don't see it in Future.

When you import the file the number of points is the number of records in the table. Open the table. Press CTRL-End to jump to the end. You'll see from the mfd_id field how many records there are in the table. That's how many points there are. Using SQL, you can write SELECT Count(*) FROM [TableName]; which will report the number of records in a table.

9 GB is not a big file for Future. But, it may be a big file if you don't have enough free space on disk.

Is export file size relevant to the issue? Future seemed to choke after the file export failed and the warning was closed.

No idea without knowing all the details. Could be. Just commenting that if you have a 9 GB file to save that's not going to happen with a shapefile.

given that I'm trying to do something that the software is not ready to do yet (or maybe ever),

Don't position it, just report. It's not up to Radian to make it possible for you to save 9 GB to a shapefile, so don't position it as "the software is not ready to do that yet." It's shapefile format that's limited to 2GB, and it is .mdb that limits mfd/mfd to under 2 GB as well. If KML has limits, well, that's KML with limits.

Look, I think it is 100% fair to say that if you command Radian to do something not allowed by the formats you are using to ask that Radian decline gracefully. No dispute about that at all. But don't confuse that with the limitations of the formats because that's not something Radian can change.

That shapefiles and .mdb and KML are what they are is OK, because there is a lot of industry know-how built around those limitations. Nothing wrong with that. If you want Radian to provide you with a format that can handle bigger data, no problem with that either. In Radian's case it is .map, the best format the engineers know how to provide. If you want big data and interoperability with other software in the same format as well, no problem with that either: use something like PostgreSQL. It's free and it can handle 9 GB.

There's nothing wrong with wanting Radian to gracefully decline when it is commanded to do something wrong, or to provide a graceful way to bail out if you command it to do something you regretted (like commanding a very long task). But put the focus on that because there you are in the business of advocating a specific priority ahead of other priorities. Keep in mind there are other users with their priorities who aren't going to have the slightest sympathy if you tried to stuff a 9 GB data set into a shapefile that, by design, can only store 2 GB. They're not going to want Manifold to take time away from their priorities to ensure Radian will decline that gracefully.

My own view is that the software should never advise it doesn't like what you ordered by issuing a series of zillions of error dialogs. I'm with you on that. I just hate that stuff and would prefer that if it was unhappy, or if disk resources were used up or whatever it just didn't do what was ordered and left it up to the user to sort out why. But to track down why it is issuing error messages we need to know precisely what was commanded and all details about the context. That's also the only way to discover if it is a bug, which we should know because bugs get fixed instantly with no issues about prioritizing more graceful error messages or a super cancel ahead of other community wishes.

dchall8
1,008 post(s)
#18-Oct-17 23:13

Good observation. It was still running. It was just in the middle of something. It was probably processing the 20 million points and giving the error message one at a time. That would explain why the other buttons in the program were not working.

It is never my intent to rearrange anyone's priorities by asking a question. If I were to ask you about how long it will take for you to do something to my project, I am not asking you to upgrade my priority. I am asking for an estimate so I can schedule other tasks while the project is being worked by the other person. I do that a lot in my business and always make it clear that's what I'm doing.

What the software (Future) is not ready to do yet is create a surface and 5-foot contour lines. I was not at all clear about where I was headed. I had hoped I could export Z values for one point cloud and create a surface in Manifold 8. I am no longer trying do that. I get it. While I was successful exporting a thousand points through a .kml handshake, lots of the information disappeared in the exchange. I understand that, too.

Thank you for the explanations. I have a better understanding now.

Dimitri


7,413 post(s)
#19-Oct-17 07:42

It was probably processing the 20 million points and giving the error message one at a time. That would explain why the other buttons in the program were not working.

Perhaps, but I hate to guess at such stuff. If Radian is processing internally in background it typically does not pull resources from the GUI for other things. In contrast, if it is waiting on Windows there are some tasks that need to be very heavily firewalled or sandboxed to prevent the use of Windows facilities (which, typically, are not multithreaded) from holding up the works. It's making a deal with the devil: reach out and use standard Windows stuff for all the right reasons (guaranteed compatibility when Windows 10 reboots your system for an upgrade for the tenth time this month, etc....) and when you shake hands with the devil you discover it becomes the devil's decision when to release your hand. No buttons working unless the devil decides he's had enough fun and lets you go.

So here's another scenario: the 9 GB exported to a KML file, because of the huge expansion induced by the KML philosophy of making everything human readable, blossomed into 900 GB. What with temp space and Windows buffers to write that to a file there was not enough free space for Windows to handle that. Radian doesn't write such files one character at a time but tries to do it in reasonably efficient swaths, so with each try (never know when you get lucky with some resources opening up), Windows said "oh, no, Mr. Bill!" and up popped an error message.

Like I say, without details we can't say. I would ask that if it is worth it for you to raise this in a thread it is worth it to send a detailed report to tech support.

It is never my intent to rearrange anyone's priorities by asking a question.

OK. I got the impression you were advocating. :-) Nothing wrong with that. Trust your instincts and if something annoys you, send in a suggestion to have it changed.

What the software (Future) is not ready to do yet is create a surface and 5-foot contour lines.

Oh, it's ready to do that right now, using SQL. You don't even need to use the API, although many would prefer to do that. Compared to all the eagerness to use Python to code functionality for other products one reads in GIS forums that seems to becoming a routine way of life.

I think what you mean is you would prefer a pre-built dialog, like a transform template, that would do that for you. That's coming, but if it is an important function for you I strongly recommend you send in a suggestion for exactly the way you would like it to be. Otherwise, it's going to get done the way other people who are sending in suggestions want it to be.

The progression from Radian to Future to 9 really is community driven, which means that those folks who speak up and send in suggestions have big influence on how the product evolves.

Talk in this forum is critical for refining ideas and getting the benefit of community insight. That is a vital function but it is a very different function than taking the benefit of those insights and actually sending in a vote, a suggestion, a statement of "this is what I want." If there is something you want, send it in.

adamw


10,447 post(s)
#23-Oct-17 17:18

So if there is a question here, how do I export LiDAR points such that I can use them in Manifold 8?

Apologies for being awfully late to this thread.

If the data size is manageable for 8 (based on what you say later - ~700 MB of uncompressed data times 3 - it should perhaps be), I would suggest either (a) exporting data to a regular database, like SQL Server, or (b) keeping data inside the MAP file and connecting to the MAP file from 8 via the ODBC driver for Radian / Future. You can also do (c) keep data unchanged in LAS files, create a MAP file with data sources for these LAS files and connect to these data sources from 8 via the ODBC driver for Radian / Future.

adamw


10,447 post(s)
#23-Oct-17 18:15

Here is another alternative:

Launch Radian / Future. Invoke View - New Command Window - SQL. Link a LAS file. Open the data source for the LAS file. Drag and drop the table from the Project pane into the right-hand list in the query builder.

Type the query: SELECT x, y, z FROM [las data source]::[las table name]

...inserting the name of the table by double-clicking it in the query builder list. The query just restricts the table to the X, Y, Z fields.

Run the query using View - Run. Invoke Edit - Export Results and export a CSV file. This should be fast.

Now launch Manifold 8 and import the CSV file as a surface using XYZ: File - Import - Surface, select CSV file, set import type to XYZ. This will likely take some time, but in the end this should produce the surface you want sans georeference data and without interpolation between the points. (To georeference it correctly, we have to alter the query a bit, the exported values of X, Y, Z are in the units internal to LAS, you would likely want the scaled values. If you want interpolation, use the CSV to create a drawing and interpolate the surface from the drawing.)

Manifold User Community Use Agreement Copyright (C) 2007-2021 Manifold Software Limited. All rights reserved.