Subscribe to this thread
Home - General / All posts - Creating a single image mosaic in Manifold 9
StanNWT73 post(s)
#06-Apr-18 16:06

I'm wondering how to create a single image mosaic out of ~1950 GeoTIFF DEM files. Each GeoTIFF is in its own sub-directory named from the original zip file that was downloaded from the USGS NED website. Therefore I would also need a way to search a directory and all sub-directories from the main directory to find the (.tif) file and import each and then mosaic into a new single mosaic image in a Manifold project or possibly just mosaic from the source directory trees?

Basically I have a USGS NED folder, within that folder I have two sub folders: a 1 arc-second DEM folder, and a 2 arc-second DEM folder. Within each of those folders, are folders for each downloaded zip file for all the 1 degree x 1 degree DEM tiles. Essentially I used 7-Zip to extra to here, where the zip file name became the folder name and within the each folder is the contents of the zip file.

I have already processed this into a mosaic in ArcGIS 10.5.1 and have a arc binary grid in a File GeoDatabase, but Manifold can't read rasters from a File GeoDatabase.

I might see if I can export a BigTiff, or JPEG2000 image, but I'm not sure of the lossless capabilities of ArcGIS for exports to JPEG2000.

Basically I'm trying to use contour areas or contour lines on this large DEM that I have, to test the capacity and speed of Manifold 9 to process large DEMs

Dimitri


4,980 post(s)
#06-Apr-18 17:21

Each GeoTIFF is in its own sub-directory named from the original zip file that was downloaded from the USGS NED website.

Hmm... I could be wrong about this, but I recall reading USGS talking about the "seamless" nature of the NED, that you could download whatever part of it you wanted as a complete, single, seamless data set. Is that so? If true, perhaps you could download a single large DEM.

How large is the DEM you want to work with?

I have already processed this into a mosaic in ArcGIS 10.5.1 and have a arc binary grid in a File GeoDatabase, but Manifold can't read rasters from a File GeoDatabase.

More precisely, ESRI's SDK for file geodatabases does not support rasters, so all software that uses ESRI's SDK cannot read rasters from file geodatabases. Must be something in the water down in Redlands...

I might see if I can export a BigTiff, or JPEG2000 image, but I'm not sure of the lossless capabilities of ArcGIS for exports to JPEG2000.

Sounds like one option. If you already have the data you need, it sounds like it would be quickest just to write it out into some standard interchange form.

Why can't you just write out the DEM from ArcGIS 10.5.1 into a lossless format? Surely Arc could not be so limited that it cannot write out a DEM? Can it write to a DBMS, like PostgreSQL?

Upcoming 9 builds, by the way, will provide merge capabilities that should help such tasks. But still, messing about with combining almost 2000 files into one seems like taking the long way 'round.

I think I'd go first with just getting an original set of data from USGS that is one big DEM. Or, if you just want a large sample to play with, download it from one of the other national cartographic bureaus (LINZ in New Zealand comes to mind, possibly the EU agencies...) which provide big data.

Basically I'm trying to use contour areas or contour lines on this large DEM that I have, to test the capacity and speed of Manifold 9 to process large DEMs

Always worth a read for adventurers in really big data: see the Performance Tips. :-)

StanNWT73 post(s)
#06-Apr-18 18:02

Hi Dimitri,

The spatial extents of my DEM are: top (83.0016106799), Left (-180), Right (-88.9984386585), Bottom (50.9966324193)

I'm working with a 1 arc-second DEM that is (327603, 115217) in size. Cell size of (0.00027778, 0.00027778), the DEM is 140.61 GB, but stored as an LZ77 compressed grid inside the File GeoDatabase. It's a 32-bit floating point data set. Originally they were 16-bit TIFF DEM tiles.

The USGS NED DEM is only available via download as 1 degree x 1 degree tiles (zipped) in various formats. They do this because they have to restrict the size of a zip file download, probably to ensure their servers don't get overloaded with single file downloads and to ensure that you get your individual files, rather than having a large file that fails due to any number of connection issues between the host server and the destination computer.

NED is now refered to as 3DEP: https://nationalmap.gov/3DEP/3dep_prodserv.html

The viewer where you can select the area and product you want is: https://viewer.nationalmap.gov/basic/?basemap=b1&category=ned,nedsrc&title=3DEP%20View

I had to choose 2 arc-second for bits of Alaska because the coverage in Alaska at the time I downloaded the data (Sept 2016) was incomplete at 1 arc-second and 1 arc second for the rest. The data is in a geographic coordinate system (lat/long) NAD83.

I've attached three JPGs showing the coverage of the 1 arc-second data. The coverage of the fused 1 arc-second data and 2 arc-second data (the 2 arc-second data was resampled before fusing with the 1 arc-second data), and the final image is a 1:100,000 scale zoom in on a portion of the Yukon and Northwest Territories border showing the detail of the DEM. The view is shown with Esri's bilinear visual resampling technique to make a more asthetically pleasing DEM view. I personally hate viewing the nearest neighbour square pixels, unless I want to see the pixel sizes at a certain map scale, but for output I always use bilinear. Note all the maps are using a Lambert Conformal Conic projection. The Yukon zoom is likely a customized LCC variant for a specific meridian.

If you feel that Manifold 9 might lock up or crash with a DEM of this size processed as a whole then perhaps I shouldn't try, but I like pushing software to its limits.

My new workstation specs are as follows:

  • Dual hexacore 3.4 GHz xeon (I'll have 24 cores if you include the hyperthreaded cores).
  • 128 GB DDR4 2666 MHz ECC RAM, expandable to 384 GB down the road.
  • Dual Nvidia Quadro P4000 graphics non-SLI
  • 2TB Samsung 960 Pro m.2 SSD boot drive, pagefile, temp, tmp directories
  • secondary m.2 drive 1TB Dell class 40 opal 2.0 drive, possible 1400M MB/s read 350 MB/s write (which is why I ordered the Samsung drive to replace the built in likley Toshiba m.2 drive as the OS drive, which it now is and I have to swap them out before I get the machine setup for use on my corporate network.
  • Thunderbolt 3 card with 2 ports plus display port
  • dual intel 550x 10GbE card
  • 2 x USB type C ports
  • 8 x USB 3.0 ports

I'm wanting to push this new machine and Manifold with some large tasks. If you remember me talking about processing the NRCan Can Vec 1:50,000 rivers (315 million points from a 15 million record polyline segment file of all the rivers in Canada at 1:50k) This took 4.5 hours roughly to decompose to points. I want to run that again with the new machine, I also want to do some contouring tasks. I want to use Manifold to do work that ArcGIS desktop 10.5.1 / 10.6 or even ArcGIS Pro 2.0 / 2.1 can't do or can't do efficiently in any reasonable time frame.

Attachments:
USGS_NED_1_arc_second_DEM_1_100000_Zoom_Yukon_NWT_Border_Aug16_2017.jpg
USGS_NED_1_arc_second_DEM_extent_of_data_Aug16_2017.jpg
USGS_NED_DEM_30m_Coverage_Oct3_2017.jpg

Dimitri


4,980 post(s)
#06-Apr-18 18:48

They do this because they have to restrict the size of a zip file download,

(!) I get the idea, but it is usually way more efficient for them to have a few big files being downloaded than zillions of little ones. Sigh! But... I do understand how big downloads can interrupt.

I've been trying for several days to get a measly 28 GB file mentioned in this thread: http://www.georeference.org/forum/t142813.9#142866 and it keeps getting interrupted. I think that is because it downloads at only about 450 KB per second, way too slow, so with 12 hours of slow download it keeps getting stuck. After all day I'm up to 18 GB now, so here's hoping...

If you feel that Manifold 9 might lock up or crash with a DEM of this size processed as a whole then perhaps I shouldn't try, but I like pushing software to its limits.

Should be fine with only 140 GB and it should never, ever lock up or crash. Not once, not ever. If it does, that's a bug to be fixed immediately. We have only one level of quality for 9 and that is it *must* be bulletproof. Always.

Take a look at the video at https://www.youtube.com/watch?v=Gvib5LqqE2o which shows Radian popping open 110 GB of images instantly. 110 GB is not 140 GB, but it's still significant.

It's also true that video is just viewing, not doing any processing with the data, but it shows that the framework of the system has no problem holding that much data, which is the foundation of all that comes later. So, depending on the processing you do it could be slow, but hey... that is why we have a community-driven Cutting Edge process. :-) People hammer at it with real-life data and where it needs to go faster we make it faster.

I want to use Manifold to do work that ArcGIS desktop 10.5.1 / 10.6 or even ArcGIS Pro 2.0 / 2.1 can't do or can't do efficiently in any reasonable time frame.

Sounds like a plan! We'll help out any way we can. By the way, I think one of the builds next week is a good candidate to have merge in it. We'll see how that goes, but the ability to merge DEMs into one is not far off. It's a matter of days, not weeks, I think.

StanNWT73 post(s)
#06-Apr-18 19:02

Too bad that 28 GB file couldn't be broken up into 500 MB or even 100 MB parts, 7-Zip and WinZip allow for custom file size increments. That's why the zip files from the USGS portal are only up to roughly 50 MB in size and yes it means a lot of files to download.

I only get about 100 KB/s on average, sometimes up to 400 KB/s with my work computer over the internet. I get lots of time outs downloading large multi-Gigabyte files. My corporate network down to Yellowknife is only 30 Mb, even though we have a fiber link connecting the two. I won't get into the reasons for that.... $$$$...

I know how quickly you fix bugs, e.g. that decompose to points issue I had last fall. We are all so used to software that is flaky and has limits that you must live with, that having a application like Manifold that eats "problem child" data sets for breakfast, is very refreshing.

Dimitri


4,980 post(s)
#07-Apr-18 06:32

I finally got that 28 GB file downloaded... after only five days or so of tries (!). We have very good Internet, fiber, but there are many hops in between our local fiber and a distant server in Australia.

One technology that can help a lot is torrent, which, unfortunately, has acquired a politically toxic aura as an untouchable word due to its endemic misuse in piracy. Those people who use it for legitimate reasons dare not even mention the name.

A good example is MapFactor's free Navigator package, for in-vehicle navigation on phones and tablets. It updates map data (derived from OSM) using torrents. If you have a fast Internet connection it can update gigabytes in minutes. But... MapFactor very carefully avoids the use of the word "torrent" and just discusses "updating" map files.

Having a fast Internet, like fiber, to the home or office doesn't help much when there is non-parallel downloading, which is what you get when you download a big file from a server like USGS. Parallel downloading, like torrent, where you can download pieces of a big file from hundreds of peers is a different matter. It is why civil servants and researchers are taking a week to download a 30 GB file while some teenage kid with a home fiber connection can download a 40 GB pirated copy of the latest Transformers movie in ultra HD resolution in five minutes.

I, for one, have always wanted to put a torrent client and capability built-into Manifold, so that people who wanted to could keep files they got from sharing or were willing to share into a given folder, and thus everybody who had Manifold could have access to such things through peer-to-peer, parallel sharing. Over time we could, as a community, evolve an increasing range of useful basemaps and other data.

After all, much of this stuff is public domain data anyway. Why should each of us individually have to spend a week to download something from USGS when, for the more popular products, it could go faster?

The answer to that, of course, is that what each of us is interested in within the vast collections maintained by organizations like USGS is just a tiny part of the collection and usually is not a shared interest. Therefore, it is unlikely you will have more than one peer for most things... and that takes you back to non-parallel download speed. Alas! But still, it seems like not a bad idea so that whatever is of common interest could be instantly available to all.

tjhb

8,215 post(s)
online
#07-Apr-18 07:52

Another answer of course is provenance, pedigree, data trust.

If I want official data then I probably do not want peer-to-peer data, at least not without the kind of safeguard which we are not yet used to demanding or receiving. All official data perhaps should be encrypted using public-key cryptography—which a product like Manifold could no doubt build right in—no big deal, but so far it hasn’t happened.

[Added: file signing e.g. using SHA-256, might well be just as good—provided it can be made easy for practically all users, without becoming so simple that subterfuge is hard to spot.]

This will only become more critical as spatial data (along with other public data) becomes more widely used. That can only make adulterated data a more attractive means to compromise private and public users alike.

StanNWT73 post(s)
#08-Apr-18 19:25

The problem with the idea of including a torrent client into manifold is that you'd get every IT department in Government blocking manifold and uninstalling it from their computer. IT departments see all computers in the corporate network as belonging to them and not the department or division that paid for the computer. A torrents client wouldn't even get through their firewall if the IT folks new anything about setting up a firewall to begin with.

Dimitri


4,980 post(s)
#09-Apr-18 08:36

A torrents client wouldn't even get through their firewall if the IT folks new anything about setting up a firewall to begin with.

If Manifold used torrent technology to connect only to other Manifold peers we could use whatever we wanted to connect, html for example. Almost all IT people would never know you were sharing files via html. That's why License Server uses html, since IT departments routinely allow html through their firewalls.

If we did this (most people within Manifold are strongly opposed, by the way...) we would do what MapFactor does and not ever use the "torrent" word. We'd call it "Open Data Catalog" or "Maps Online" or similar. After all, when people commit their organization's data to ESRI-hosted storage I bet none of them know that data is actually hosted in Nigeria and nobody has the slightest idea which parts of OSM data they use are hosted in Ukraine.

[Just kidding... my point is that you have no idea where that data is hosted, other than whatever ESRI may tell you, and they won't publish online exactly where your data is stored.]

I understand that people worry about peer to peer sharing in terms of what that means about the authenticity of data. But... at the same time they are eager to take advantage of "mirrors" hosted by organizations that do absolutely zero verification of the personnel who operate their servers. When you download SRTM data from CIAT in Italy instead of USGS (because the Eros data center and other USGS servers are so totally junk slow or offline too much...), you probably didn't think that you are trusting the totally unverified student volunteer who is operating those servers, loading data, completely ignoring whether XY axis order is reversed or preserved and all that other good stuff.

The result? Spend days trying to download less than 100 MB worth of SRTM tiles in absurdly small pieces, because the USGS servers are way too slow and more often offline than online. It's absurd. If those were shared and available via torrent you'd have them quicker than you can read this sentence.

There are ways around this that, of course, guarantee authenticity to experts while at the same time, of course, being totally ignored by 99.999% of humanity. For example, suppose Manifold could download direct from some link instead of using a browser. You could have a blockchain style verification enforced by Manifold that tracked each such download so that thereafter anybody who pulled peer-hosted copies of that data via torrent could get an automatic, unforgeable verification that all those pieces did indeed originate in the same, say, USGS URL and that they were not modified. You could have the option to grab only those data sets which had such trace-back-to-the-root authenticity, ignoring data sets which individual users might construct.

That, at least, would be a step forward to get around the execrably slow and unpleasant experience of downloading direct from, say, USGS, and similar national cartographic bureaus. Even better would be taking advantage of cool data sets which other Manifold users author or create, but that would be a matter of individual choice.

tjhb

8,215 post(s)
online
#09-Apr-18 08:56

Dimitri,

I understand that people worry about peer to peer sharing in terms of what that means about the authenticity of data. But... at the same time they are eager to take advantage of "mirrors" hosted by organizations that do absolutely zero verification of the personnel who operate their servers.

Everybody serious wants and needs verifiable original sources. It's as simple as that.

No one wants anything else, as far as I know.

Above all, there is no point in catering to anything else. That would just be negative.

Dimitri


4,980 post(s)
#09-Apr-18 10:11

I agree with you in principle, but there is an implied contradiction in this...

Everybody serious wants and needs verifiable original sources. It's as simple as that.

No one wants anything else, as far as I know.

Why the "everybody serious" qualifier if "no one" wants anything else?

The reality is that you've put your finger on it with the "everybody serious" qualifier. I agree 100%. Unfortunately, "everybody serious" is not "everybody." The other issue is that within the population of "everybody serious" you have very many people who are very serious and diligent and who want to do right but who do not have the technical skills to know when they are getting verifiable original data and when not.

It is not so easy to tell verifiable original sources in a sea of data that often is posing as original data but which is copied from somewhere else. The desire is simple, I agree, but getting that desire satisfied can be far from simple.

No one wants anything else, as far as I know.

Strongly disagree. 1) Very many people want pretty data that works "good enough" for their purpose. If it works most of the time but is wrong in details that are unlikely to matter, well, that's OK. 2) Data from original sources is often awful and must be re-cast into useful data.

I'm one of those people, by the way, so it's not like I'm trying to criticize others while posing as perfect. I'll drive all the way across Europe using OSM data for street routing when I know perfectly well it is often wrong in key details. But it is "good enough" for those stretches in between phone coverage when you can use Google to navigate. Google, also, is not "original source" but it is well-curated.

And last but not least, why could not communal sharing of data we all download from original sources not be a process which increases access to original sources as opposed to decreasing it? People often turn to unverifiable data of dubious origin just because it is easier and quicker to get than going to the source.

Also, there is this effect: 2) Data from original sources is often awful and must be re-cast into useful data.

SRTM is a good example. If you want data from the original source in a useful form, get ready to send a hard disk to USGS with return postage so they can load it up with a 600+ GB data set and in a couple of weeks return it to you. Otherwise, you have to download SRTM data in tiny chunks and then merge them together into more useful form. That's not much of a processing step, but it does introduce very many opportunities for departures from the original source.

In another example, many people in the OSM community are totally dependent upon extractions into shapefiles that a relative handful of third parties produce. That's not original source data and it's not at all verifiable, but it is an irreplaceable foundation for what they do. Very few people can work with OSM data in its original, source form.

I'm just saying, there's probably a middle ground where having very rapid access to data from trustworthy mirrors or from trustworthy curators and value-added users is probably something many people would like.

Dimitri


4,980 post(s)
#09-Apr-18 10:19

No one wants anything else, as far as I know.

Ooops.... today must be Brain Fade Monday, as I forgot the obvious: what most people want is free data.

dchall8
477 post(s)
#10-Apr-18 16:48

Something else people want is to get paid for doing a bit of coding. Given Manifold's newfound speed, seemingly unlimited scripting ability, and massively parallel processing with a GPGPU, if you add parallel torrenting I would think someone would find a way to mine cryptocurrency using your software. Maybe those people are the non-serious people that Tim is talking about.

joebocop
339 post(s)
#06-Apr-18 19:16

If the "images" on disk are themselves parts of a larger mosaic, does this (python-scriptable) pseudo-code workflow make sense?

  1. get top level folder object, iterate over files collection, import each "tif" into a specific folder within the map file
  2. iterate over each imported component name in the folder to create the text for a new query component which creates a new table component and inserts all records from each of the imported components into that single new table
  3. Create a new image from that hefty new table

Perhaps I am misreading, and also, I haven't tried the above myself, but does that fail to satisfy some requirement?

I'm in Yukon; thought I'd weigh-in to try and help out a fellow north of 60. Happy spring!

StanNWT73 post(s)
#06-Apr-18 19:33

Would it help if you could see that kind of geospatial cartographic products I'm producing?

StanNWT73 post(s)
#10-May-18 19:10

Here's a link to the kind of map products I'm producing and the complexity of them:

http://www.nwtpetroleum.com

Click on GIS Data

Go through the various bullets which are individual themed pages.

StanNWT73 post(s)
#16-May-18 15:47

The 1923 DEM 1 arc-second grids, are actually Esri arc binary grids. if they were all GeoTIFFs it would be easy to just put them all in a single directory and import them, but it's a little more complicated this way and I'm not sure of how to write code that will look through a directory tree for the grids and import them. I am trying to get FME in translate the arc binary grids to GeoTIFFs but it's coming up with errors and I'm trying to work with Safe Software to work through that issue. It's possibly a directory tree + file name length issue? Once I figure that out I can do the same to the 500+ 2 arc-second grids that fill in the holes in Alaska but resampled to 1 arc-second to merge with the true 1 arc-second grids. I have already merged these in ArcGIS, but would like to import the individual grids into Manifold and merge them to test the speed documented in cutting edge threads. My other option is to try and export the full arc binary grid from the file GeoDatabase as a single image format, but it's 141 GB so I think my only option is a JPEG2000 file. BigTIFF isn't supported by FME or ArcGIS, it is supported by ERDAS Imagine which I have but it won't load the arc binary grid from the file GeoDatabase, however I don't think that BigTIFF is supported by Manifold.

adamw

8,061 post(s)
#16-May-18 15:57

9 does support BigTIFF.

Regarding importing a bunch of files in different folders, you could try this: invoke File - Import (or File - Link), go to the root folder that contains all files you want to import, set the search box in the right-top of the dialog to the desired filename mask (eg, *.shp, or, in case of ADF grids, probably something like w001001.adf could work), wait until the dialog finds all files matching the mask in the subfolders, then select the ones you want to import and click Open. That's batch import done using built-in Windows tools.

Dimitri


4,980 post(s)
#16-May-18 18:06

There is an example of that batch import using Windows in the Example: Merge Images topic. In the third illustration of the topic, instead of leaving All Files (*.*) in the file filter box next to the File name box, if you wanted to see only ADF files you would choose ADF Files (*.adf).

I do that sort of thing all the time and I'm always amazed at how fast it goes, but I've never done 500 files at once, so I don't know how that would go. I guess worst case you could copy/paste them into subfolders to bring in 100 or so at a time. :-)

StanNWT73 post(s)
#16-May-18 18:43

Hi Dimitri,

I did see that merge images example in the help topics, it did look pretty simple. If I have an entire single directory tree of TIFFs or JP2 files, or whatever, it would be a piece of cake for the non-scripting savvy, but if you want traverse directories within a folder for all subfolders containing the data source you want that requires scripting or an import system that has a built in "search sub folders" option. This might be a useful thing to have built into Manifold, not to cater to the programmers who prefer command line to a GUI, but to the folks that are more visual, button pushing, menu drop down and select folks?

I am trying an export to a single ERDAS Imagine file from the single arc binary grid in the file GeoDatabase, but I do want to try the full import a massive number of DEM files and merge them in Manifold. I like pushing software to see what it can do. I don't think the Manifold developers are adverse to this as it improves the product. I'm just not a programmer myself.

StanNWT73 post(s)
#16-May-18 18:22

Hi Adam,

Is this invoke File - Import in SQL in a query window or in the File -> Import dialogs... If so that isn't going to work because it wants me to dig down until it finds an .adf file, and I'd have to do that for 1923 separate imports... I'm assuming you're referring to writing code. I've uploaded a short listing of the directory tree, you can see the repetitive nature of the folders within each 1' x 1' DEM grid folder. The simply menu system doesn't appear to allow me to import within subfolders, because it wants a actual file or files at the dialog prompt.

Attachments:
USGS_NED_DEM_ArcBinaryGrid_FolderTree_May16_2018.jpg

tjhb

8,215 post(s)
online
#17-May-18 00:22

Is this invoke File - Import in SQL in a query window or in the File -> Import dialogs... If so that isn't going to work because it wants me to dig down until it finds an .adf file, and I'd have to do that for 1923 separate imports... I'm assuming you're referring to writing code.

Taking a step back and re-reading Adam's post...

Regarding importing a bunch of files in different folders, you could try this: invoke File - Import (or File - Link), go to the root folder that contains all files you want to import, set the search box in the right-top of the dialog to the desired filename mask (eg, *.shp, or, in case of ADF grids, probably something like w001001.adf could work), wait until the dialog finds all files matching the mask in the subfolders, then select the ones you want to import and click Open. That's batch import done using built-in Windows tools.

Use the Windows search box at upper right to filter matching files anywhere in the entire folder tree. (You can use the Manifold filter box at lower right in addition, to define the import type, but this is not usually necessary. It is the upper Windows box that does the work here.)

Once the filtered list is fulling populated ("wait" might mean a few seconds or less on a fast drive), press Ctrl-A to select all matched files for import. (Manifold won't list them all in the filename box; that is normal.) Then OK to import the whole lot.

I wouldn't have thought of this but after Adam has pointed it out it is obvious. Very useful!

Dimitri


4,980 post(s)
#17-May-18 05:54

I see from the .jpg you attached that each ADF file is about 54 MB in size. 1,923 of those would end up being around 104 GB. I've created 60 GB files that merge together all the terrain elevation tiles for Europe, and I've worked with plenty of projects in the 120 GB range for various rasters, so I don't think there would be any issues. It's just dealing with the bureaucratic hassle of a couple of thousand small tiles, each of which probably is accompanied by metadata commentary, etc.

It is situations like this that really call out for using a DBMS that can store each image as a field in a record. Using a file system hierarchy to organize such data makes it so much more difficult. As least you can use tricks like Adam and Tim have noted, and then once the images are in Manifold you can use the filter box. Let us know how it goes!

StanNWT73 post(s)
#22-May-18 16:27

Hi Dimitri,

I have some interesting observations from my tests importing GeoTIFF versions of the arc binary grids. I was able to use FME to translate the data. For the 1 arc-second DEMs I just created GeoTIFFs with no compression and no pyramids. For the 2 arc-second grids, I used LZW compression and set it to create 29 pyramid levels. I think it only made 2 given the extent of each grid which is 1' x 1'. There are 1924 1 arc-second 1' x 1' grids and 503 2 arc-second 1' x 1' grids.

When I imported the 2 arc-second grids it imported 1509 grids, there were two extra files for each grid, which I assume are lower resolution "pyramids" for each grid. I'm going to re-do the 2 arc-second translation without pyramids to compare later. The interesting thing is that with the raster layers in Manifold it took 40.782 seconds to merge 1509 tiles. I believe but didn't save the log pane contents before saving and closing the map file with the 2 arc-second merged DEM that it took around 8600 seconds to import. Now there's really only 503 DEM 1' x 1' tiles. However, there were pyramids created by FME which likely built the extra two TIFFs for each DEM. I will be re-running this import and merge with a separate set of 503 2 arc-second DEM tiles with no pyramids for comparison. The size of the map file without the merge in it is 24 GB, the size of the file with merge in it is 31 GB. It took 125 seconds to save the file.

When I ran the import for the 1 arc-second 1' x 1' DEM tiles, it took around 8600 seconds but again I will re-run the import to confirm as I saved the map file and closed before saving the log pane contents. I did however, save the log contents for the 1 arc-second DEM. It took 4026.494 to merge the 1924 tiles in Manifold. The size of the file without the merge in it is 77 GB.

Would the fact that there were image pyramid tiles for each tile of the 2 arc-second grid account for the 40 seconds vs. 4026 seconds? There were 503 spatially distinct tile areas for the 2 arc-second grids and 1924 spatially distinct tile areas over a much larger spatial extent for the 1 arc-second grids. Also the size of each tile is potentially four times the size since it's 1 arc-second vs. 2 arc-second. But a 100 x increase in the time to merge is interesting.

I am going to take each merged grid and source from the separate map files into a new map file and merge them together. and see how long that takes.

Ultimately I want to test the contouring lines and areas with this large DEM.

tjhb

8,215 post(s)
online
#22-May-18 20:10

When you merged the "2-second" images, what order were they in? For example, if the quarter-resolution images were on top (for each rect), I think you would have got an 8-second result. If the half-resolution images came first, a 4-second result. If the original 2-second tiles came first for each rect, then a full 2-second result.

Lower layers for the same rect would (I think) be ignored, so that in any case, the merge would only involve 503 images (the uppermost for each rect).

Those factors need to be considered when making the timing comparison with merging 1924 1-second images.

What resolution did you actually end up with for the 2-second merge?

Doing the maths, 1924 / 503 is 3.825. Multiplying that by 4, for the case where 2-second images were on top, gives 15.3002 times the number of pixels compared with the 1-second case. While if 8-second images were on top (and my other assumptions are right), then the factor would be 3.825 x 64 = 244.8032.

tjhb

8,215 post(s)
online
#22-May-18 20:27

(If 4-second on top then 3.825 x 8 -> 61.2008.)

tjhb

8,215 post(s)
online
#22-May-18 20:46

Ooops. 4-second -> 30.6004 times.

tjhb

8,215 post(s)
online
#22-May-18 21:16

Oh lordy, right the first time--but written down wrongly. Should have checked twice. Apologies.

So: if 4-second images are on top then the relative number of pixels (compared with the 1-second case) is 3.825 x 16 = 61.2008.

StanNWT73 post(s)
#23-May-18 21:22

After re-generating a separate 2 arc-second set of no pyramid GeoTIFFs it took 34 seconds to merge the 503 images in Manifold.

When I tried to create data sources for the previous merged 1 arc-second and 2 arc-second merged rasters, which were merged in Manifold and saved in separate (.map) files along with their respective raster component tiles (1924) and (1509) respectively, that was fine. But when I tried to copy and paste them into the new map file I used to create the data sources, it seemed to work. However, when I created a new map, the these copied merged rasters showed a red circle with a white exclamation point in them, which means I guess that they were a failed raster copy?

When I tried to use just the merged rasters only as a data source, add them to a new map in a new (.map) file then use the merge rasters (edit -> merge), it looked like the merged worked, but when I went into the style for the newly made raster and tried to click the box for image style which allows you to select channel 0 and it calculates the statistics which allow you to classify the raster, well that caused Manifold to crash with a typical error box that shows up when manifold stops and has to shut down. One caveat which shouldn't matter is that for the 2 arc-second (.map) file with the merged grid, the merged DEM had a stylized grid with hillshading turned on, the 1 arc-second DEM (.map) file didn't. I can't see that mattering at all.

I'm trying to use a source data of the (.map) files for the 2 arc-second without pyramids merge raster and the 1 arc-second DEM merge raster a load both into a new map file and drag them into a new map window. I'll stylize the 1 arc-second raster the same as the 2 arc-second raster and them try and merge them in the new (.map) file. This should work given that the sources are both (.map) files, and the merge result is stored in the new (.map) file, should it not?

Dimitri


4,980 post(s)
#24-May-18 05:44

However, when I created a new map, the these copied merged rasters showed a red circle with a white exclamation point in them, which means I guess that they were a failed raster copy?

No, it means there was a message to read, most likely offering to build intermediate levels. See the Notes at the bottom of the Images topic.

Dimitri


4,980 post(s)
#24-May-18 05:48

well that caused Manifold to crash with a typical error box that shows up when manifold stops and has to shut down.

Crashes are extremely rare. To make them even more rare, it is essential that every instance when one happens gets reported. Please contact tech support immediately and report the crash, exactly the error message, and including, if possible, how to repeat it.

They'll need detailed information on your system, such as what version of Windows you are using, how much free space you have on disk, and so on. I know that is a hassle but it is really important when crashes are rare to grab all info when one does occur.

StanNWT73 post(s)
#24-May-18 17:25

Hi Dimitri,

I've repeated the crash in two different ways. Just this morning after importing two BigTIFF files, one for the 1 arc-second DEM and the other the 2 arc-second DEM, both with LZW compression, the import went fine. 5218 seconds to import the 1 arc-second grid, 1144 seconds to import the 2 arc-second grid. Saved the project, that took 1140 seconds. The 1 arc-second grid as a BigTIFF is 37.5 GB, the 2 arc-second grid is 15.1 GB the saved (.map) file, is 92.7 GB. When I tried to change the style of the 1 arc-second grid so that I could classify it into intervals and create a terrain colour scheme then pick the hillshading, as one does with a DEM, it crashed after a certain amount of time just after clicking the black box under the style menu, the one that launches the style editor.

This is essentially the same crash as I had before but this time on the 1 arc-second grid. It's odd because the 1 arc-second grid which was merged in Manifold from individual 1 arc-second 1' x 1' tiles I could change the style and create hill shading for. The same with the 2 arc-second grid tiles. I will use 7-zip to compress the two (.map) files to upload them to support. I will send in a case about the crash.

The good thing is that these rasters once in Manifold do instantly draw and zooming in and out is instant.

I'm also trying to import the full DEM that I fused together in ArcGIS, which I have exported as a BigTIFF. I want to compare the ArcGIS fused DEM of 1 and 2 arc-second grids, the 2 arc-second was resampled to 1 arc-second in ArcGIS, as well as the two DEMs that are mosaics of their respective tiled data sets, with what Manifold does when it merges the 1 and 2 arc-second data. I want to have a numerical difference surface to create a heat map of the differences if any between the two environments that made the surfaces. I also want to contour the full DEM in Manifold and compare that to ArcGIS Pro. I'm not even going to try and contour that extensive DEM in ArcGIS desktop, even with 64-bit background geoprocessing installed. Plus I still have yet to get my new workstation deployed. I'm trying to test as much as possible on my current machine then retest everything on my new workstation to give myself some performance improvement metrics to show my management team that the new machine was worth the investment.

Attachments:
Manifold_Crash_Screen_Capture_Trying_to_Change_Style_on_DEM_May24_2018.jpg

Dimitri


4,980 post(s)
#25-May-18 05:44

Great! Have you sent a report to tech support? That's the key thing. Give them all those details and - very important - tell them about your machine: what version of Windows you are using, how much free space you have on disk and so on.

The point of that is to try to separate any crashes into a) those that happen when people run out of Windows resources, like no more free disk, and b) those that happen because of an error within Manifold. Manifold should gracefully handle a) as well, of course, but it really helps Engineering to get any such clues.

StanNWT73 post(s)
#25-May-18 16:36

Hi Dimitri,

Well I sent in a tech support bug report just a few minutes ago trying to document as much as possible. Forgot to add to the email that I have a 20TB storage RAID, when my map files are stored and I've got around 16-17 TB free space on it. I've got around 450 GB free space on my windows OS drive, where my pagefile and temp and tmp folders are. I told them about the C drive space, just forgot the storage drive space.

This is the second crash incident I've filed, if you remember the issue of dealing with the 50,000 scale river network for all of Canada last October - December. However, that worked out fine with a fix to Manifold. I'm assuming this will be as successful. I've never encountered a software company fixing their software so quickly and diligently as Manifold does.

Even when Manifold was maxed out and my saves didn't seem to be doing anything on Tuesday morning after leaving the merge running from last Friday, I tried to save the document first thing Tuesday morning and when it didn't seem to be saving I tried to save as later in the morning, then when nothing happened by lunchtime or later in the afternoon I exited Manifold, it actually then quit but I noticed that the resouces where still used by Manifold in the Windows processes visible in the Task Manager. It actually only then saved both the original and "save as" files one after the other then released the resources. This was after I had quit manifold and the application windows had disappeared. I think this is impressive.

StanNWT73 post(s)
#31-May-18 00:32

I was able to contour lInes for the gym 1 arc second DEM which is a BigTIFF export from an Esri File Geodatabase raster. It took 7.88 hrs with a contour internal of 10 m between 0 and 5000 m elevation. I'm making a series of mbx files to upload to support to figure out why the error was occurring. Would you want to have the contoured DEM as well? I'm ultimately trying to compare Esri and Manifold contouring.

Dimitri


4,980 post(s)
#31-May-18 07:14

Would you want to have the contoured DEM as well?

That's a question for tech. Once you launch an interaction with tech, let them take the lead. In general, if they need something they'll ask for it.

StanNWT73 post(s)
#25-Jun-18 17:27

Finally switched workstations to my new beast. I'm on Windows 10 Enterprise x64 now. Got a drop box account to upload the various permutations of the DEM (.mbx) files. I also uploaded the resultant Manifold map file that contains the CanVec rivers and the decomposed points, so that if support wishes they can overlay the 1:50,000 rivers and the vertices onto the 1 arc-second DEM. Not sure if you or Adam can access that stuff once support downloads them or not. However, they would be great files to play around with just to push Manifold in testing on your end. The data is non-proprietary, since they all come from public sources, I've just compiled them.

StanNWT73 post(s)
#25-Jul-18 19:56

I Folks,

I've not received any feedback from support yet. I was away on vacation and got back yesterday. I'm hoping this are resolvable.

I will be trying out the newest edge build soon and will try out the next full release to see if the issues persist, but until support tells me they have been able to reproduce the issues and try and address them or see if there's a fix I don't know if a new edge build or full build will fix it.

Dimitri


4,980 post(s)
#26-Jul-18 03:54

I sent in a tech support bug report ...

I've not received any feedback from support yet.

A request for tech support is a very different thing than a bug report. If you launched a tech support incident you spent one or more tokens and would have gotten a response. If you sent in a bug report, you should not expect any feedback: see the Bug Reports page.

If you sent in a bug report, that's a good thing and no doubt tech is grateful you sent in a bug report. They don't waste any contributions and will diligently hunt down all issues based on bug reports. Although there is no expectation of feedback or reports, they do like to touch base and provide feedback in cases where what was reported ends up being a real bug. But the forum is not the place to get feedback on a bug report. Let them work the process.

If you have any reason to suspect a communications problem, such as spending a support token and not getting a reply, or having a feeling that tech has tried multiple times to get through to you to ask you a question about a bug report, etc., the forum is not the place to resolve that. Instead, carefully read the Email Problems page and get a gmail account that you check directly with google through a browser (and not by polling gmail using some email client that may be causing problems...). Use that gmail account.

tjhb

8,215 post(s)
online
#26-Jul-18 04:14

Does Manifold really want to encourage use of gmail, and have all correspondence, and all related contacts, harvested for marketing and sale by Google? Just don't, gmail is evil.

Dimitri


4,980 post(s)
#26-Jul-18 11:58

There are pluses and minuses to everything. It's not like operating a classic email client/server setup is without risk, as various news reports in recent years have noted.

In a perfect world, there would be no need to ever suggest using any web-based email because people would not have email problems like those listed in the Email Problems page. If you review that list, you'll see it includes various configuration issues that may be difficult for inexpert users to comprehend, let alone resolve.

When looking for "silver bullet" advice that always works, especially for inexpert users, the advice to use "stock" gmail through a web browser always works. It's also a sure-fire solution for harried experts who don't have time to run down whatever new "value added" brainstorm their ISP may have introduced. Name another free web-based email system that always works and seems never to block legitimate emails. It would be great to learn of an alternative.

I would be the first to agree that the Google business model, just like Twit and Fakebook, raises privacy issues that are worth thinking about. But, for now, one positive aspect of Google is that it is a known quantity. People think of gmail as a reasonably safe option that is not some hyper-risky darknet thing they've never heard about.

adamw

8,061 post(s)
#26-Jul-18 12:17

I checked and provided I found the right issue, we have the fix in testing.

It's all resolvable as long as we have enough data. In this particular case, there's absolutely enough data and clear instructions, thanks for that.

StanNWT73 post(s)
#27-Jul-18 01:46

Thanks Adam.

Yeah I definitely provided enough data. Sucks that it took so long for support to download it.

Feel free to use that data to demonstrate or test things. It was all free data that I built into layers that I rutinely use but there are no issues with using. Sourcing url can be provided if necessary if you want to refer to the correct sources in your documentation for anything you would put public links up to.

adamw

8,061 post(s)
#17-May-18 09:26

It's the ordinary File - Import command and it can handle subfolders to an extent - see Tim's post above.

Following from your screen: create a new MAP file, invoke File - Import, navigate to ...\USGS_NED_DEM\DEM_1arc_second_arc_grid, then click the search box in the top right of the dialog, enter 'dblbnd.adf' (could use a different filename as long as there is only one match per subfolder), wait a couple of seconds for the dialog to scan the subfolders and find all files with such a name, select the results, then click Open. The system should import each selected file, you can see the path of each imported file in the log window.

I'd select the first five or so results for a test run just to feel the process. The involved system dialog might have a limitation on the total number of characters in all paths it returns, but if we are talking about importing thousands of files, this is best done in packs of 100 or so anyway (otherwise you run the risk of starting something that will take longer than you can afford to complete, better do it in chunks).

mdsumner


4,205 post(s)
#11-Apr-18 02:15

Consider gdalbuildvrt from GDAL to build a VRT (virtual raster, composed of multiple files/processes), then use Manifold+GDAL to read that and get into native form. It requires having GDAL and using its command line tool, but is very low-resource (on GDAL side at least, not sure how Manifold will go) and fewer steps, so might be useful.

http://www.gdal.org/gdalbuildvrt.html


https://github.com/mdsumner

Manifold User Community Use Agreement Copyright (C) 2007-2017 Manifold Software Limited. All rights reserved.