georeference.org
Subscribe to this thread
Home - General / All posts - Importing USGS GeoPDFs to Manifold
hydrokiwi

126 post(s)
#18-Sep-11 19:14

USGS publishes their DRG topo maps as GeoPDFs. - used to be Geotiff files.

Is there an easy way to import these into Manifold complete with layers etc.

I see there was a thread about this back in 2009 - any new developments?

Cheers

John

joebocop
217 post(s)
#18-Sep-11 19:33

I'm also interested to hear about any good techniques in this domain. I've been saving the PDFs as TIFs, and then manually georegistering them. Blech!

Joe

hydrokiwi

126 post(s)
#18-Sep-11 19:44

Here is a link with a powerpoint presentation about what the US Army COE is doing in this area.

tec.army.mil/.../GeoPDF/Distro_Army_geo-referenced_pdf_file_project.ppt

Cheers

John

hydrokiwi

126 post(s)
#18-Sep-11 20:00

See below from 2010.

Looks like some serious money is being made from ARCGIS users by TerraGo.

Looks like Global Mapper is the best tool for converting right now.

Any other suggestions?

Question: Can I import a US Topo into my geographic information system (GIS)?

Answer: Yes, but with many caveats.

The US Topo was not intended to be a GIS product. It was designed to serve users who need medium-scale topographic and orthoimage maps, but who are not GIS users.

US Topos are derived from GIS data. Almost all these data are from USGS and other government sources, and most are available for free through web services or file download sites. The US Topo represents a repackaging of these data, not new data creation. The primary design objective was to provide these data in a convenient and familiar form to people who need maps but who are not professional cartographers. The traditional 7.5-minute quadrangle layout and PDF format were selected as the best way to accomplish this objective; the GeoPDF extensions were adopted because they added some cartographic value at no cost to the end user.

This repackaging actually degrades the original data, because information must be filtered and generalized to create a readable quadrangle map. This degradation is justified because it makes the map easier to understand and use for the target audience of non-GIS users. The point is that the US Topo is an output of, not an input to, GIS systems.

Nevertheless, displaying a symbolized quadrangle map in a GIS environment can be useful for many things. Below are some partial solutions to the problem of importing a US Topo into GIS software. The USGS expects that more and better solutions will become available as the US Topo program grows and software developers see opportunities in the product. There are historical analogies to this situation in other USGS products. In 1994 the USGS decided to distribute digital raster graphics (DRGs) in GeoTIFF, even though it was a new and unimplemented format. When the first DRGs were created in 1995 hardly any consumer software, including the major commercial GISs, could take advantage of the geospatial tags. By creating over 50,000 public domain GeoTIFF maps, the USGS helped drive the acceptance and widespread implementation of the format.

Currently known methods for importing US Topo images into software other than PDF viewers:

1. Import to ArcGIS® with TerraGo commercial software

GeoPDFs can be imported directly into ArcMap® using TerraGo Publisher® for ArcGIS software. According to TerraGo (December 2009), this software is the only technology that enables direct import to ArcGIS. GSA price for the software $2,197.65 plus 20% maintenance ($439.53) for a total first year price of $2,637.18.

2. Reformat the GeoPDF as a GeoTIFF using Global Mapper®

Global Mapper has both a free version and a commercial version ($350 for one license). The free version does not include data export functions, so the procedure described below will work only with the commercial version.

Global Mapper 11.xx can read TerraGo GeoPDF files and export them as other formats. If exported as a GeoTIFF with JPEG compression, the file size stays about the same, image quality stays about the same, and the result can be read by most geospatial software. Global Mapper will also clip the collar painlessly, and is one of the easiest ways to reproject a spatial dataset.

A significant shortcoming of this procedure is that all GeoPDF layers are exported as one TIFF image. The ability to select individual layers is lost, and all vector layers are rasterized in a single image plane. On the plus side, this does create a background display that is simple to manipulate while being detailed and visually appealing.

The following procedures were tested with Global Mapper 11.0:

a. Open a GeoPDF in Global Mapper

In version 11.xx, GeoPDF is one of the "supported commonly used types."

Loading a US Topo takes longer than loading other files in Global Mapper, but once loaded, pan and zoom operations are fast.

Notice that Global Mapper aligns the UTM/USNG grid with the display coordinate system. A US Topo displays "straight" in Acrobat, like a framed paper map hanging on a wall, but in Global Mapper it displays "tilted," like a DRG. This is not an error; it is software behavior more suitable for a GIS environment.

b. Optional — remove map collar

WARNING: Removing the map collar discards important metadata, including all coordinate annotation.

Go to Tools / Control Center / Options / Cropping

In the tests I ran, the option to "Automatically Crop Collar" did not work quite right, but the option to "Crop to Manually Specified Lat/Lon Boundary" did, and automatically filled in the correct bounding coordinates. Select this option, click the "Specify Lat/Lon Boundary" button, check the bounding coordinates, and click OK.

c. Change background color

Go to View / Background Color

Select pure white

This eliminates "slivers" along the edges of the exported dataset. The pure white (or any other color) of the map collar can be made transparent in ArcMap®, which is almost as good as clipping the collar, but doesn't discard the information in the collar.

d. Export as a GeoTIFF

Go to File / Export Raster and Elevation Data / Export GeoTIFF

For an output file approximately the same size as the GeoPDF input, select JPEG-in-TIFF as the file type. Not all TIFF readers can read this type, but ArcGIS 9.3 is one that can. 8 Bit Palette and 24 Bit RGB can be read by almost all TIFF readers, but file sizes will be much larger.

The FGDC Metadata file attached to every US Topo is lost in this procedure. Save that file separately from Acrobat or another PDF viewer.

If the above Frequently Asked Questions do not answer your specific issue, contact us at http://nationalmap.gov/ustopo/ustopo_feedback.html

Last update: February 22, 2010

tosborn

224 post(s)
#18-Sep-11 20:51

GDAL recently added the ability to read geopdf files. That means you can use gdal_translate to easily convert geopdf's to geotiffs. The command is:

gdal_translate mygeopdf.pdf mygeotiff.tif

since writing geotiffs is the default, andwhere mygeopdf.pdf is repaced with the name of your pdf file and mygeotiff.tif is replaced with the name you want your geotiff file to have.

Free Windows binaries of GDAL 1.8 can be downloaded from http://www.gisinternals.com/sdk/ .

Tim

Attachments:
gdal.jpg

hydrokiwi

126 post(s)
#18-Sep-11 21:53

Tim:

Thanks - worked like a charm.

I guess it flattens the Geopdf.

The Geopdf appears to have vector contour etc layers which can be switched on and off in the Terrago pdf toolbar but get converted to raster in the GDAL conversion.

Anyway it gets me a good raster background layer for quickie maps.

Thanks

John

hydrokiwi

126 post(s)
#18-Sep-11 22:30

Update:

I notice the imported converted Geotiff file shows UTM zone 11 in the Manifold assign projection dialog (which is correct) but the UTM grid seems to be off.

When I open the GeoPDF in Acrobat with the Terrago plugin the UTM grid is OK.

Looks like manual georegistration of the GeoTiff may still be necessary.

Cheers

John

tosborn

224 post(s)
#19-Sep-11 17:11

Could this be because the geopdf is NAD27 and you are viewing it in Manifold as WGS-84?

hydrokiwi

126 post(s)
#19-Sep-11 17:34

No It was way off - much more that the typical shift.

My best final solution was to upgrade to Global Mapper 13 which does a good job.

The only problem with GM is missing pixels between map sheets with the collars removed.

Cheers

John

snowbank2 post(s)
#19-Sep-11 14:51

Avenza's Map Publisher 8.5 plugin for Adobe Illustrator will also apparently import GeoPDFs with the separate layers of GeoPDF data maintained as separate layers in Illustrator which can be subsequently exported as shapefiles. Hence it might be possible to maintain vector layers such as contours in the GeoPDF as vectors rather being converted to rasters. Whether or not the geoPDFs contain vector attribute information that are maintained through the Avenza process I don't know. It is likely that not all USGS GeoPDFs produced to date were created equally - some may be merely raster data that will not be extractable as theme layers but the few I have seen contain vector layers (hydro, contours, transportation, labels) and a raster layer (an orthophoto). There was a thread on CartoTalk about MapPublishe and GeoPDFs: http://www.cartotalk.com/index.php?showtopic=7286&hl=geopdf

This is an expensive conversion route if you don't have access to current versions of Adobe Illustrator and MapPublisher and it might not lend itself to any sort of batch processing. I am impressed with GeoPDFs for their encapsulated capability for users to customize display by turning layers on and off, panning and zooming, and legibility compared to DRGs but dismayed the USGS has not provided tools to extract the component data layers. It seems like a natural for Safe Software to add this sort of conversion to FME but I don't see any mention of it yet.

hydrokiwi

126 post(s)
#19-Sep-11 17:44

I sent an email to US Topo regarding the conversion issue.

Their detailed response is at the bottom of this post:

Right now I guess it is a priority issue and maybe budget too. I hope that some clever folks will add full functionality to existing software like Manifold.

Despite the fact that various layers can be downloaded from other sources I believe there is a valuable convenience factor here. The fact that ARCGIS folks will pay $2197.65 plus $439.53 for annual maintenance for this functionality gives some idea of a proxy for market value. (see below)

Import to ArcGIS® with TerraGo commercial software

GeoPDFs can be imported directly into ArcMap® using TerraGo Publisher® for ArcGIS software. According to TerraGo (December 2009), this software is the only technology that enables direct import to ArcGIS. GSA price for the software $2,197.65 plus 20% maintenance ($439.53) for a total first year price of $2,637.18.

RESPONSE FROM USGS

Thanks for your comments. We are of course aware of the PDF

conversion/import issue. The primary objective of the US Topo program is

to serve people who need topographic maps but who are not GIS users.

Examples include emergency first responders, wildfire fighters, local law

enforcement, National Guard ground units, some earth scientists and land

managers, and some recreational users. Primary design requirements were a

traditional-looking symbolized map, in a digital file that could be

displayed on any normal office computer without unusual or expensive

software, and that could be printed at correct scale without specialized

expertise or unusual software. In our opinion, PDF is the only format in

general use that satisfies these requirements.

There are at least three different solutions to the problem of making the

data available to GIS and other advanced users:

1) In a sense, these data already are available. With the exception of

the roads layer, US Topo is simply a repackaging of computable and

public-domain GIS data. Anyone is free to retrieve the source data for

any purpose they see fit, including making their own maps. We understand

better than anyone that this is not easy, but the decision (in late 2008)

to design and produce a new quadrangle map series was driven largely by

the belief that we were already serving the advanced GIS user to a much

greater degree than we were serving non-specialist map users.

2) We are hopeful that GIS vendors will implement PDF importers. We had a

similar experience with GeoTIFF in the mid-1990s, when the USGS created

the first large quantities of GeoTIFFs. We caught a lot of flak for this

for a year or two, for the same reason: very little software could read

them. However, the GIS software community did respond, and now few can

remember when GeoTIFF was not a de facto standard.

3) We are working on publishing US Topo data layers in other formats.

Though we recognize this is important, it is not a top priority compared

to improving the current product and meeting production schedules (nor is

it particularly easy). Though we hope to start publishing some layers in

other formats in 2012, we do not yet have a definite schedule.

> I hope this is not a proprietary issue with Terrago etc otherwise

> the public will be paying a fortune for Terrago ARCGIS extensions

> or in waste of time trying to convert maps and reconstitute vector map

layers.

The TerraGo GeoPDF extensions provide some simple GIS functionality "for

free" - no cost to users, and if you aren't interested in georeferencing,

you can still use the product as a plain PDF. PDF and the GeoPDF

extensions are both published formats, and there are no legal barriers to

anyone writing original software to read, create, or manipulate these

files.

-------------------------

Larry Moore

lmoore@usgs.gov

From:

Hydrokiwi@yahoo.com

To:

USTopo@usgs.gov

Cc:

archive_ask@usgs.gov

Date:

09/19/2011 09:03 AM

Subject:

Digital Map Feedback,Use of US Topo Data

Transaction=GSF7JXRC [19SEP2011 15:03:12UTC]

Customer email: Hydrokiwi@yahoo.com

Customer: John Christensen

Subject: Digital Map Feedback,Use of US Topo Data

Originating page: http://nationalmap.gov/ustopo/ustopo_feedback.html

Primary response: USTopo@usgs.gov

USGS PERSONNEL: This email was generated through the Contact USGS system.

When replying to the customer PLEASE BE SURE TO CC archive_ask@usgs.gov.

(Customers, please do not send email to archive_ask, as it will not be

answered.) If you answer by phone, simply forward this email to

archive_ask@usgs.gov. You can see more information about replying to

customers at <http://answers.usgs.gov/usgs/responding.htm> (USGS only).

========================================================

Hi:

I like your new maps.

To maximize the public value of these data there needs to be a conversion

route from the Geopdf to conventional raster and vector layers.

I have used GDAL and Global Mapper but these flatten the data to a single

raster pdf.

I hope this is not a proprietary issue with Terrago etc otherwise the

public will be paying a fortune for Terrago ARCGIS extensions or in waste

of time trying to convert maps and reconstitute vector map layers.

I hope you can provide a no-cost path to use of these important map data.

Simple conversion of Raster layers to Geotiffs or ECW and vector layers to

shp files would be OK.

BTW the documentaion says that USTopo was not intended for GIS use - I

believe this data will get used and could save time in creating simple

maps.

Thanks for listening

John Christensen

joebocop
217 post(s)
#18-Sep-11 22:19

GlobalMapper FTW! Thanks for the help on that.

- EDIT: 64bit GlobalMapper doesn't seem to support this functionality in v13. 32bit v13 works fine.

Joe

Manifold User Community Use Agreement Copyright (C) 2007-2011 Manifold Software Limited. All rights reserved.