Import Drawing - SHP, Shapefiles

ESRI's .shp format, also known as "shape format" or "shapefiles," is used with ArcView, a popular GIS package of the early 1990's. Shape format has been openly published by ESRI and is widely used for data interchange in GIS. Shapefiles usually consist of three similarly named files with differing extensions: a .shp, .shx and a .dbf file. The .dbf file is a dBase database system format file that is used to store data attributes for the drawing. Manifold's importer will import .dbf attributes if present and will import .shp geometry information without data attributes if no .dbf is present.

 

Shapefiles may also include a .prj file that purports to give coordinate system (projection) information. Manifold can extract all information present in a .prj file; however, users are cautioned that .prj is only a partial standard and should not be relied upon to give the same level of complete projection information that is available in more modern GIS formats.

 

images\xbtn_box_select_all.gif

Select All - Check all the boxes in the fields pane.

images\xbtn_box_select_none.gif

Select None - Uncheck all the boxes in the fields pane.

images\xbtn_box_select_invert.gif

Select Inverse - Check all unchecked boxes and uncheck all checked boxes in the fields pane.

Fields

Check fields to be imported.

Import measure

Creates new drawing containing points, each point of which provides a user-defined measure at that location.

Import Z

Create an additional drawing with points that provide a Z coordinate if Z information is present.

 

Several formats in common use allow objects such as lines to be defined not just by X,Y coordinates but also by Z coordinates at each coordinate location that defines the line. Checking the Import Z box causes an additional drawing to be created filled with points, each of which has a Z coordinate taken from the accessory Z values in the coordinate sequences that define lines or other objects. This drawing can then be used to create a surface.

 

Shapefiles and Projections

 

The experienced GIS operator will use shapefiles only with unprojected data. Unfortunately, many GIS users have been tempted into using shapefiles to save data in projected form.

 

Shapefiles were not originally designed to save projection information. Recent extensions to ArcView allow the partial saving of projection information in a supplemental .prj file that accompanies the other shapefile files. However, because the notion of a supplemental .prj file has been around only in recent years, most shapefiles that have been published do not use this extension.

 

A further problem is that some applications use shapefile writing libraries published to Internet that contain errors in how .prj files are written. Although Manifold will read .prj files when they accompany shapefiles and Manifold uses a variety of strategies to deal with invalid .prj data or other flaws, it is still best not to rely upon .prj files.

 

Shapefile format is therefore best used to interchange data from unprojected drawings. When such shape files are imported into Manifold they will appear in Latitude / Longitude projection without any difficulty. When importing unprojected data from shapefiles, the only variable of concern is which datum was used. Normally, the description of the datum used will be found in whatever descriptive commentary accompanies the drawing.

 

When importing an unprojected shapefile keep on the lookout for any information that tells you what datum was used for that shapefile. Importing an unprojected shapefile will by default import the shapefile using Latitude / Longitude projection using the WGS 84 centroid. Such imports will be acceptable for many GIS purposes; however, for full accuracy one should find out what centroid (that is, datum) was used for the shapefile being imported and use the Edit - Assign Projection dialog to specify it for the imported drawing.

 

Be Wary of Projected Shapefiles

 

Serious difficulties can arise when people project drawings and then save the resulting projected data as a shapefile. In such cases there is no guarantee that the author of the shapefile will warn us that the shapefile contains projected data and cannot be interpreted as a Latitude / Longitude map. Worse yet, there is no information contained within the shapefiles that specifies the projection and parameters used. If these are not provided within some readme.txt file or (in some cases) an accompanying .prj file or other accompanying documentation, the shapefiles lose their geographic context.

 

The best first step is to import a shapefile using default settings on the importer. Open the drawing and see what coordinates are reported in the status bar when the mouse passes over the drawing. If they are reasonable latitude and longitude values in the expected range, the shapefile was an unprojected data set using degrees and has been imported correctly.

 

Since shapefiles cannot save datum information it is always wise to take a moment to find out what datum was used to save the data to the shape file and to select it as the correct datum for this drawing in the Edit - Assign Projection dialog. Quite often the authors of unprojected shapefiles will include documentation with the shapefiles that will inform us what datum was used. Regrettably, not all authors are so considerate.

 

The case becomes more difficult if projected shapefiles are in play. Projected shapefiles on import may result in a drawing that provides the expected visual appearance when it is opened. However, if the latitude and longitude values in the status bar are nonsensical numbers such as 48593 degrees the shapefile was a projected file. The task now becomes one of finding out what projection and projection parameters were used. These can then be manually specified using the Edit - Assign Projection dialog.

 

Failing to import a projected drawing from .shp format correctly will lead to unpredictable effects when the drawing is used in a map. If we import a projected drawing from .shp we must tell Manifold the correct projection to use to interpret that data since the format does not save this important information. This is so important that Manifold will nag us to verify the projection of new components to make sure we have reviewed the projection that has been assigned and have confirmed it is the correct projection.

 

Importing a Projected Shapefile

 

We import a projected shapefile by first importing it in the usual way and then using the Assign Projection dialog to assign the correct projection.

 

To import a projected drawing from shapefiles:

 

1. Import the drawing using File - Import - Drawing. Use default settings.

2. Open the drawing.

3. Use the Edit - Assign Projection dialog to specify the projection information that should be used.

 

The Edit - Assign Projection dialog allows us to manually provide the correct projection information for the coordinates properties. Manifold will then be able to make sense of the data just imported. If the projected shapefile included a .prj file there may be partial projection information already in the Current Projection dialog. If so and if the information is accurate, that reduces the amount of information we need manually supply.

 

Do not confuse the use of a chosen projection view in a map window (via Edit - Assign Projection) with the use of Edit - Assign Projection in a drawing window to specify projection information missing from legacy formats. If a projected drawing is imported from shapefiles and we fail to tell Manifold the correct projection to use, that drawing has not yet been correctly imported. Subsequent use of the drawing in maps may cause bizarre effects or lengthy delays as the system attempts to compute a new projection for the map based on fundamentally inaccurate coordinates.

 

Example

 

images\sc_import_shape_01.gif

 

We've imported a shapefile showing the United States into a drawing. When opened, the drawing appears visually as expected.

 

images\sc_import_shape_02.gif

 

However, we notice that the status bar reports impossible latitude and longitude values when the mouse passes over the drawing. This tells us that the shapefile contained projected data. We hunt around in the documentation that accompanied the shape file and read the following in a text file:

 

The US map is in Lambert Conformal Conic centered on -100, 40 with standard parallels at 33 and 45.

 

We are immediately grateful that the author of the map took a moment to tell us this necessary information, albeit in a minimal way. To put this information to work we open the drawing and open the Edit - Assign Projection dialog for the drawing.

 

images\sc_import_shape_03.gif

 

This dialog reports the settings Manifold is using to interpret the numbers in the drawing. We can change it to tell Manifold to use a different interpretation. Although this dialog is the same as those used in the Projection system, it does not change the actual data in the drawing. It simply tells Manifold how to interpret what is in the drawing.

 

In the present instance, the dialog shows that Manifold thinks the data in the shapefile is to be interpreted as Latitude / Longitude data in degrees. We need to change that.

 

images\sc_import_shape_04.gif

 

We begin by choosing Lambert Conformal Conic as the projection from the hierarchy of projections available. We then double-click into each of the values to change it to those specified by the author of the map. "-100" is obviously a longitude since latitudes range between plus or minus 90. This tells us the author centered the map on longitude -100 and latitude 40. We therefore use 40 for the Center Latitude and -100 for the Center Longitude.

 

"Parallel" is a synonym for "latitude." We can guess that the 33 and 45 numbers are intended for use as the 1st Standard Latitude and 2nd Standard Latitude respectively. The illustration above shows the dialog as we enter 33 for the 1st Standard Latitude. Press OK to apply the values.

 

images\sc_import_shape_01.gif

 

After applying the new coordinate properties there is no change in the appearance of the drawing.

 

images\sc_import_shape_05.gif

 

There is a change in the values reported by the status bar. Now, when the mouse passes over the United States the values reported are the expected latitude and longitude values for this part of the world.

 

Comments

 

Suppose the author of a projected shapefile did not see fit to provide documentation on what projection settings were used? This is bad news. Without information on projection and parameters the shapefile loses much of its value in a geographic context. We may be able to salvage it by georegistering it if it includes features that can be matched to a known good drawing. Georegistration can be a tedious process, though, so one is forgiven for cursing the author of a shapefile who chose to publish projected data in a shapefile without also publishing the data necessary to use it.

 

Why isn't Manifold able to know what projection is to be used for a shapefile? That's a limit of shapefile format, not Manifold. Modern formats save projection information. Shape format does not.

 

Database Drivers and Shapefiles

 

The shapefile standard calls for use of ".dbf" files to save database information within shapefiles. Unfortunately, there is no common standard for .dbf in use. .dbf originally was the dBase II database format used in early PCs. It lives on as a living fossil database format in modern times. However, since different companies have "extended" .dbf in various ways the .dbf files accompanying one's shapefiles may be inconsistent from system to system.

 

Original dBase II .dbf format is limited to no more than eight characters for field names beginning with an alphabetic character. .dbf file names are also limited to eight ordinary alphabetic characters plus a three character extension. Various recent versions of .dbf format (such as, for example, .dbf format as used in Microsoft's Visual FoxPro DBMS) may extend the number of characters possible both for field names and for file names.

 

Note that some ESRI products will create and read shapefiles with unusual .dbf file names, even those with spaces in them, depending on the operating system or ESRI product involved. However, in Windows it is a massive violation of .dbf spec to use .dbf file names with spaces in them for .dbf files. Manifold normally uses only the Microsoft standard drivers that are installed in your Windows system; however, shapefile import is an exception because ESRI products are willing to violate the standards set by Microsoft .dbf drivers. To deal with such ESRI departures from Microsoft standards Manifold is equipped with a special Manifold .dbf driver that is used only when importing shapefiles.

 

When used to import shapefiles, the Manifold .dbf driver is willing to accept nearly any naming of fields and files that are used in the .dbf portion of the shapefile. When importing tables from .dbf files in all cases other than the import of shapefiles, Manifold always uses the host .dbf drivers resident on your Windows system.

 

Regional Settings

 

When importing data from text file formats like .csv or from .dbf or shapefiles the Regional Options in the system have to match settings within the file that is being imported. This is because such simple formats are too stupid to understand that in different countries people use different symbols to denote decimal points (that is, either a dot character or a comma). To change Regional Options, do the following:

 

§       Go to the Control Panel and open the Regional Options applet.

§      Go to the Numbers page,

§      Ensure that the Decimal Symbol is set to '.' (dot),

§      Ensure that the List Separator Symbol is set to ',' (comma),

§      Press Apply to apply changes if there were any,

§      Import the .csv file or shapefile desired.

§      After import, restore the original settings of Decimal Symbol and List Separator Symbol if you like.

 

Another option is to invoke the Data Sources applet (hidden within Administrative Tools folder of the Control Panel if you're running Win2K) and create a file DSN for the .csv file you want to import. After creating the DSN configure it to use the comma character as a data separator.

 

Shapefiles and PRJ Files

 

The use of .prj files together with shapefiles to specify projections is a recent development that, unfortunately, has come into play without any definitive standards being published to precisely specify how the .prj should code its contents. Although the use of .prj files works most of the time, it is something of a haphazard system. If a shapefile is accompanied by a .prj file Manifold will analyze the .prj to try to determine what projection and projection parameters are intended for the shapefile.

 

Manifold recognizes over 490 projection types used in .prj files and so will be able to extract the projection information in almost all cases, reading more different types of .prj than almost all other software packages that can read shapefiles. However, because .prj files do not use standardized nomenclature for projections it is possible that in some cases it will not be possible to read the desired projection even if the shapefile is accompanied by a .prj file.

 

Another problem is that some third party applications write flawed .prj files. Manifold contains special code to identify and work around the most common .prj flaws. Manifold will also take reasonable steps to deal with partial or unusual .prj situations. For example, when encountering a .prj file that includes datum information (GEOGCS) but no projection information (PROJCS) Manifold will extract the datum but use the latitude / longitude coordinate system.

 

Despite these special Manifold features to deal with the idiocy occasionally found in .prj files, it is always possible you may encounter a .prj file that Manifold cannot import in a way that captures the coordinate system supposedly in use. manifold.net is always on the lookout for .prj files containing previously-unknown standards to add to our "zoo" of strange examples. If you encounter a shapefile using a .prj that does not import into Manifold correctly, please contact tech support for instructions on how to FTP it to manifold.net for examination.

 

See Also

 

Projections and Legacy Formats for a rant on the hassles involved in using some legacy formats.

Import a Shapefile for a step by step example of importing a projected shapefile and setting the correct datum.