ASTER DEM HDF and ASTER Satellite HDF import Tutorial

ASTER Satellite HDF and ASTER DEM HDF import Tutorial

____________________________________________________________________________________________________________

Disclaimer (the usual). Try at your own risk. Best practice would be to backup your .hdf / .met source files before you start.

____________________________________________________________________________________________________________

This site concerns how to import Aster DEM files in HDF format into ArcView 3.x or ER Mapper. I have now also added a brief section on how to import normal Aster files (level L1A & L1B) in HDF format into ER Mapper and prepare them for further image processing. Lastly, I have included a section on how to create 3D scenes in ER Mapper by draping Landsat or Aster images over digital elevation models that are created from contours in ArcView.

Data type Costs Source Scene size

ASTER DEM Free EOS NASA c. 62*62 km

ASTER HDF 55 U$/scene EOS NASA c. 62*62 km

Landsat 5 TM Free ESDI c. 170*180 km

Landsat 7 ETM Free ESDI c. 190*190 km

Landsat 5 & 7 Mosaics Free ESDI c. 730*530 km

Various DEMS (some global) Free see Terrain map 30m or 90m or 1km

Online Viewers

Global Visualization Viewer (ASTER, Landsat; for fast availability check and path/row search)

Aster DEM Global coverage Viewer to quickly check if an Aster DEM exists for your area of interest

Global Seamless Data Viewer (SRTM DEM global coverage, MODIS and more, from USGS)

ArcExplorer Web (Global ArcIMS servers, including 1km World Landsat maps, ESRI World base maps, World elevation zones. To save your links, register with ESRI)

Earth Imagery WebECW Viewer that shows Landsat mosaics for the whole world, most are available for free via ESDI. Download the ECW plugin when it asks you.

Earth Imagery Africa Same Viewer but direct link to Mosaic of Africa

Note Users without ER Mapper can still use free utilities to go from ASTER DEM in HDF to a DEM format that ArcView and other programs can use (scroll down to steps 1A to 1C).

Update Geotiff4.exe is now capable of handling southern hemisphere as well as northern hemisphere images

Note ER Mapper 6.3 SP1 can import some HDF DEM’s but seemingly not all (probably applying to level 3 DEM’s).

Update ER Mapper 6.4 imported my level 3 Aster DEM correctly (implying that anyone with 6.4 might be able to skip steps 1A & 1B from now on).

Advantage Using ER Mapper to import HDF files is faster than using Geotiff, and the resulting images will be cleaner. (No border around the scene. This is important when stitching several DEM’s together).

ASTER DEM (HDF) instructions

Short version (For Users of ER Mapper 6.4)

Long Version (using free utilities only)

Introduction

1) DEM: Steps for Northern hemisphere

A) Download ASTER DEM (HDF)

B) Convert HDF to USGS DEM

C) Import USGS DEM into ArcView 3.x

Direct_import

Short version via Global Mapper

2) DEM Steps for Southern_hemisphere

A) Download ASTER DEM (HDF)

B) Export HDF as Geotiff

C) Export Geotiff as USGS DEM

D) Import USGS DEM into ArcView 3.x

Direct_import

Short version via Global Mapper

ASTER Level L1A and L1B Satellite imagery instructions (For ER Mapper 6.x only)

New Creation of 3D scenes in ER Mapper using free global DEM and free Landsat Mosaic data

New Convert free SRTM DEM’s to ArcView for gridding & contouring, with ER Mapper 6.4

New Convert free SRTM DEM’s to ArcView .for gridding and contouring, using 3DEM FREE

Creating 3D Scenes in ER Mapper from ArcView grids or contours

If ArcView refuses to import USGS DEM

How to import a GTOPO30 USGS DEM into ArcView 3.x

Error calling unlink for file: Returned Error Code 13

General guidelines for preventing ArcView project corruption

Useful links

___________________________________________________________________________________________________________

For users of ER Mapper 6.4 (shortened version)

To use ER Mapper, start with the HDF import wizard which you will find under the Batch Processing Toolbar. (Note that if you want to know what level your DEM is, you can find it in the .met file, by opening it in WordPad and searching for ‘level’).

Important To get a Geotiff from your HDF file, with its height information preserved, there are two options in ER Mapper. The first uses ERM to open a file directly and then save to Geotiff. The second imports directly to Geotiff from the HDF wizard and skips the .ers output step. The second option is good if you have many files, as they can be batch processed.

Top…

Direct open

1) In ER Mapper, go to File>Open

2) Select “All supported files” in the File open dialog box

3) By now you should see your source .hdf and .met files

4) Open the .hdf file

(For people with 6.4, all Aster DEM HDF formats should now open directly)

(For people with 6.3, only level 2 DEM HDF will probably open)

5) Save the resulting single band, pseudolayer image as “Geotiff”

6) In the “Save as” dialog, set the output type to “Multilayer”, and “Signed16bit”, and select the “Delete output transforms” box. Also go to “options” and select “Save Geotiff information”

7) Save the file.

Note: Now you have two options, one is to continue in ER Mapper, the other is to go to Global mapper.

ERM 6.4

8) Reopen the .tif, and open the algorithm window

9) Add intensity layer (Edit>Add raster layer>Intensity)

10) Change colour table of your pseudolayer to pseudocolor, under the “Surface” tab

11) Turn on sun shading on the intensity layer and rename the intensity layer to ‘Shading’. Clip the surface, and save your now colourful looking DEM as an algorithm

Global Mapper

(If you reopen the .tif in ER Mapper, it will still look like a single pseudolayer unless you add an intensity layer with the .tif in the same surface, but if you reopen in Global Mapper, it will contain the height information of a DEM)

12) Open tif in Global Mapper and click “yes” when Global Mapper asks if it is a DEM

13) Export as USGS DEM (only with licensed Global mapped)

14) See step 1C if you want to go on to ArcView

HDF Import wizard

In ER Mapper, go to Toolbars>Batch processing>HDF Wizard

1) Choose: Import one file>Next

2) Select your .met file

3) Tick the “Produce .ers raster image” box

4) Click the folder icon, and select output directory

5) Write file name and select output as “Geotiff”

6) Make sure the extension becomes .tif, otherwise manually correct this

7) Continue the wizard… (select 30m cell size for an Aster DEM)

8) When import is finished, you can open this Geotiff in Global Mapper for example

9) When Global Mapper asks for the characteristics, click “Yes”, this will force it to read the height data

10) Export as USGS DEM (only with licensed Global mapper)

11) See step 1C if you want to go on to ArcView

Note: If the HDF import wizard crashes, producing the error: …….met “is not an hdf file, or is not a supported product”

try changing your .met extension into .hdf.met, so for example ‘Aster1203090.met’ becomes ‘Aster1203090.hdf.met’

Top…

Advanced If you have multiple ASTER HDF images that you want to stitch together, use Global Mapper first before going to ArcView. Note that if you have ER Mapper 6.4, it is worth redoing the import of scenes previously imported with Geotiff, to clean up pixels in the scene border. Otherwise, contours will follow the scene boundaries rather than cross them. Then continue with the following steps.

12) In Global Mapper, Choose “File> Open into current” to add more images into the same view

13) Export the combined images as one DEM

14) Continue with step 1C, i.e.,

15) Load ArcView

16) Load Demshift.ave, run the script

17) Let the script load the combined DEM

18) Let the script export to a grid and add it

19) Activate 3D analyst and Spatial analyst

20) Create contours of your DEM (your created grid)

21) Decrease the delay in contour display by removing contour intervals you do not often use from the legend (but keep them in the file so you do not need to repeat this procedure if you change your mind)

Alternatively, skip step 14-20 and export contours as a shapefile from within Global Mapper

Top

_______________________________________________________________________________________________

Introduction

There are two methods to get Aster DEM's that do not automatically import correctly into ER Mapper (pre 6.4) and ArcView (3.x with Spatial Analyst), one works for the Northern hemisphere, the other for both hemispheres, but the latter is more lengthy and needs a licensed version of Global Mapper. Method 1 uses free utilities only. If anyone knows how to shorten the procedure below, please email me at muze7…at...westnet.com.au. This also applies if you know how to import a .dem file into ER Mapper, as the USGS import utility crashes in my ER Mapper 6.3 SP1.

Update: According to the maker of Geotiff4 (see http://www.terrainmap.com/rm22.html#ASTER), Geotiff4 is now updated so it can handle southern hemisphere images. I have not tried it yet. I will leave the info up on this site for reference however.

Update2: John Childs from Terrainmap.com also advices that a new utility ‘MicroDem’ might be a very powerful replacement program for Geotiff4. See: http://www.usna.edu/Users/oceano/pguth/website/microdem.htm. Note: I have not tested this yet, as the full install is 30Mb, but it looks good.

Update3: MicroDem seems to only handle Aster DEM’s in Geotiff format?? This would mean that Geotiff4.exe is still needed.

Top

____________________________________________________________________________________________________________

DEM: Steps for Northern hemisphere (1)

Top

1A) Download DEM’s (or normal Aster or Landsat TM, ETM etc.)

Go to the EOS NASA gateway for satellite images, and find and download your ASTER DEM's. So far, ASTER DEM’s are still free. For quick selection of Aster L1B level satellite images rather than DEM’s use the Global Visualization Viewer at the USGS. The viewer will show L1B images only. DEM’s are derived from L1A images.

Update: You can now also use this viewer to quickly display coverage for Landsat data, and any image you are interested in can now be ordered directly from the viewer. This is generally much faster than using the main EOS NASA gateway.

Costs:

ASTER DEM Free EOS NASA

ASTER HDF 55 USD / scene EOS NASA **

Landsat 5 TM & 7 ETM Free ESDI

** You can also apply to NASA to get an educational user status so all other type ASTER images will be free to you if approved. Note however this took four months in my case.

Once ordered, you will receive an email with the FTP address for downloading your Aster DEM’s via FTP pull. The download email usually takes about half an hour to be generated.

The DEM will come in an .hdf file and .met file. The .met file is the data header. Keep both. If you have an older version than ER Mapper 6.3 Service Pack1, be careful when trying to open the .met file with the HDF import wizard because it might disappear. So make a backup directory first. In any case, the HDF import wizard does not work for me yet on my DEM's. It should in the future though, for all DEM types, perhaps when ERM 6.4 comes out.

Note: You can also choose to search for level 2 (or 3) DEM’s only

Update1: ER Mapper has released a patch to fix this bug where the .met file disappears

Update2: ER Mapper 6.4 seems to have no problem with my level 3 DEM’s

Top…………

1B) Convert the generate .hdf format into a USGS .dem. We will use Geotiff4 for this.

Go to http://www.terrainmap.com/, an excellent site with great GIS tutorials by the way, and scroll down until you see GEOTIFF4. This great and free program will do the conversion for you with a few user inputs. (See http://www.terrainmap.com/rm22.html#ASTER for full story). Quick instructions from the site are:

"The program will prompt you for the grid spacing, a corner tie point, and the UTM zone. (Since ASTER images all seem to be oriented north and south, GEOTIFF only requires one tie point.) This data is available in the metadata (.met) file. The latitude and longitude are given in decimal degrees in the metafile. Look for the "GRING" data object. The southwest coordinates of the extent of the image should be the last pair of coordinates, as they start with the northwest corner and proceed clockwise.

You will need to convert to decimal degrees referencing the WGS84 ellipsoid. A handy UTM conversion applet will aid this task (http://www.cellspark.com/UTM.html)." The tool also outputs the UTM zone. Note that even though this applet correctly converts southern latitudes, Geotiff will still output a NUTM zone. Thus, you will need a different approach, outlined in step 2, for southern hemisphere images.

"The vertical datum flag in the output ASCII file will be set to local mean sea level and the horizontal datum to WGS 84 by GEOTIFF4 so that hopefully, everything will match up correctly.” Make sure you enter the maximum row and column numbers while saving. Otherwise you will end up with only part of your image. The output DEM will be in the NUTM projection for the specified zone.

After you have done the conversion, and exported as ASCII DEM, you can import the file into virtually any GIS application." (Note by author of this site: I was not able to get ER Mapper to import this DEM though). You can use another free tool, 3dem, from http://www.visualizationsoftware.com/3dem/downloads.html, to import your DEM to do other useful things with.

Top…………

1C) Import .dem into ArcView GIS 3.x

ArcView 3.x has an option File: import data source: USGS Dem (only with Spatial Analyst). However, the export in Geotiff4.exe and 3Dem will be in UTM/WGS84. ArcView will not be able to convert the grid automatically to Geographic. If like me, you want to use your DEM in WGS84/geodetic projection, you have to use an ArcView script (http://arcscripts.esri.com/details.asp?dbid=10711) to convert the grid to Geographic. If you prefer the UTM projection follow these steps.

A) Direct Import for UTM

1. In ArcView, go to File>Import data source>USGS DEM

2. If the import fails, see the Note2 below

3. Output the grid and add to your view as a theme

Otherwise, download the script mentioned above and then follow these steps:

B) Use of Demshft.ave for Geographic

1. Go to Window>your project>Scripts>New

2. Go to Script>Load text file and navigate to where you downloaded the script (the standard directory is x:\esri\ AV_GIS30\ArcView\samples\scripts)

3. Load demshft.ave

4. Go to Script>Compile, then Script\run

5. Load your DEM and let the script output a Grid

6. If the script crashes after creating the grid, just remember where you put it and add it manually, because it did work. (It should be in your working directory). Now you are at the same stage as at 3. above but with a geographic projection

Notes a) To save a grid permanently, go to Theme>Save dataset (to move or delete a grid, only use

the Manage Data Sources option under “File”, not Windows Explorer)

b) For more background on DEM’s, Grids, and the demshft.ave script, go to:

http://www.okanogan1.com/natural/ecology/AVHelp/DEM/using_dem_data.html or

http://web.mit.edu/gis/www/arcgis_dem/

Top…………

After Method A) or B), continue by generating contours from the new grid.

C) Short version via Global Mapper

If you have a licensed version of Global Mapper, you could simplify the above steps. What you would then do is the following:

1. Open the Geotiff or DEM created by Geotiff4.exe

2. Go to Tools>Configure>Projection and change from UTM to Geographic, and select “Arc Degrees” rather than seconds

3. Apply and OK; your projection has now changed to Geographic (=Geodetic in ER Mapper)

4. Export as a DEM, and use ArcView to create contours, or

5. Create contours with Global Mapper and export them as an ArcView shapefile

After A), B) or C), you could convert the theme into a 3D contours shape file, then into a TIN. In 3D Viewer, you could now drape for example Aster satellite images over your ASTER DEM. (This assumes you have the Spatial Analyst extension installed and activated). See http://www.esri.com/news/arcuser/0101/avfiles/lesson10.pdf (last section).

Note: If the resulting contours do not perfectly match ground control points of existing themes, the contour (or grid) can be shifted in the x and y directions with an ArcView extension, called ArcView Utility Tools, v. 1.03, found at

http://www.dnr.state.mn.us/mis/gis/tools/arcview/extensions/tools/tools.html. Install the extension, and find the “Shift Coordinates” icon in the new toolbar. This is all explained in the PDF that accompanies the extension.

Top…………

____________________________________________________________________________________________________________

Note

If ArcView 3.x refuses to import your USGS DEM and you get an error like:

“ Syntax error at or near symbol )) “

Note that both input and output directories should not:

- Contain spaces

- Be more than 8 characters long

- Contain any other characters than letters

- Your Windows and USER TEMP directories should conform to these rules too

- Your filenames should conform to those (DOS) rules also

- Otherwise it could be: full or corrupt Windows temp or ArcView working directory. Clean your TEMP file and import the

DEM to a new directory

If it still does not work, try and put both input DEM and output GRID into same directory (C:\temp worked for me)

Top…………

______________________________________________________________________________________________

DEM: Import for Southern hemisphere (2) (need licensed Global Mapper)

Top…………

2A) Download DEM, as mentioned above under Download_DEM (step 1A).

2B) Export HDF as Geotiff using HegWin

Do step A as mentioned above under 1A. Then download an .hdf reader utility called heg_win (free) from http://hdfeos.gsfc.nasa.gov/hdfeos/softwarelist.cfm. Get the latest version (Heg v. 1.0 at present, 2003 date). Before you install it, you need Java SDK 1.2.1 at least (free from Java website) and install that first. Read the readme file that comes with the download of Heg v. 1.0 for correct installation of the HegWin tool.

Note: The Java kit you need is the developers’ version. Find it at Java

Then import your .hdf file and export as Geotiff (select the band, name the output file and run). For more detailed instructions see the NASA guide.

Note: For some reason, ER Mapper and HegWin do not like being open at the same time, thus close ERM first.

2C) Export Geotiff as USGS DEM using Global Mapper

Then, if you have the licensed version or special trial version of Global Mapper, http://www.globalmapper.com/download.html, import the Geotiff. By now, your image will be in WGS84/Geodetic datum/projection with decimal degrees. However, you cannot export as USGS DEM until you change the horizontal units. Go to tools/configure/projection and change the units to arc seconds rather than arc degrees. Then export the USGS DEM. Now, you can import this into ArcView but your units will be arc seconds. If you rather use decimal degrees, follow these instructions:

Top…

2D) Import USGS DEM into ArcView 3.x

A) Direct Import for UTM

1. In ArcView, go to File>Import data source>USGS DEM

2. If the import fails, see the Note2

3. Output the grid and add to your view as a theme

Otherwise, download the script mentioned above and then follow these steps:

B) Use of Demshft.ave for Geographic

Go to Window>your project>Scripts>New

1. Go to Script>Load text file and navigate to where you downloaded the script (the standard directory is x:\esri\ AV_GIS30\ArcView\samples\scripts)

2. Load demshft.ave

3. Go to Script>Compile, then Script\run

4. Load your DEM and let the script output a Grid

5. If the script crashes after creating the grid, just remember where you put it and add it manually

Now you are at the same stage as at (3) above but with a geographic projection

Top…

Notes a) To save a grid permanently, go to Theme>Save dataset (to move or delete a grid, only use

the Manage Data Sources option under “File”, not Windows Explorer)

b) For more background on DEM’s, Grids, and the demshft.ave script, go to:

http://www.okanogan1.com/natural/ecology/AVHelp/DEM/using_dem_data.html or

http://web.mit.edu/gis/www/arcgis_dem/

After Method A) or B), continue by generating contours from the new grid.

C) Short version via Global Mapper

If you have a licensed version of Global Mapper, you could simplify the above steps. What you would then do is the following:

6. Open the Geotiff or DEM created by Geotiff4.exe

7. Go to Tools>Configure>Projection and change from UTM to Geographic, and select “Arc Degrees” rather than seconds

8. Apply and OK; your projection has now changed to Geographic (=Geodetic in ER Mapper)

9. Export as a DEM, and use ArcView to create contours, or

10. Create contours with Global Mapper and export them as an ArcView shapefile

Top…

Note 3: If the resulting contours do not perfectly match ground control points of existing themes, the contour (or grid) can be shifted in the x and y directions with an ArcView extension, called ArcView Utility Tools, v. 1.03, found at

____________________________________________________________________________________________________________

ASTER Level L1A and L1B Satellite imagery instructions (For ER Mapper 6.x only)

Introduction

Your files will come in a pair of .hdf and .met files for each image. At present, Aster Level 1A, version 2 & 3 will need manual georeferencing after importing into ER Mapper, but level 1B, version 2 & 3 will import directly as georeferenced files. (Note that if you want to know what level your Aster image is, you can find it in the .met file, by opening it in WordPad and searching for ‘level’).

An ASTER HDF satellite imagery file comes in three groups of bands, VNIR, SWIR and TIR, of 15m, 30m, and 90m cell sizes respectively. You will need to import each group separately, but if you have a group of files, you can batch process each band group for multiple scenes in one go.

For level L1A and L1B, start with the HDF import wizard which you will find under the Batch Processing Toolbar.

Level L1A Import

1) Choose: Import one file>Next

2) Select your .met file

3) Tick the “Produce .ers raster image” box

4) Click the folder icon, and select output directory

5) Write file name (e.g. …VNIR) and select output as e.g. “.ers”, press next

6) Set ‘Null cell value’ to zero, tick the ‘Rotate image to true North’ box

7) Set resampling to ‘nearest’

8) Set appropriate cell size (see above)

9) Input bands you want to import (1-3 for VNIR, 5-10 for SWIR and 11-15 for TIR)

10) Repeat the above steps but then select the next group of bands and adjust your cell size accordingly

Top

When the import is finished, you can open the resulting .ers file in ER Mapper. Your image will now be in WGS84 datum and local projection. The latter just means your data is not georeferenced properly. In order to georeference each group of bands, you will need to manually geocode them, using the four corner points specified in the .met file. Look for the GRING object. Then continue as follows:

11) Go to Process>Geocoding Wizard

12) Select input file you just made (e.g., 702_43_VNIR.ers) and select Polynomial, go to 2) of the wizard

13) Choose linear (because we will be using only four known points), go to 3)

14) Choose WGS84/NUTM (or SUTM) projection and coordinate type info, go to 4)

15) Now open your .met file in WordPad and go to GRING object to have the coordinates at hand

16) Add three Ground control points (GCP’s) with the + sign in the wizard

17) Select point 1

18) On the algorithm image window, move to the top left corner, select the pointer symbol if it is not already selected, and click on the pixel to position GCP point 1. Note you should not select the black edge corner but the corner of your data

19) In the wizard, stand on point 2

20) In the image select the top right corner data edge

21) Repeat for point 3 but select the lower RIGHT hand corner

22) Repeat for point 4, now select lower LEFT hand corner

23) In wizard, move back to point 1 and enter the first listed coordinates from the .met file

24) Repeat, in order!, for the coordinates from the .met file for point 2 to 4

25) Click save button to save your four GCP’s in a file. Also check your RMS error is smaller than 4. If it is larger, then you have not ordered the points correctly

26) Specify an output name, e.g. 702_43_VNIR_GC.ers and rectify the file

27) Repeat this procedure for the SWIR and TIR files, but in this case, use your saved GCP file, as the coordinates for each corner point will be the same. After you load the GCP file, just adjust the position of the four points on your screen. If you cannot find points 2 to 4, zoom out until you see them.

28) Test your three files by adding them all into the same surface, and turning some of each on and off, to check for correct overlap

If you need any of your three image .ers files in another datum/projection, you need to go to Process>Geocoding Wizard>Map to map reprojection. In any case, you will now have three geocoded files, e.g. 702_43_VNIR_GC.ers, 702_43_SWIR_GC.ers and 702_43_TIR_GC.ers. Note that ER Mapper might default to a band combination of 123 for red-green-blue rather than 321 when you open the VNIR file. Thus, before you clip the image to get a true colour image (well, as close as you can get with ASTER), make sure that the bands are displayed as 3 (red), 2 (green) and 1 (blue).

The three files will each have a different cell size which is an advantage when you go on to further processing, as you will not have wasted hard disk space by making all bands 15m in size, nor losing resolution on the VNIR by making them all 30m for example.

You can process these further, e.g., apply noise reduction, by going to View>Toolbars>Aster>Aster processing wizard.

Top

Level L1B Import

1) Choose: Import one file>Next

2) Select your .met file

3) Tick the “Produce .ers raster image” box

4) Click the folder icon, and select output directory

5) Write file name (e.g. …VNIR…) and select output as e.g. “ .ers”, press next

6) Set ‘Null cell value’ to zero, tick the ‘Rotate image to true North’ box

7) Set resampling to ‘nearest’

8) Set appropriate cell size (see above)

9) Input bands you want to import (1-3 for VNIR, 5-10 for SWIR and 11-15 for TIR)

10) Repeat the above steps but then select the next group of bands and adjust your cell size accordingly

When the import is finished, you will have three files, e.g. 702_43_VNIR.ers, 702_43_SWIR.ers and 702_43_TIR.ers. You can open these in ER Mapper. Note that ER Mapper might default to a band combination of 123 for red-green-blue rather than 321 when you open the VNIR file. Thus, before you clip the image to get a true colour image (well, as close as you can get with ASTER), make sure that the bands are displayed as 3 (red), 2 (green) and 1 (blue).

Your image will now be in WGS84 datum and UTM projection (NUTM or SUTM depending on hemisphere) and have correct coordinates (go to View>Geoposition>Extents to check). If you need the image in another datum/projection, you need to go to Process>Geocoding Wizard>Map to map reprojection.

The three files will each have a different cell size which is an advantage when you go on to further processing, as you will not have wasted hard disk space by making all bands 15m in size, nor have lost resolution by making them all 30m for example.

You can process these further, e.g. apply noise reduction, by going to View>Toolbars>Aster>Aster processing wizard.

Note: If the HDF import wizard crashes, producing the error: …….met “is not an hdf file, or is not a supported product” try changing your .met extension into .hdf.met, so for example ‘Aster1203090.met’ becomes ‘Aster1203090.hdf.met’

Top...

Creation of large scale 3D scenes using free global DEM and free Landsat Mosaics (with ER Mapper 6.4 and Global Mapper), without need for manual georeferencing

These are at 90m resolution, 10 times better than the old GTOPO30, but worse than 15m ASTER DEMs, but the advantage compared to ASTER is they are much easier and faster to process, thus suitable for regional studies. Do not use these as a definitive source for the US, as higher resolution data is available in the form of NED or SDTS formats, although there are 30m SRTM available for the US as well. These 30m SRTMs also exist for other continents but they are not publicly available see NASA.

Note: SRTM’s come in Geodetic projection and WGS84 datum, which is perfect for ArcView 3.x

Note: The method outlined below will keep your georeferencing information automatically preserved, which will save you a lot of time. The steps below assume however that you have ER Mapper 6.4. If you find your ER Mapper version has no SRTM import wizard, you can still follow these steps, but at step 4 of section 1 you need to import HGT files via:

Utilities>Import ASCII and binary grids> Binary BSQ> Import, enter Geodetic projection and WGS84 datum, and under ‘setup’ select ‘signed 16-bit integer’, put in 1201 for rows and columns, 1 for nr of bands and put in 32,768 for the Null cell value, and ‘Motorola’ as bite order. Unfortunately, this method does not preserve the geocoding information, so you would manually need to edit the .ers header file.

Note: More information on this and a tutorial on how to patch holes in SRTM data and edit .ers headers can be found at: Yale SRTM tutorial.

Section 1: download and process SRTM DEM (in HGT format) to get ER Mapper DEM file

1. Download free new higher resolution SRTM global DEM data from http://edcftp.cr.usgs.gov/pub/data/srtm/, now including Africa! This site is fastest if you know the coordinates of the tiles you need, listed for the SW corner point. Otherwise, go to the interactive Global Seamless Data Viewer

2. Download all the tiles you need, you can merge them if you like, as explained below

3. Decompress the zip files

4. Go to ER Mapper, toolbars, batch processing, and click the white tool button, this is an import wizard for the SRTM .HGT files

5. (batch) process your SRTM files, by letting the wizard create an .ers header

6. Open the ers file, change the colour table for the pseudolayer to pseudocolors

7. Add an intensity layer in same surface, turn on sun shading

8. Clip the image, you should now have a colourful DEM

9. If you have batched processes several images, you can add the other tiles by copying and pasting a surface with the pseudo and intensity layer once, selecting the new file and double clicking it. I found this works better than the image display and mosaic wizard, as that led to uneven colour schemes in adjacent tiles, and some manual work is involved in any case, as you would have to add the intensity layers for each tile if you use the wizard.

10. If you want to save this as one combined algorithm, save as .alg, if you want a merged image for future processing, like contour creation, get rid of the intensity layers, move all pseudolayers into the same surface, then save as .ers file and set it to ‘delete output transforms’

Top...

Back to importing SRTM into ArcView

Section 2: download and process Landsat Mosaic into ER Mapper image file

1. Go to ESDI, select map search, select ‘TM Mosaic’ box in lower left corner.

Note: Do not select the ETM box as these files are 220Mb compressed, which translates to about 1.4 Gb in Geotiff format, which you would need to decompress to later, this requires 4 hours of processing on a ~2.5 GHz computer. The TM files are 20-40 Mb, which is easier to deal with, and not much worse for regional studies in terms of quality

2. Download the tiles you want, these will be in MRSID format

Note: MRSID is natively supported by ArcView using the MRSID extension, but these ESDI files are all in UTM format. If you prefer the Geodetic projection for further usage of the image files in ArcView for example, you need to reproject these.

Note: For further processing in ER Mapper, you need to convert these files to Geotiff, then recompress to ECW format

3. Use Global Mapper to reproject the UTM into Geodetic projection, then export as Geotiff (this can all be done by batch processing if necessary)

4. Use ER Mapper to compress Geotiff into ECW, use a 25 or 30 times compression factor (this can all be done by batch processing as well)

Note: File size in ECW format will be roughly 3 times the MRSID format, but I found higher compression factors than 30 degrade the original image too much for my liking

Section 3: Integrate your DEM with your image to get a 3D drape in ER Mapper

11. Open the .ers file of the DEM you have made and the tile that you want in your 3D scene (or your merged .ers file)

12. Add an intensity layer in a new surface, using the same file, turn on sun shading, clip the band

13. Remove the pseudolayer surface

14. File>Open into new surface the image you want to drape, which would be the .ECW image that you made from the MRSID file, which should be correctly georeferenced

Verify the DEM and your image overlap exactly. An easy way to do this is to use the transparency slider under the “Surface” tab. This is why your grid and your image are in different surfaces; otherwise all bands will become transparent.

16. Now move the intensity layer into the surface with your image (deleting the old one)

17. Go to 3D perspective (in algorithm window). Enjoy!

Quick shortcuts for moving around in 3D are:

Left mouse: move image around horizontal axis
Right mouse: zoom in or out
Left + right mouse: move image
Right + left mouse: rotate image around vertical axis

Top...

Convert SRTM DEM’s to ArcView .BIL for gridding & contouring, with ER Mapper 6.4 not Free

1. See Step 1: download and process SRTM DEM to get an .ers file from your DEM

2. In ER Mapper 6.4 go to Utilities>Export Raster>Arc/Info BIL, select your file and export. This will create a correct .hdr header file

3. Add your BIL file to ArcView, create a grid, add grid, create contours

This process will generate contours with incorrect values. I have not yet found a solution to this.

Note: If your .BIL seems empty, open the .hdr header and change byte order from M to I (Motorola

to IBM), save and add the .BIL again.

Note: This method preserves all geocoding information

Note: SRTM’s come in Geodetic projection and WGS84 datum, which is perfect for ArcView 3.x

Top...

Convert SRTM DEM’s to ArcView .BIL for gridding and contouring, using 3DEM Free

Step 1: Downloads

1. You will need one ESRI straightforward script: Demshift.ave. Download this at ESRI Arcscript direct link.

2. Download 3DEM version 18.8 or higher, and install 3DEM

3. Download the SRTM DEM files from http://edcftp.cr.usgs.gov/pub/data/srtm/, now including Africa! This site is fastest if you know the coordinates of the tiles you need, listed for the SW corner point. Otherwise, go to the interactive Global Seamless Data Viewer

Step 2: Processing in 3DEM

Decompress the zip files
In 3DEM, go to File>Load Terrain Model>SRTM, and select your file
If there are holes, go to Operation>Patch missing data
Export with File>USGS DEM

Note: If you were to import this with File>Import Data Source>USGS DEM, the file would not be correctly georeferenced yet. The source file is in Geodetic projection, but you need to apply a script to actually move it to the correct position. I suspect the cause is that the ArcView USGS DEM import expects a UTM projection, but in this case it is not because SRTM comes in Geodetic.

Step 3: Processing in ArcView 3.x (with Spatial Analyst active)

Demshift.ave

1. Go to Window>your project>Scripts>New

2. Go to Script>Load text file and navigate to where you downloaded the script (the standard directory is x:\esri\ AV_GIS30\ArcView\samples\scripts)

3. Load demshft.ave

4. Go to Script>Compile, then Script\run

5. Load your USGS DEM and let the script output a Grid

6. If the script crashes after creating the grid, just remember where you put it and add it manually, because it did work. (It should be in your working directory).

7. Create contours from new grid

8. Save your grid permanently; go to Theme>Save dataset

Note: to move or delete a grid, only use the Manage Data Sources option under “File”, not Windows Explorer

Note: For more background on DEM’s, Grids, and the demshft.ave script, go to:

http://www.okanogan1.com/natural/ecology/AVHelp/DEM/using_dem_data.html or

http://web.mit.edu/gis/www/arcgis_dem/

9. Problem is the coordinates are not accurately mapped, ok with demshift but then the grid

Note: This method preserves all geocoding information

Note: SRTM’s come in Geodetic projection and WGS84 datum, which is perfect for ArcView 3.x

Top...

Creating 3D Scenes in ER Mapper from ArcView grids or contours

You would need to do this only if you have an ASTER HDF DEM level 3, and do not have ER Mapper 6.4 to process it (see start of page on how to go from HDF to geocoded BIL). In that case, you would end up after all of those steps with an image that you can contour in ArcView, and then export to ER Mapper. If you do have 6.4, or if your DEM was level 2, you can create contours directly in ER Mapper using the contouring wizard. So the process described below is a very longwinded way for users who prefer to have their 3D scenes in ER Mapper rather than ArcView, but need ArcView as an intermediate between ASTER HDF level 3 and ER Mapper.

In ArcView

Generate contours from your grid that you made after importing the USGS DEM that you made from the Aster HDF file (see steps 1C or 2C, depending on hemisphere).

à Activate the 3D analyst extension and go to Surface>Create contours)

Clean up holes in the contours derived from the DEM with information from topographic maps if you have those, by adding contours manually
Save your shapefile as a new PolylineZ Shapefile

If you have trouble copying and pasting contour lines because the elevation attribute is not preserved, a trick is to select the contours that you want copied and then merge your source theme with a theme that will contain all the contours you want to have combined. For me, the geoprocessing wizard failed to merge the selected features so I used the “merge theme” option of the ArcView utilities extension. See the link in note3.

Top...

With 3D Analyst and Spatial Analyst active, create a grid from your (updated) contour shape file (Theme>Convert to Grid; Choose file name for new grid, set Output Grid Extent to same scale as source shapefile!)
Add the grid to your view and then choose File>Export Data Source
Choose Grid to Export, Choose file name for new grid
Choose “Binary raster”
You will now get two files, an .flt and an .hdr file. Their structure is similar to an .ers file (header) and its source file
Open the .hdr file with WordPad
Note the nr of columns (cells) and rows (lines), the coordinates, and the cell size

Note: We will not actually use this file in ER Mapper; we just need to know the coordinates so we are using this procedure to create the header file with the appropriate coordinate. Therefore you must:

a) Set the scale of the grid the exactly same as the extent of your shapefile, as ER Mapper will do the same; otherwise your georeferencing will not match

b) Choose a low output resolution for your grid to speed up the process, this will not affect your final resolution as that depends on the detail of your contours in the shapefile only

Top

In ER Mapper, go to

Utilities>Import vector format
Select your shape file that represents contours (PolylineZ)
Fill in the right projection and datum (Geodetic in case of the geographic projection that is native to ArcView)
Under “Import attribute”, select Elevation
Click Ok
Go to Process>Gridding wizard
Select the .erv file you just produced
In the last screen, select IEEE4 Byte real, and lower the cell size until your file size is a reasonable size
Note down the numbers of cells and lines (i.e., columns and rows)
Click finish (this might take awhile)
Look at the resulting algorithm, which contains a pseudolayer with your grid, and the same layer as shading
Judge whether it has as much detail as your original contours, if not, increase the file size in step 17
Look for possible errors and note that holes in the original DEM will be filled in as flat surfaces
Save the algorithm

Your new grid file will unfortunately have lost its coordinates so we need to manually register the image. To do that, we will use the header information stored in the .hdr file as listed in step 9.

For ER Mapper, we require the coordinates of the top left pixel of the image, while the .hdr file gives us the lower left coordinate. Thus, you need to convert the latitude only using the nr of rows and columns listed in the .hdr file.

Newlat = (cell size * # rows) + oldlat
Newlong = oldlong (because the image is rectangular, the top left coordinate is usually not the top left pixel of your actual data, which may follow the oblique paths typical of satellite derived images)

Now we have the new coordinates of the top left corner. However, our cell size of the binary file output by ArcView is different from the cell size of the grid produced by ER Mapper. An easy way to calculate the cell size of the ERM grid is by dividing the # columns from the .hdr file by the # of columns you noted down while using the gridding wizard.

Newcellsize = oldcellsize * (# columns .hdr / # columns wizard)

Now we have enough information to register the gridded image.

Top

In ER Mapper, go to

File>Open
Select your gridded .ers file
Do not open it yet!
Click “info” in the File Open dialog box
Click “edit” in the Dataset Information dialogue box
Click “Raster info”
Click “Registration point”
Fill in the old longitude coordinate from the .hdr and the newly calculated latitude coordinate (leave x, y at 0)
Click Ok, and go to “cell size”, fill in your calculated cell size (leave z at 1)
Click Apply, Ok , and Close
Reopen your grid file
Go to View>Geoposition and check whether coordinates look ok at a first order approximation
Close grid and reopen your saved algorithm
In the algorithm dialog box, reload both the pseudolayer and the shaded layer with your grid file (only upon reloading will the coordinate system reset and will the window pan to include your dataset; remember your coordinates have changed)
Save new algorithm

Now you need to check your manual registration went Ok.

Add a surface and load an image that is known to be correctly georeferenced, preferably the one you want to drape over your grid (e.g., a three-band Aster image that is the same area as your original Aster HDF DEM).
Verify they overlap exactly. An easy way to do this is to use the transparency slider under the “Surface” tab, but make sure your grid and your image are in different surfaces; otherwise all bands will become transparent.
Add a height layer into the image you want to drape, in the same surface
Load the grid file into the height layer, and delete the other surface with the grid
In the algorithm box, select View Mode to be “3D Perspective”
If all went well, you should now see a 3D image of your Aster, or Landsat image for example, draped across a terrain model you created by gridding your contour shapefile
Play with ‘shininess’ under the “3D Properties” tab to get rid of excessive lighting effects
Play with the “terrain detail” settings under the “3D view” tab to generate more or less detail.

Note: This is the reason I choose to use ER Mapper to display 3D scenes rather than ArcView, because with ERM you have a choice in the amount of detail you want to generate.

Quick shortcuts for moving around in 3D are:

Left mouse: move image around horizontal axis
Right mouse: zoom in or out
Left + right mouse: move image
Right + left mouse: rotate image around vertical axis

Top...

Alternative route

You could also skip the import of contours and just use the grid that ArcView outputs in step 7 above.

In that case, go to ER Mapper

Utilities>Import Binary grids>BIL

Fill in projection data
Click setup
Fill in the nr of rows and columns listed in .hdr file
Fill in Band = 1, select IEEE4ByteReal and press ok
Repeat from step 24, but now use the cell size listed in the hdr file
Check your georeferencing
If ok, then create a grid from your new .ers file
Now repeat from step 40

Reason I do not use this alternative method is that although it looks shorter, gridding takes forever in this fashion...

Top...

Import GTOPO30 USGS DEM files into ArcView 3.x

Generally, changing the .dem extension to .BIL in Windows Explorer works for this type of USGS Dem, not for the Geotiff DEM output though. So if it is a USGS DEM, once you have added the .BIL image, you can create grids and contours from the resulting grid.

From ESRI website, article 14707:

Download a GTOPO30 DEM. This file comes as a .TAR file, which contains the *.dem and *.hdr files. Both files are required for this operation.

1. Uncompress the TAR file using WinZip 6.2 or higher. Make sure that under Option>Configuration>Miscellaneous the ‘TAR file smart conversion’ is NOT ticked

2. Change the file extension from *.dem to *.BIL. You can do this from Windows Explorer or a Command Prompt window.

3. In ArcView, add the *.BIL to a view as an image theme.

4. Make your Spatial Analyst extension active, if it was not already

5. Make the BIL image the active theme in the view.

6. Select Convert to GRID from the Theme menu.

7. Specify a name and directory location for the new grid on the Convert dialog box.

8. Click OK.

9. Select Yes when asked to add the grid to the view.

10. Make the new grid the active theme in the view.

11. Select Map Calculator from the Analysis menu.

12. Enter the following equation into the expression box on the Map Calculation dialog.

    ([in_grid] >= 32768).con([in_grid] - 65536, [in_grid])

Note: in_grid is the grid created in steps 4-6, so replace with your file name if needed

12. Click Evaluate and close the Map Calculation dialog.

13. Make the new Map Calculation Theme the active theme.

14. Select Map Calculator from the Analysis menu.

15. Enter the following equation in the expression box on the Map Calculation dialog.

 ([Map Calculation 1] = -9999).setnull([Map Calculation 1])

Note: Map Calculation 1 is the grid created in step 12.

16. Click Evaluate and close the Map Calculation dialog.

17. Your new grids have to be saved, as you will lose them if you do not save your project, and even if you do, the grids will be in your temp folder with a temporary name. There are two options 1) Theme>Save Dataset or 2) File>Manage Data sources>Copy Grid. Create a directory first for grids only, and make sure you know which grid you need to save (See Theme>Properties for file name and location). Add your new grid to test if it works, then delete the temporary grids using the same Data Source manager.

Notes: - The first formula is a conditional form of calculation, needed on PC’s to set the byte order

properly (Intel rather than Motorola). For any values higher than 32768, 65536 will be

subtracted.

- The second one has to do with ocean values being -9999 in GTOPO30 data, which have to be set to 0.

See http://www.esri.com/news/arcuser/0101/avfiles/lesson10.pdf for an excellent tutorial on the above.

Top.....

Error calling unlink for file: Returned Error Code 13.

1) Try the ESRI solution first if your error is of a general nature (i.e., you do not know yet what is causing it)

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=12600

However, this did not work for me. Other possible causes and workarounds are listed below.

2) The error occurs when files are orphaned on the hard drive but still connected to the ArcView project. If the error appears you should save the project and try again. If this is unsuccessful these files must be cleaned up.

Step 1. Identify the working directory in the Project. To do so, open a View and click on “File… Set Working Directory”. You need to remember this location for the next step. A good technique is to highlight the path and copy it into the Windows buffer.

Step 2. Close the project & quit ArcView

Step 3. In Windows, navigate to the Working Directory you found above.

Step 4. Select all the files starting with “delta”, e.g. delta1.dbf, delta1.shp, etc. Delete these files. Doing so will not harm the project but will allow you to keep working.

Step 5. Re-start project & continue to work.

Source: http://www.epa.gov/nerlesd1/land-sci/agwa/manual/trouble.htm

If the unlink error 13 is related to images displayed through the ER Mapper plug-in:

This is a bug with ArcView, corresponding to when Image Analysis Version 1.0 is installed but the extension is not loaded, displaying an ER Mapper or ECW image may result in a 'Error calling unlink for file C:\TEMP\ximg2.BIL' type of error message. This occurs when you have both the ER Mapper and ECW Image Support extension and either the IMAGINE Image Support or TIFF 6.0 Image Support extension loaded at the same time and have installed the Image Analysis extension. You can avoid this error by turning off the IMAGINE Image Support or TIFF 6.0 Image Support.

Source: ER Mapper Knowledgebase, article 56 (www.ermapper.com)

If this does not resolve the bug instantly, close and reopen your project. You may also have to delete all images that generate the error, and then add them again one by one. Display the full image ‘theme’ to test if the error does not come back.

Top….

General good practices to keep your project from corrupting:

1) Check that your Windows and USER TEMP directories conform to these rules:

Should not contain spaces
Not be more than 8 characters long
Not contain any other characters than letters

2) Check that your Windows temp or ArcView working directory are not full or corrupt.

Solution: clean your TEMP directories (delete all files and subdirectories, and move ones that you want to keep elsewhere).

3) Check that all your files are linked to their proper source directory.

Remove all links to files that generate errors and then add the files manually.

4) Regularly remove all attribute tables. They will be regenerated when necessary.

(Note: this does not apply if you manually change the tables, I think). This will keep your project clean.

Top...

_______________________________________________________________________

From ER Mapper FAQ:

How to convert a Lat / Long Coordinate to an Eastings / Northings  coordinate or vice versa using ER Mapper tools

ERM supplies that can help you in converting single sets of coordinates to and from Lat / Longs and Eastings / Northings.

The programs are called "togeo" and "fromgeo" and they are found in the $ERMBIN directory.  They will ask for a datum and map projection and a set of coordinates.

NOTE: This is also a good way of debugging projections that you have added to the GDT database.

Note:  I have not yet tested this to see if it works.

Useful links:

ESRI ArcView

http://support.esri.com/index.cfm?fa=knowledgebase.techArticles.articleShow&d=16605

HowTo: Add DEM data to ArcView, ESRI Article 16605

http://www.esri.com/news/arcuser/1099/webdata7.html

Rotating Point Symbols in ArcView, by Mike Price; very useful for geologists/cartographers

http://www.esri.com/news/arcuser/0701/avconvert.html

Importing ArcView GIS Projects into ArcGIS, by Mike Price

http://www.okanogan1.com/natural/ecology/AVHelp/index.html

Using ArcView 3.2 with Extensions, by George Wooten (also has great DEM tutorials)

http://support.esri.com/index.cfm?fa=knowledgebase.techArticles.articleShow&d=15322

Error: Segmentation Violation

http://support.esri.com/index.cfm?fa=knowledgebase.techArticles.articleShow&d=10364

HowTo: Salvage a damaged ArcView project file: Article 10364

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=14074

HowTo: Beat Segmentation Violations before they happen: Article 14074

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=16425

HowTo: Improve redraw speed for features of a theme

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=21848

HowTo: Find joined or linked tables and fields in a project.

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=15500

Bug: Join option is greyed out

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=11930

HowTo: Project Decimal Degree X/Y coordinate values

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=18854

Bug: Floating point grid does not display or label properly

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=15826

Bug: After joining tables the field names appear, but the attribute information does not.

http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=18840

Bug: ArcView hangs with "Reading updtruls.db" in status bar

Free imagery

http://photojournal.jpl.nasa.gov/PIADetQuery.html

Planetary Photojournal: Resource for great images of various Spacecraft, including ASTER sensor, Landsat, and SRTM (great for teaching)

http://photojournal.jpl.nasa.gov/mission/SRTM

Example of the Planetary Photojournal: Great SRTM images

Free data

Global bathymetry (ETOPO2) = 2 min resolution

http://www.ngdc.noaa.gov/mgg/fliers/01mgg04.html

Note: (This data set has a higher resolution than the global topo data that comes with ER Mapper)

For ER Mapper grid production: download the grid as bin; you do not need the header, import as binary raster/BIL and enter rows and columns that the web site gave you; put in 0.0333 as cell size (= 2 minutes resolution of this data set). Now your grid is reasonably georeferenced.

Earthquakes (US and global), can display as maps and can get the data

http://wwwneic.cr.usgs.gov/neis/epic/

For ArcView import, choose the export as comma delimited file, copy and paste in WordPad, save as text file, import with Excel and select comma as separator, convert the lat/longs to numbers, save as DBF version 4, add table to ArcView, add event theme and pick your table. You should now have a shapefile with all your earthquake data.

Global 90m DEMS (SRTM Digital Elevation Model in HGT format)

http://edcftp.cr.usgs.gov/pub/data/srtm/

See Methods of importing SRTM for ArcView and ER Mapper

GIS User Guide

http://www.yale.edu/ceo/Documentation/CEOGuide.html

An excellent GIS user guide from Yale University on many aspects of GIS and image formats

Top...

The above is an integration of my own experiences with (partial) information from several GIS websites and with advice received from NGIS and ER Mapper in Perth. This does not mean to imply that the methods outlined above are particularly endorsed or recommended by these sources. Thus the material included on this site is provided for informational purposes only.

Last updated 2-07-2004

free hit counter