about altitude precision in DEM file formats

[yip]

Hi all,

Firstly, sorry about the language, english is not my primary language.



Some time ago, orux added Elevation data features in the app.

If not using the online option, the app needs the DEM data files to be in either of these formats:

• HGT

• DEM+HDR

(which I believe is BIL+HDR)[/list]

To better understand them,

Here I will sum up the main differences between the two (as I see them 'cause I'm no expert), and I'll add the Geotiff which is also pretty standard. I'm sure people here knows a lot better, feel free to comment.



HGT:

• -

Coverage: 1 file for each 1° x 1° tile[/list]

• -

Resolution: fixed, either 1 arc/s or 3 arc/s (30m or 90m)[/list]

• -

Vertical Precision: from -32000m to 32000m, in steps of 1m[/list]

• -

Data Size: 25MB, only uncompressed (for 1 arc/s file)[/list]

DEM+HDR:

• -

Coverage: free size file (described in the HDR file)[/list]

• -

Resolution: free I think[/list]

• -

Vertical Precision: from -32000m to 32000m, in steps of 1mm (millimeter) probably[/list]

• -

Data Size: 50MB, only uncompressed (for 1 arc/s file)[/list]

GeoTiff:

• -

Coverage: free size file (described in the file itself)[/list]

• -

Resolution: free I think[/list]

• -

Vertical Precision: from -32000m to 32000m(?), in steps of 1mm (millimeter) probably[/list]

• -

Data Size: ~5MB, compressed, 50MB uncompressed (for 1 arc/s file)[/list]

In other words, it seems that there is a lot of bloat in GeoTiff and DEM/BIL formats, just to retain this extra vertical precision that is not so reliable anyway, depending on the data source.

and that explains why a set of (compressed) HGT files is always smaller than its original GeoTiff source at the same resolution.



Based on these thoughts, for me HGT is the clear winner. it takes half the size of DEM+HDR, but carry the same precision; except for the vertical resolution but most people would'nt need a millimeter precision for their outdoor activities ...

However, It has drawbacks:

-It is not available for resolution better that 30m (1 arc/sec)

-It is not compressed (but neither do the DEM+HDR format)



Well, I'm not certain of all that, please feel free to add your own experience on the matter  :oops:



yip

[Maki]

Hi, thanks for the info that HDR+DEM is actually HDR+BIL, I wasn't able to figure it out by myself.



It is a very interesting format because you can use any grid size.

About file size you can cut it in half (same as HGT) by setting the "-ot Int16" option in GDAL which will force 16 bits integers rather than 32 bit floating point data.



In theory you could also use other projections, I mean the format supports them. Unfortunately I'm apparently not able to use the Lambert93 (used by IGN France public DEMs) in OruxMaps, I'll need to investigate more.

[davidecapod]

Hi, I am trying to use HDR+DEM in OruxMaps, but without luck.

I have in .ASC file I got from http://tinitaly.pi.ingv.it">http://tinitaly.pi.ingv.it, and I converted it to HDR+BIL using "gdal_translate -a_srs EPSG:32632". The result is working properly in QMapShack, for example.

Then I put them in oruxmaps/dem folder, renaming .BIL to .DEM, but OruxMaps does not like it.

What am I doing wrong?

Thanks

Davide

[mitxelin]

Hi Davide



EPSG:32632 is a projected SRS and Orux will not load it ( as far as I know). You can use gdalwarp instead of gdal_translate to create your HDR+DEM using EPSG:4326.



Try this:

Code Select Expand

gdalwarp -s_srs EPSG:32632 -t_srs EPSG:4326 -of EHdr yoursourcefile.asc yourtargetfile.dem

trick1: yes you can force .dem for the target file extension, so you don't have to rename the file later

trick2: if you have created a .vrt file with several .asc files for QMapShack you can just use that .vrt file as source for gdalwarp

trick3: A HDR DEM consists of several files with the same name and extensions .dem .hdr .prj ..., you must keep them together. You would like to create them in a specific folder, then simply copy that folder to your device.



Hope this helps  

[davidecapod]

Hi mitxelin, thanks for your help.

Now the HDR+DEM is loading in OruxMaps (good!), but... it is not right.

In both OruxMaps and QMapShack I see the slopes (and also elevation shading) with "stripes", as the conversion did not work properly.

After some googling, I read for example here https://tilemill-project.github.io/tilemill/docs/guides/terrain-data/">https://tilemill-project.github.io/tile ... rain-data/">https://tilemill-project.github.io/tilemill/docs/guides/terrain-data/ to add "-r bilinear".

With this switch, no more stripes; but the result is anyway different from the original .ASC one: it seems to lose some "sharpness" and details (visible mainly on slopes).

What is the best method to keep original data and quality? I also tried with other -r switches (cubic, cubicspline, ....)

I see with gdalinfo that the sizes are different; is it possible maybe to keep same size to avoid resampling?

Sorry for the lot of answers, but I am no expert with the various gdal* tools...

Thanks

Davide

Code Select Expand


Driver: AAIGrid/Arc/Info ASCII Grid
Files: w51040_s10.asc
Size is 5010, 4755
Coordinate System is `'
Origin = (399950.000000000000000,5147500.000000000000000)
Pixel Size = (10.000000000000000,-10.000000000000000)
Corner Coordinates:
Upper Left  (  399950.000, 5147500.000)
Lower Left  (  399950.000, 5099950.000)
Upper Right (  450050.000, 5147500.000)
Lower Right (  450050.000, 5099950.000)
Center      (  425000.000, 5123725.000)
Band 1 Block=5010x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999



Driver: EHdr/ESRI .hdr Labelled
Files: out.dem
       out.dem.aux.xml
       out.hdr
       out.prj
Size is 5812, 3831
Coordinate System is:
GEOGCS["GCS_WGS_1984",
    DATUM["WGS_1984",
        SPHEROID["WGS_84",6378137,298.257223563]],
    PRIMEM["Greenwich",0],
    UNIT["Degree",0.017453292519943295]]
Origin = (7.696733186277390,46.479219586200863)
Pixel Size = (0.000113147243321,-0.000113147243321)
Corner Coordinates:
Upper Left  (   7.6967332,  46.4792196) (  7d41'48.24"E, 46d28'45.19"N)
Lower Left  (   7.6967332,  46.0457525) (  7d41'48.24"E, 46d 2'44.71"N)
Upper Right (   8.3543450,  46.4792196) (  8d21'15.64"E, 46d28'45.19"N)
Lower Right (   8.3543450,  46.0457525) (  8d21'15.64"E, 46d 2'44.71"N)
Center      (   8.0255391,  46.2624860) (  8d 1'31.94"E, 46d15'44.95"N)
Band 1 Block=5812x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999


[RichLangford]

As this is the most recent post highlighting issues with DEMs, I thought this would be a good place to share my trials (and tribulations) from the past few days.



To find out how to get DEM data other than SRTM data into Oruxmaps, I first looked at converting the SRTM HGT files to DEM+HDR. This allowed me to better understand when the conversion was successful, as I could always go back to the underlying HGT file for comparison. HGT files almost identical to BIL files, without a separate header; a hex comparison shows that there are differences, and you cannot simply rename HGT as BIL. The coordinates for an HGT file come from the file name, so if you change the name the file will either fail to load or load in a different position.



Most software converting HGT files modifies the data file slightly and writes a header that may or may not work with Oruxmaps. Here is how to make sure it works for your data.



For example GDAL conversion to BIL+HDR in QGIS changes the HGT ULYMAP to reflect the starting point and loads almost correctly in QGIS, so seems to be a reliable utility. The only error is taking the cell corner and not the cell centre as the starting point. ER Mapper writes a header that correctly interprets the cell coordinates with a note "# registration point is center of lower-left cell", and correctly assigns the coordinates to the middle of the cell, but does not generate the correct HDR for a BIL. However, this can be rewritten to make it work by simply following the model HDR from GDAL. Landserf does not write a BIL that can be read, regardless of editing the header. Other software may do the job, such as Ilwis and Saga, or you may have access to high end software such as MapInfo and ArcGIS. The principles will be the same; does the software write the correct data and header file?



The key to success seems to be a good HDR file, which can be summarized using this example of Lidar data from an historic gold mining area in Tasmania:



BYTEORDER   I

NROWS      2703

NCOLS      3695

NBANDS      1

NBITS      16

ULXMAP      147.036925

ULYMAP       -43.190753

XDIM      9.9999999999973581e-006

YDIM      9.9999999999986354e-006



The data file is 34Mb, and the data are freely available online, so let me know if you want a copy.



The process that seems to work most reliably is using the QGIS GDAL Warp (reproject) to get any dataset in WGS84, then QGIS GDAL convert to produce the BIL(DEM)+HDR that Oruxmaps needs. The only problem I have encountered is that HGT files will take precedence over DEM+HDR files, so you will have to remove or disable (rename) HGT files in an area of interest.