Mapbox format offline maps

[Hauke]

Huge step forward: Was able to create my own Offline map successfully! Lots of manual work, and still need to investigate some details to get it nice, but the "Proof of concept" is done and works! I'll write up a full guide at a later point, when I worked all out.
Thanks again Tronpo, your example offline map was very valuable for this!

[Tronpo]

I'm glad you accomplished!! Looking forward to seeing your progress

[Hauke]

Quick Update: I've now all things in place - a detailed write-up will follow, but I may need a few weeks time for that, other projects looming. So here's what I currently achieved:

An offline atlas that contains:

• The basemap.de rendering, slightly altered to show different kinds of wood (needleleaved, broadleaved, ...)
• The contour lines from basemap.de (these are excellent in quality!)
• An overlay of OpenStreetMap data, like POIs, access rights (e.g. parking places for only customers or that require a fee) etc., generated from the Geofabrik pbf files via tilemaker into mbtiles format
• Sprites and Glyphs of basemap plus my own additions (Can recommend free texture packer for generating the sprites files)

Using QGis I downloaded the basemap data into two mbtiles DBs (map data + contour lines), and added my OSM mbtiles.

I created this for the German states Northrhine-Westfalia and Rhineland-Palatinate, which add up to about 50.000 km² --> This just eats up 650 MB! That is just amazing!

I really like that I even can edit the style of basemap - I guess to save rendering resources I might disable objects in the map that are of no interest for me.

This is very new, so I need to gather a bit of real world experience with it - one thing I can observe: If I use this offline atlas, adding also Orux Maps hillshading, my phone's battery eats away rather quickly... I hope that this is just because I played around so much just now, enjoying my product... To be on the safe side, I'll bring a powerbank for the next hiking tour :-)

Also, when browsing around a lot in the map, at some point something breaks and the rendering of new areas is incomplete. Loading another map and switching back to mine fixes it. I guess it's a bit too much for the limited resources of my phone...

[Tronpo]

Hello, you already have the first follower of your thread about these maps.
Congratulations and thank you for sharing your knowledge

[Hauke]

This is not the promised writeup, but I am so blown away by the possibilities of the Mapbox maps in Oruxmaps that at least I'd give some more information on what I've achieved so that impatient people could follow my thoughts if they are tech savvy. Here's what I have achieved:

• Creating offline maps from online sources, both raster and vector
• Creating a vector offline layer from OpenStreetMap data
• Combining these created maps - again, both raster & vector, and even online and offline!
• Adding my own glyphs, sprites and other style settings

A few more details:

1) Creating offline mbtiles files from an online source

1a) Vector

Currently I did this with basemap.de and using QGIS (exists - like all other tools i mention here - for Win, Mac and Linux: https://www.qgis.org/de/site/). What I did is adding a vector tiles source. As URL you use for the map data:
https://sgx.geodatenzentrum.de/gdz_basemapde_vektor/tiles/v1/bm_web_de_3857/bm_web_de_3857.json
and for the excellent contour data:
https://sgx.geodatenzentrum.de/gdz_basemapde_vektor/tiles/v1/bm_web_hl_de_3857/bm_web_hl_de_3857.json
You can then add the vector tile sources to your project, and then from the Toolbox you select "Download vector tiles" and that let's you download them into an mbtiles file.

For the map data, max zoom level is 15, but 14 is enough, perhaps I'll try to go down to 13 if I analyzed the style files a bit more. It saves a lot of space to not go too deep. And with vector, at some point you do not gain real information anymore.
For the contours it is enough to download to level 13, unless you're interested in contour lines smaller than 10m spacing.
These mbtiles work flawlessly in Oruxmaps.


1b) Raster

I am currently planning my vacation in the Böhmerwald, which is half in Germany and half in the Czech republic. Unfortunately the Czech republic does not (yet?) offer vector tiles, but excellent raster tiles via WMS! For downloading them you could also add them to QGIS - if you want to, add an ArcGIS REST Server with the URL
https://ags.cuzk.cz/arcgis1/rest/services

What I did is use Mobile Atlas Creator (MOBAC - https://sourceforge.net/projects/mobac/ - I guess it is well-known to Oruxmappers), where I added the "standard" BeanShell Script for WMS, using the source:

Code Select Expand

"http://ags.cuzk.cz/arcgis1/services/ZTM/MapServer/WMSServer?request=GetMap&service=WMS&version=1.3.0&layers=0&crs=EPSG:3857&width=256&height=256&styles=default&format=image/jpeg&bbox="+ mercatorTileEdges(x,y,zoom)
There is also a WMTS, but its zoom levels are non-standard and I could not get it work.

The advantage of MOBAC is that you can do a polygon select instead of a rectangle, which in my case saved a lot of download of empty tiles in the border region between Germany and Czech republic.

Both, QGis and MOBAC allow the download of an mbtiles database, which again works flawlessly in Oruxmaps (at least MOBAC does, did not really try QGIS).


2) Creating a vector offline layer from OpenStreetMap data

When hiking, I use the "official" maps of a country, like basmeap.de or the Czech Geoportal maps, and overlay them with a rendering of OSM data. This rendering contains (and some more):

• tracks, paths and other non-car-ways - all that might be relevant while hiking and might not be contained in the official maps
• POIs of my choice, e.g. benches and picnic sites, shelters, observation towers, waterfalls etc. - anything that I find attractive or useful while hiking
• landscape features and relevant landuse, like cliffs, vineyards, ridges, glaciers etc.
• Special way features like via ferrata, assisted trails, the SAC hiking scale classification etc.
• names of many of the above features

The goal of this rendering is a) to have all the information I want to have in the map, and b) to identify ways that the official maps do not know, or vice versa, ways that are missing on OSM. I actively contribute to OSM, and I sometimes do a small detour on my hiking tours if I see a way that is not yet on OSM and I go there and map it afterwards.

To create this layer, my procedure is

• Download the OSM info from Geofabrik as PBF files, source is: https://download.geofabrik.de/
• If I have more than one PBF file, I use Osmosis (https://github.com/openstreetmap/osmosis) to merge the PBF files into one (tilemaker - see below - can also merge data, but it does a bad job)
• I use tilemaker (https://github.com/systemed/tilemaker) to filter the relevant OSM information out and store them in a mbtiles file, which again works nicely with Oruxmaps

After first successes with smaller regions I went bold and downloaded the whole Germany PBF from Geofrabrik, ran it through tilemaker - and lo and behold! - I took less than 5 minutes to process, and my OSM layer for whole Germany(!) is just 650 MBytes - thats nothing! So cool...


3) Combining these created maps - again, both raster & vector, and even online and offline!

All this revolves around is creating your own Mapbox style file - where I started from was the style file that came with basemap.de maps. I use the Maputnik editor (https://maplibre.org/maputnik/) to modify the style, and in order to use my selfmade vector layer, I provide it to Maputnik via tilemaker-server (part of the tilemaker install). Since this is a very simplistic server, I need to put it behind a reverse proxy to do HTTPS, using nginx. Sounds complicated, is simple after all. The cool thing is that in the style file you can define multiple data sources - and these can be raster or vector, and they can be online (just use the URLs of the tile servers online), or offline, stored as mbtiles on the Smartphone. For this, the source URL needs to be

Code Select Expand

      "tiles": [
        "http://localhost:8998/YourOwn.mbtiles/{z}/{x}/{y}.pbf"
      ],

Or .png or .jpg if it is raster. And again, you can specify multiple mbtiles sources this way. In my current version for the Böhmerwald vacation, I have four data sources:

• The basemap.de downloaded mbtiles (Size: 250 MB for a region of ~55.000 km²)
• The basemap.de downloaded contour mbtiles (Size: 28 MB for the same region)
• The Czech geoportal raster download mbtiles (Size: 490 MB for a meagre 4.000 km² to zoom level 15 - that illustrates the superiority of vector maps very well...)
• My OSM layer vector mbtiles file (Size: 215 Mb for whole Czech republic plus Bavaria - together 150.000 km²!)

I also have a second version of the style that has the basemap.de and the Czech geoportal as online sources, and my OSM layer as offline! This works as long as I have data reception with my phone (The Czech WMS layer is very slow - may need to troubleshoot this at some point...).

All this makes me _so_ flexible during hiking - it is a dream!


4) Adding my own glyphs, sprites and other style settings

Glyphs (aka font data for texts) are a nasty thing IMHO, since each font needs 256 files. Fortunately, you can either download them from existing vector offerings like basemap.de, or, even easier, you use https://maplibre.org/font-maker/ and put in the TTF file of your choice. After processing, you can download a ZIP with your glyphs - nice!
You also need sprites - i.e. one big PNG file which contains your images you want to use for POI-icons, line-patterns or area-textures, plus a JSON file that tells the renderer how to cut out the individual images again from the single PNG. Strange concept IMHO... Whatever. Problem is: There can be only _one_ sprite-file for the whole style! So you need to merge the sprite from other vector sources plus your own icons into a new sprite. For this I used "Free texture packer" (http://free-tex-packer.com/) - it is not perfect and the JSON needs a bit of editing afterwards, but it takes the bulk of the work.


As I said: This is just like a dream come true! I have such a flexibility in putting maps together as I want, and from a bit of practical use it seems that even battery consumption is not too bad. Admittedly based on statistics of one, the plus in battery consumption compared to old-style raster maps, is 10% max. But I'll be careful here, more real life experience needed.

The ability to zoom in without the need to carry tons of raster data is just awesome! Especially if you have a place where the rendering otherwise is very crowded due to many overlapping POIs.


Still, I have some issues, and some ideas which I hope OruxMaps developers may pick up:

• Bug: when oruxmaps is put into the background, new tiles are not rendered anymore when I bring back Oruxmaps to the front. I need to switch to a different map and back, or restart Oruxmaps. My guess is that the tile-server on port 8998 dies. This is unfortunately _really_annoying, and when working with raster maps makes the whole thing unusable.
• When I change sprites or the style, some settings/changes are only picked up if I restart the phone. Even closing and restarting Oruxmaps does not help. This is interestingly the same with the Maputnik editor! I need to restart my computer(!) to have some changes apply. Closing the browser is not enough.
• When I pick the Oruxmaps functionality "Open an offline map here...", the offline mapbox maps are not listed. It would be really nice if they were.
• Switch to MapLibre! Mapbox has become kindof closed source and as far as I understand costs money. Maplibre is a fork, it is basically compatible, and works just as well! In places even better - with Oruxmaps' rendering sometimes textures are strangely distorted when scaled up to the highest zoom levels. Maputnik/Maplibre have better results.

Find attached a screenshot of the Czech/German border region with some of my renderings explained. If you are interested in having the resulting files and/or the process files to look deeper, let me know, I'll make them available for download.


Let me repeat: I am so happy with this new atlas options! Thanks to the Oruxmaps developers! I consider my 10€/year subscription very well invested. If the tile-server bug is fixed, this will be my new standard for hiking, that's for sure!