Mapbox format offline maps

[Hauke]

Huge step forward: Was able to create my own Offline map successfully! Lots of manual work, and still need to investigate some details to get it nice, but the "Proof of concept" is done and works! I'll write up a full guide at a later point, when I worked all out.
Thanks again Tronpo, your example offline map was very valuable for this!

[Tronpo]

I'm glad you accomplished!! Looking forward to seeing your progress

[Hauke]

Quick Update: I've now all things in place - a detailed write-up will follow, but I may need a few weeks time for that, other projects looming. So here's what I currently achieved:

An offline atlas that contains:

• The basemap.de rendering, slightly altered to show different kinds of wood (needleleaved, broadleaved, ...)
• The contour lines from basemap.de (these are excellent in quality!)
• An overlay of OpenStreetMap data, like POIs, access rights (e.g. parking places for only customers or that require a fee) etc., generated from the Geofabrik pbf files via tilemaker into mbtiles format
• Sprites and Glyphs of basemap plus my own additions (Can recommend free texture packer for generating the sprites files)

Using QGis I downloaded the basemap data into two mbtiles DBs (map data + contour lines), and added my OSM mbtiles.

I created this for the German states Northrhine-Westfalia and Rhineland-Palatinate, which add up to about 50.000 km² --> This just eats up 650 MB! That is just amazing!

I really like that I even can edit the style of basemap - I guess to save rendering resources I might disable objects in the map that are of no interest for me.

This is very new, so I need to gather a bit of real world experience with it - one thing I can observe: If I use this offline atlas, adding also Orux Maps hillshading, my phone's battery eats away rather quickly... I hope that this is just because I played around so much just now, enjoying my product... To be on the safe side, I'll bring a powerbank for the next hiking tour :-)

Also, when browsing around a lot in the map, at some point something breaks and the rendering of new areas is incomplete. Loading another map and switching back to mine fixes it. I guess it's a bit too much for the limited resources of my phone...

[Tronpo]

Hello, you already have the first follower of your thread about these maps.
Congratulations and thank you for sharing your knowledge

[Hauke]

This is not the promised writeup, but I am so blown away by the possibilities of the Mapbox maps in Oruxmaps that at least I'd give some more information on what I've achieved so that impatient people could follow my thoughts if they are tech savvy. Here's what I have achieved:

• Creating offline maps from online sources, both raster and vector
• Creating a vector offline layer from OpenStreetMap data
• Combining these created maps - again, both raster & vector, and even online and offline!
• Adding my own glyphs, sprites and other style settings

A few more details:

1) Creating offline mbtiles files from an online source

1a) Vector

Currently I did this with basemap.de and using QGIS (exists - like all other tools i mention here - for Win, Mac and Linux: https://www.qgis.org/de/site/). What I did is adding a vector tiles source. As URL you use for the map data:
https://sgx.geodatenzentrum.de/gdz_basemapde_vektor/tiles/v1/bm_web_de_3857/bm_web_de_3857.json
and for the excellent contour data:
https://sgx.geodatenzentrum.de/gdz_basemapde_vektor/tiles/v1/bm_web_hl_de_3857/bm_web_hl_de_3857.json
You can then add the vector tile sources to your project, and then from the Toolbox you select "Download vector tiles" and that let's you download them into an mbtiles file.

For the map data, max zoom level is 15, but 14 is enough, perhaps I'll try to go down to 13 if I analyzed the style files a bit more. It saves a lot of space to not go too deep. And with vector, at some point you do not gain real information anymore.
For the contours it is enough to download to level 13, unless you're interested in contour lines smaller than 10m spacing.
These mbtiles work flawlessly in Oruxmaps.


1b) Raster

I am currently planning my vacation in the Böhmerwald, which is half in Germany and half in the Czech republic. Unfortunately the Czech republic does not (yet?) offer vector tiles, but excellent raster tiles via WMS! For downloading them you could also add them to QGIS - if you want to, add an ArcGIS REST Server with the URL
https://ags.cuzk.cz/arcgis1/rest/services

What I did is use Mobile Atlas Creator (MOBAC - https://sourceforge.net/projects/mobac/ - I guess it is well-known to Oruxmappers), where I added the "standard" BeanShell Script for WMS, using the source:

Code Select Expand

"http://ags.cuzk.cz/arcgis1/services/ZTM/MapServer/WMSServer?request=GetMap&service=WMS&version=1.3.0&layers=0&crs=EPSG:3857&width=256&height=256&styles=default&format=image/jpeg&bbox="+ mercatorTileEdges(x,y,zoom)
There is also a WMTS, but its zoom levels are non-standard and I could not get it work.

The advantage of MOBAC is that you can do a polygon select instead of a rectangle, which in my case saved a lot of download of empty tiles in the border region between Germany and Czech republic.

Both, QGis and MOBAC allow the download of an mbtiles database, which again works flawlessly in Oruxmaps (at least MOBAC does, did not really try QGIS).


2) Creating a vector offline layer from OpenStreetMap data

When hiking, I use the "official" maps of a country, like basmeap.de or the Czech Geoportal maps, and overlay them with a rendering of OSM data. This rendering contains (and some more):

• tracks, paths and other non-car-ways - all that might be relevant while hiking and might not be contained in the official maps
• POIs of my choice, e.g. benches and picnic sites, shelters, observation towers, waterfalls etc. - anything that I find attractive or useful while hiking
• landscape features and relevant landuse, like cliffs, vineyards, ridges, glaciers etc.
• Special way features like via ferrata, assisted trails, the SAC hiking scale classification etc.
• names of many of the above features

The goal of this rendering is a) to have all the information I want to have in the map, and b) to identify ways that the official maps do not know, or vice versa, ways that are missing on OSM. I actively contribute to OSM, and I sometimes do a small detour on my hiking tours if I see a way that is not yet on OSM and I go there and map it afterwards.

To create this layer, my procedure is

• Download the OSM info from Geofabrik as PBF files, source is: https://download.geofabrik.de/
• If I have more than one PBF file, I use Osmosis (https://github.com/openstreetmap/osmosis) to merge the PBF files into one (tilemaker - see below - can also merge data, but it does a bad job)
• I use tilemaker (https://github.com/systemed/tilemaker) to filter the relevant OSM information out and store them in a mbtiles file, which again works nicely with Oruxmaps

After first successes with smaller regions I went bold and downloaded the whole Germany PBF from Geofrabrik, ran it through tilemaker - and lo and behold! - I took less than 5 minutes to process, and my OSM layer for whole Germany(!) is just 650 MBytes - thats nothing! So cool...


3) Combining these created maps - again, both raster & vector, and even online and offline!

All this revolves around is creating your own Mapbox style file - where I started from was the style file that came with basemap.de maps. I use the Maputnik editor (https://maplibre.org/maputnik/) to modify the style, and in order to use my selfmade vector layer, I provide it to Maputnik via tilemaker-server (part of the tilemaker install). Since this is a very simplistic server, I need to put it behind a reverse proxy to do HTTPS, using nginx. Sounds complicated, is simple after all. The cool thing is that in the style file you can define multiple data sources - and these can be raster or vector, and they can be online (just use the URLs of the tile servers online), or offline, stored as mbtiles on the Smartphone. For this, the source URL needs to be

Code Select Expand

      "tiles": [
        "http://localhost:8998/YourOwn.mbtiles/{z}/{x}/{y}.pbf"
      ],

Or .png or .jpg if it is raster. And again, you can specify multiple mbtiles sources this way. In my current version for the Böhmerwald vacation, I have four data sources:

• The basemap.de downloaded mbtiles (Size: 250 MB for a region of ~55.000 km²)
• The basemap.de downloaded contour mbtiles (Size: 28 MB for the same region)
• The Czech geoportal raster download mbtiles (Size: 490 MB for a meagre 4.000 km² to zoom level 15 - that illustrates the superiority of vector maps very well...)
• My OSM layer vector mbtiles file (Size: 215 Mb for whole Czech republic plus Bavaria - together 150.000 km²!)

I also have a second version of the style that has the basemap.de and the Czech geoportal as online sources, and my OSM layer as offline! This works as long as I have data reception with my phone (The Czech WMS layer is very slow - may need to troubleshoot this at some point...).

All this makes me _so_ flexible during hiking - it is a dream!


4) Adding my own glyphs, sprites and other style settings

Glyphs (aka font data for texts) are a nasty thing IMHO, since each font needs 256 files. Fortunately, you can either download them from existing vector offerings like basemap.de, or, even easier, you use https://maplibre.org/font-maker/ and put in the TTF file of your choice. After processing, you can download a ZIP with your glyphs - nice!
You also need sprites - i.e. one big PNG file which contains your images you want to use for POI-icons, line-patterns or area-textures, plus a JSON file that tells the renderer how to cut out the individual images again from the single PNG. Strange concept IMHO... Whatever. Problem is: There can be only _one_ sprite-file for the whole style! So you need to merge the sprite from other vector sources plus your own icons into a new sprite. For this I used "Free texture packer" (http://free-tex-packer.com/) - it is not perfect and the JSON needs a bit of editing afterwards, but it takes the bulk of the work.


As I said: This is just like a dream come true! I have such a flexibility in putting maps together as I want, and from a bit of practical use it seems that even battery consumption is not too bad. Admittedly based on statistics of one, the plus in battery consumption compared to old-style raster maps, is 10% max. But I'll be careful here, more real life experience needed.

The ability to zoom in without the need to carry tons of raster data is just awesome! Especially if you have a place where the rendering otherwise is very crowded due to many overlapping POIs.


Still, I have some issues, and some ideas which I hope OruxMaps developers may pick up:

• Bug: when oruxmaps is put into the background, new tiles are not rendered anymore when I bring back Oruxmaps to the front. I need to switch to a different map and back, or restart Oruxmaps. My guess is that the tile-server on port 8998 dies. This is unfortunately _really_annoying, and when working with raster maps makes the whole thing unusable.
• When I change sprites or the style, some settings/changes are only picked up if I restart the phone. Even closing and restarting Oruxmaps does not help. This is interestingly the same with the Maputnik editor! I need to restart my computer(!) to have some changes apply. Closing the browser is not enough.
• When I pick the Oruxmaps functionality "Open an offline map here...", the offline mapbox maps are not listed. It would be really nice if they were.
• Switch to MapLibre! Mapbox has become kindof closed source and as far as I understand costs money. Maplibre is a fork, it is basically compatible, and works just as well! In places even better - with Oruxmaps' rendering sometimes textures are strangely distorted when scaled up to the highest zoom levels. Maputnik/Maplibre have better results.

Find attached a screenshot of the Czech/German border region with some of my renderings explained. If you are interested in having the resulting files and/or the process files to look deeper, let me know, I'll make them available for download.


Let me repeat: I am so happy with this new atlas options! Thanks to the Oruxmaps developers! I consider my 10€/year subscription very well invested. If the tile-server bug is fixed, this will be my new standard for hiking, that's for sure!

[afgb1977]

Fabulous.
Congratulations.
👏🏻👏🏻👏🏻

[Tronpo]

Hello, a question, the mbtiles you generate are raster? You can generate vector tools.
The increase in battery consumption is normal, in both online and offline json styles the rendering work is done by the entire terminal (phone / tablet
Congratulations.

[Hauke]

Thanks for the congrats :-) I owe part of my success to Tronpo - thanks again for providing your ES-maps!

@afgb1977 & Tronpo: Did you do something similar yourself? If so: Which tools did you use? Especially sprite creation is still not as good as I'd like it to be - happy for any hints here! What I am also not done with is a bit of workflow automation, like e.g. marking a region in MOBAC or GQIS and having the whole map creation done per script - do you have anything in place already?

How about the bug I have with the offline tile server dying - can you reproduce it? For me it is absolutely reproducible - it happens really every time Oruxmaps is sent to the background.

Regarding your question, Tronpo: For Germany I'm 100% vector mbtiles, as soon as Czech republic comes in, I have three vector mbtiles (basmap.de, contour lines and OSM data), and one raster mbtiles (Czech Geoportal map).

Battery use: I was positively surpised by only 10% plus. I was afraid that it would be much more. I guess it is partly because the rendered tiles are somehow cached.

I have two more issues:

• Zoom levels as displayed in Oruxmaps do not match Zoom-levels as per style file/Mapbox spec. Why is that so?
• When I have a line with a pattern of sprites along, or a area with a texture pattern, when I zoom in deep, the patterns sometime become distorted, as if the scaling fails. Strange enough, I can have the same pattern side by side in the map, one OK, one distorted. Any idea what may go wrong here?

[Tronpo]

I've only been able to work with the json styles online, I don't have a computer so my resources are limited. Android + browser + text editor and above all a lot of curiosity.
But maybe knowing that I can user mbtiles raster I'll ask you more questions soon, when I finish my rendertheme mapsforge project.
Regarding the style zoom levels and their correspondence in Oruxmaps, it has always been like this, I don't know if it's because of the mapbox sdk but, the zoom levels are one less in 2D but they coincide in 3D
Your offline maps Work  in 3D?

Best regards!!

[Hauke]

My offline maps "work" in 3D in the sense that I can use them in the Oruxmaps 3D view mode, but I did not include any elevation information, so it is not real 3D. I'm not sure if I'll go down that route - my usecases are limited, and I guess it eats memory again. But I may change my mind :-)

[afgb1977]

You must include the lines of code that point to the 3d resource.
mapbox-terrain-dem

[Hauke]

After a week of hiking in the Šumarva/Böhmerwald here's the real-world experience with multi-layer, self made, mixed vector/raster maps in the Mapbox-format - the TL;DR-version: It's great!!!

A bit more details (some are a bit repetitive to what I wrote earlier - forgive me, it is because I am so pleased with the results!):

• Power consumption: I'm not sure if it really eats more battery - if so, it is marginal. We did 8-9 hr. tours with intense use of Oruxmaps, and after that battery was somewhere between 45-60% remaining, which is matching my experience with pure raster based maps. Thats really good news, and I assume attribute to the caching - i.e. Oruxmaps renders once and then displays bitmaps out of the cache, which would not be much different from displaying pure raster maps.
• Zooming: as expected, it is extremely helpful to be able to zoom in deep if need be. Some places have really a lot information, and zooming in deep allows to get it all. Without the need to store tons of images that eat up tons of space! See the two zooming example screenshots attached.
• Ability to mix raster and vector: The Czech republic Geoportal does only offer raster, so being able to combine "legacy" raster and "modern" vector is making my life so easy!

For me this is now definitely the new standard! Still, there's a bit room for improvment, and some suggestions:

• The tileserver-crash-bug: My workaround was to at the beginning of a tour, after a fresh start of Oruxmaps, once scroll through the map along the planned route (and a bit left and right) at all relevant zoom levels. This made Oruxmaps to build up the cache, and as long as I did not close Oruxmaps, all cached parts of the map worked flawlessly, even after Oruxmaps was in the background. Which was good enough exept for one tour where we needed to take a detour because of a closed area. The map-parts for the detour where not cached, meaning I had to redo the caching procedure again with the new route. Also, once I wanted to look what tower I could see on the horizon, but that part was not cached, and I gave up.
  In terms of the impact, for me it means it is not a critical bug, but a considerably annoying bug, and I really hope it gets fixed.
• No digital zoom availabe: This is also annoying. See the two screenshots below, which show the same region at two zoom levels. The small path "vanishes" at the lower zoom level. I would really have appreciated if I could digitally zoom out while staying at the same map zoom level in order to see the small path in the larger context for spontaneous planning in the field. With pure raster maps this was possible. I only need this ever once in a while, but not having it then hurts. And it is needed not only for the raster layer as you might think: If I take e.g. the Austrian vector maps, smaller paths also are not part of the lower zoom levels, so the same would apply here.
• Really a minor thing, kindof "nice to have": It would be cool to have the offline mapbox format maps available in the list of Offline-maps if you use the "offline map at current position" button.
• The icing on the cake would be if I could show/hide the different layers of a map on the fly. The map for this vacation consists of several layers:
  
  • The German basemap.de vector map layer
  • The German basemap.de vector countour lines layer
  • The Czech Geoportal raster layer
  • My OSM selected information vector layer
  • The Austrian basmap.at raster layer (since there was a chance we might enter Austria during one tour)
  If I would be able to temporarily hide e.g. the Czech raster layer, that would potentially have been very convenient, because the Czech raster layer in the border region to Austria overlapped parts of the Austrian map due to missing transparency.

[Hauke]

Big writeup finally done - find it in my Blog! Any comments very welcome!