Desert Atlas: A Self-Hosted OpenStreetMap App for Sandstorm
Desert Atlas: A Self-Hosted OpenStreetMap App for Sandstorm
By Daniel Krol
– 05 Dec 2023
Hello, my title is Dan. That is my first time posting on the Sandstorm weblog. I obtained concerned with Sandstorm nearly a decade in the past. I imagined a day when open knowledge can be simply deployed by way of Sandstorm onto a mesh community (a lofty aim, I do know), so I created a package for an current software referred to as Kiwix for simple internet hosting of web sites like Wikipedia.
I nonetheless have this imaginative and prescient in thoughts. As we speak, I’m asserting the results of a extra bold effort.
Introducing Desert Atlas
Sandstorm, meet OpenStreetMap. OpenStreetMap, meet Sandstorm.
Desert Atlas is the world map for Sandstorm. With Desert Atlas, you may privately collaborate with pals to seek for locations and save them as bookmarks to be used on the go.
You might be accustomed to OSM telephone functions like Organic Maps that obtain total areas of the map without delay so you may search and browse privately. Desert Atlas was impressed by this mannequin. In contrast to many OSM internet functions, the map areas are totally hosted in your Sandstorm grain, and downloaded with the identical ease of point-and-click that you simply anticipate from Sandstorm. In contrast to Natural Maps, Desert Atlas makes it simple to share your map with a buddy or plan a visit collectively on a personal server that you simply belief.
Once you’re able to take your map on the street, Desert Atlas can also be a companion for Natural Maps (and OsmAnd). You’ll be able to export your bookmarks out of your sandstorm grain to your OSM telephone app to navigate privately.
This has been the results of fairly a little bit of effort on my half, together with studying extra concerning the OpenStreetMap ecosystem than I ever thought I might. However since I needed to be taught a number of issues, I didn’t be taught them very deeply. In case you are impressed by this mission and have deeper data about a few of its elements, I’ve layed out some areas the place you may have the ability to make a big effect fairly simply.
The hole between Natural Maps and Google Maps
Natural Maps means that you can seek for locations, navigate, and create bookmarks that may be imported or exported as KML files. It does all of those in your telephone with out hitting the community. The one concession to privateness is that the map knowledge has to return from someplace. The underlying map areas should be downloaded initially and up to date periodically. Natural Maps’ server is aware of what areas you might be downloading, however they’re every roughly the scale of a small nation, and it occurs possibly as soon as every week. This offers them a lot decrease decision than, say, requesting a selected intersection on demand from a centralized map web site.
However what if you wish to bookmark some spots for an evening in town and share them with some pals? Exporting and sending bookmark information is a bit inconvenient. There’s a web-based sharing characteristic for Natural Maps, however it reveals the placement to Natural Maps’ server. If you wish to plan the night collectively, sharing KML information is very inconvenient. You may end up utilizing a special centralized service (Google Maps involves thoughts however there are OSM-based choices), even in case you import the outcomes to Natural Maps.
That is the place a personal internet app is useful. As an alternative of collaborating and sharing on a centralized service, you should use Desert Atlas on a trusted Sandstorm server. Once you’re performed, you may every export the areas to Natural Maps for navigation and comfort. (And in case you’re not sharing or collaborating, you may all the time persist with Natural Maps. It’s most likely nonetheless safer than a personal internet server sitting on the Web.)
Some Comparable Choices
For comparability, I’ve checked out a pair different choices for self-hosted OSM. I’ve not taken the time to strive them out in depth, so apologies if I get one thing fallacious.
Facilmap for YunoHost and Nextcloud Maps are made to be simple to arrange and have comparable use circumstances to Desert Atlas. They allow you to bookmark, plan journeys, and share the outcomes, all privately. However slightly than downloading areas like Natural Maps, they get underlying map data from and send search terms to third-party companies (similar to openstreetmap.org), which leaks some utilization data.
Headway is a mission that solves this privateness downside. It installs the “full stack” of self-hosted OSM comparatively simply. As you will notice under, the OSM ecosystem has lots occurring, and Headway simplifies setup fairly a bit. Nonetheless, it makes use of Docker, which (within the humble opinion of Sandstorm followers) remains to be not so simple as Sandstorm. (I’ve seen a little bit dialogue about porting it to YunoHost, however it has not occurred as of this writing).
Desert Atlas takes a way more stripped down method than Headway. Constructed for Sandstorm, with lots fewer shifting elements. I’m not conscious of every other providing that concurrently allows you to spin it up so simply (due to Sandstorm), level and click on to obtain the areas you want, collaborate with pals, and have this degree of privateness.
The Easiest Model of All the things
So how does this really work? I saved it easy, which isn’t to say simple! Whereas the implementation was not that arduous, it took a number of effort to search out the appropriate instruments and learn to combine them. All of the heavy lifting was already carried out by these instruments. Large due to all those that created them.
OpenStreetMap is, amongst different issues, a canonical database representing the world map. It’s brazenly obtainable and editable like Wikipedia. OSM functions similar to Desert Atlas have to obtain this knowledge in a technique or one other. Utility creators often present their very own copy of the info in a customized format that their apps obtain. Additionally they use their very own servers to spare the assets of openstreetmap.org. OpenStreetMap gives periodic snapshots of its database as one large protobuf file often called planet.osm.pbf. Utility creators can obtain it and extract what they should make periodic snapshots of the info obtainable to their customers.
Usually, the 2 elements of the OSM app expertise are tiles and search.
Tile format
The normal OpenStreetMap stack features a tile server which imports the uncooked protobuf knowledge right into a Postgres database and generates a grid of sq. png information for each zoom degree. Postgres is tough to get working on Sandstorm as a result of it’s made to be multi-user (although at one level I really obtained a tile server to run and generate tiles in a check atmosphere!) Alternately, because the supplier of the map knowledge for the app, I may have taken the trail of pre-generating the png information for the app to obtain, however I think the ensuing file measurement can be large.
Then at some point on Hacker Information I noticed one thing referred to as Protomaps. It’s a mission for vector-based tiles that render within the browser. It makes use of a file format referred to as pmtiles. A single pmtiles file represents a area of the map in any respect zoom ranges. You should utilize protomaps.js together with the Leaflet UI framework to view it. As you scroll or zoom, it makes vary requests for the precise subset of the file that it must show it. No want for a database, or to generate something inside the Sandstorm app. Every downloadable area of the map has one pmtiles file.
And the way will we create these pmtiles information? I didn’t need to repeatedly copy all the world map from the Protomaps mission. I got down to generate them myself from uncooked OSM knowledge. The easiest way I may discover was to first convert them to a different vector format referred to as mbtiles utilizing a instrument referred to as tilemaker, after which from mbtiles to pmtiles utilizing go-pmtiles from the Protomaps mission.
Tile Schema
I generated my first pmtiles file and… nothing confirmed up. It seems that on high of tile file codecs, there’s a idea of a schema. Protomaps.js seems to have its personal schema. Tilemaker makes use of a lua script to find out the schema of the mbtiles file it creates. The default lua script that comes with tilemaker relies on the OpenMapTiles schema. I initially got down to begin with this lua script and edit it till the info it produced appeared just like the Protomaps schema. Nonetheless, I used to be a little bit nervous as a result of I used to be suggested that OpenMapTiles has an unsure mental property scenario. I could or could not have misunderstood or overreacted, however at any price I had an alternate. The oldsters at Geofabrik created their very own schema referred to as Shortbread that’s licensed CC0 and comes with its personal lua script for tilemaker. To avoid any concern over IP points in creating my lua script to generate information within the Protomaps.js schema, I began from Shortbread.
Search Knowledge
The usual possibility for an OpenStreetMap search service is named Nominatim. It makes use of Elasticsearch below the hood, which once more is a bit heavy responsibility for Sandstorm. I requested myself, “what’s the SQLite of search?” It seems the reply is… SQLite! There’s a plugin called FTS5 that performs moderately properly on Desert Atlas for looking out names within the database. Desert Atlas doesn’t but assist deal with search, so it stays to be seen how properly suited FTS5 is for that.
To generate the search database, I decide what I want to extract from the uncooked protobuf for a given area utilizing pyosmium. The result’s saved to a CSV that will get bundled with the pmtiles file for a similar area. When a consumer downloads a area inside Desert Atlas, the CSV is imported into the grain’s SQLite search database.
Along with looking out inside downloaded areas, I made a decision that it might be helpful to have cities and huge cities constructed into the app in order that the consumer can seek for their metropolis earlier than they obtain any areas. This makes it simpler for customers to orient themselves and discover which area they need to obtain. For this half, I used a pre-baked non-OSM database referred to as GeoNames as a shortcut.
Splitting the world
Natural Maps is usually break up by “administrative area”. It may very well be a rustic, state, or metro space, relying on how dense the knowledge is. Ideally Desert Atlas would work the identical manner.
Geofabrik provides up-to-date uncooked OSM protobuf knowledge break up by areas in such a manner. Nonetheless, they’ve a obtain price restrict that I made a decision was prohibitive for this objective, so I sought out to obtain OSM’s planet.osm.pbf and break up it myself. Geofabrik additionally has a publicly obtainable geojson file that defines the areas they provide for obtain, which I may in precept use to do the identical splitting myself. Nonetheless, the precise course of that they use to slice the planet with this json file shouldn’t be publicly obtainable. In the long term I want to determine this out.
For now, I discovered a “ok” shortcut and I made a decision to take it. I break up planet.osm.pbf utilizing a instrument referred to as splitter from mkgmap. This instrument splits uncooked OSM knowledge into rectangles concentrating on a most uncooked knowledge measurement per area. Sadly the render and search extractions performed for Desert Atlas don’t correspond 1:1 with the uncooked knowledge, so the downloadable areas in Desert Atlas fluctuate fairly a bit in measurement, however I’m taking pictures for about the identical area measurement as Natural Maps.
UI
This half was easy. As talked about above, I used Leaflet, which is the go-to internet UI framework for OpenStreetMap. It’s extensible and labored properly for me. I additionally used a plugin referred to as Leaflet-Search that I used to be ready to hook up with the straightforward Python primarily based backend and question the SQLite database.
So, in a nutshell…
Periodic map era course of (at the moment ~2.5 days)
- Obtain
planet.osm.pbf(world map uncooked protobuf) planet.osm.pbf-> rectangular area osm.pbf information (utilizing splitter from mkgmap)- every area osm.pbf -> area .pmtiles with protomaps schema (utilizing tilemaker and go-protomaps, with due to Geofabrik for the shortbread schema which I edited till it grew to become the protomaps schema)
- every area osm.pbf -> area .csv with search knowledge (utilizing pyosmium)
- every area .pmtiles + .csv -> area .tar.gz -> add to S3
App packaging
- bundle geojson information for low-res world and USA maps
- bundle cities and huge cities from GeoNames
When a grain is created on the consumer’s Sandstorm server, the bundled GeoNames knowledge is imported into the grain’s SQLite+FTS5 search database. The bundled geojson world and USA maps show due to Leaflet. The grain downloads whichever areas the consumer asks for from S3. For every area, the pmtiles file is able to use as-is by protomaps.js and Leaflet, and the search CSV file is imported into the search database, with Leaflet-Search offering the UI.
Getting concerned
In case you are an OSM developer, you might or is probably not facepalming proper now. Possible some a part of this might have been performed in a greater manner. However you additionally know that OSM has a giant, complicated ecosystem, with some ways of doing the identical factor. This was my first publicity to a number of it. This was a number of work, and I wished to forge an affordable however environment friendly path to a minimal viable product.
It’s a modest submission of a usable proof of idea. I didn’t spend a lot time getting hung up on discovering the perfect manner of doing all the things on the primary strive, however at this level I’m vast open to enhancements, together with changing any a part of this with one thing higher.
See here for some areas that I feel may very well be low hanging fruit, particularly for individuals who can train me about areas I’m much less accustomed to (tiles, search, UI, entrance finish) or simply normal enhancements (rewriting the server in Go). Hit me up if you need to assist!