How one can make self-hosted maps that work in all places and price subsequent to nothing • MuckRock
For at the very least a decade now, I’ve advised journalists constructing interactive maps two issues:
- It’s best to create maps with a mode that matches the story you’re telling and matches your newsroom’s broader design sensibilities.
- It’s best to use a hosted service to do that. Internet hosting a full map stack your self is extra value and hassle than it’s value.
These two concepts have some pressure between them. It’s best to personal your map design however not the infrastructure that helps it. It is because the costliest a part of constructing and internet hosting a map is the time a developer places into constructing and sustaining it, and whereas hosted companies aren’t free, they’re usually cheaper than your hourly price. For essentially the most half, I advised individuals to make use of Mapbox and deal with it as cash nicely spent.
I nonetheless strongly consider the primary thought, however I’m much less certain of the second. Over the previous few years, an rising ecosystem of open-source instruments has made self-hosting a map stack each viable and inexpensive. Past value, self-hosting creates a stage of safety and sturdiness that’s arduous to seek out with third-party companies, that are all the time susceptible to being purchased, shut down or just damaged by neglect.
Brandon Liu, who constructed and maintains one of many central items of this ecosystem, calls this new period “post-scarcity web mapping,” and he makes a compelling case for taking true possession of our mapping instruments.
So, if we wish to host our personal maps, how can we do it?
Again to (map) faculty
Earlier than I dive into this, it’s value remembering what an online map is and the way it works. The perfect free useful resource on that is mapschool.io. Go evaluation that if you happen to want a refresher on projections, rasters versus vectors, heatmaps versus choropleths and different cartography fundamentals. This explanation of vector tiles can also be value studying. Maps are sophisticated and it’s okay to really feel behind.
Information for mapping
A map is a illustration of information, however what knowledge? There’s a variety of choices, with an rising array of open and permissively licensed knowledge to attract from. Some key assets that can be significantly helpful for self-hosters:
OpenStreetMap is a free map of the world, created by volunteers and obtainable beneath a permissive license. Mapbox is constructed on OSM (and constructed the id editor used replace the underlying knowledge). Geofabrik supplies extracts of OSM knowledge from continents all the way down to US states.
Natural Earth supplies free downloads of cultural, bodily and raster knowledge helpful for extra zoomed-out maps. It’s place to get nationwide, state and generally county borders, metropolis places, coastlines and shaded reliefs.
Authorities sources: US Census, US Geological Survey, NOAA and different authorities businesses have a number of information too quite a few to listing right here. I’m often layering this on prime of OSM and Pure Earth.
Overture is a brand new challenge that mixes OSM, constructing outlines and point-of-interest (POI) knowledge in an interoperable format. POI knowledge has been particularly arduous to get till now, as a result of it has the obvious worth to entrepreneurs. Google, Mapbox and Foursquare have POI knowledge, nevertheless it’s costly and restricted.
Tiling
Maps you possibly can pan and zoom work by storing knowledge in sq. tiles, often 256 or 512 pixels per aspect. These will be raster photos or vector knowledge, and they are often saved in numerous archive codecs. The primary two codecs lately are MBTiles and PMTiles.
MBTiles is the older format, created by Mapbox and utilized in manufacturing since round 2010. It shops tile knowledge in a single SQLite database. It’s widely supported at this level, and something that may learn from SQLite (which is to say, most something) can learn tile knowledge. Drop an MBTiles file into QGIS and it’ll render a map. The one draw back to MBTiles is that it wants a server.
PMTiles is newer, a part of Brandon Liu’s Protomaps challenge. Like MBTiles, it’s a single file format for vector or raster knowledge, nevertheless it has one distinct benefit: it’s optimized for serving over HTTP utilizing byte-range requests. Put a PMTiles file on Amazon S3, Google Cloud Storage or perhaps a VPS working nginx and you’ve got tiled map knowledge.
The simplest place to begin is by downloading pre-built map tiles. MapTiler offers downloads of planet- or regional-scale vector and raster tiles in MBTiles format, plus a server to run regionally (or wherever). It’ll even serve raster photos that work with Leaflet maps.
Protomaps presents daily builds of the entire planet constructed utilizing OSM and Pure Earth. It’s a large single file – about 111 GB every – however you don’t essentially want the entire thing. The pmtiles CLI utility can extract tiles from a bounding field or GeoJSON area. For a sufficiently small space, tiles can slot in Git Large File Storage and a full map stack can reside on GitHub Pages.
The pre-tiled knowledge you obtain from MapTiler or Protomaps will produce a reasonably normal road map that can work as a base for many visualizations. However if you happen to want one thing extra particular, or with totally different knowledge layered in, there are three instruments that can tile knowledge regionally: Planetiler, tilemaker and tippecanoe.
Planetiler is how Protomaps generates every day builds. As its identify implies, it’s constructed to generate a planet’s value of tiles, although it will possibly additionally end up a smaller area. It makes use of a sophisticated Java-based construct system, nevertheless it’s extremely quick. After you have all of your knowledge downloaded (which might take some time), a high-end machine can generate a planet tileset in beneath an hour.
Tilemaker is an identical challenge that builds a single executable and has a less complicated configuration setup (utilizing JSON and Lua). It isn’t supposed to work at planet-scale, however for smaller areas, it’s a lot quick.
Tippecanoe is the oldest of the three instruments. Erica Fischer created it whereas working at Mapbox and now maintains it at Felt. Tippecanoe is what I take advantage of to tile knowledge layers, like on this wildfire map. It has extra knobs and dials that may tweak the best way your knowledge results in tile layers, nevertheless it’s additionally going to take the longest to generate tiles.
All three instruments can generate MBTiles or PMTiles.
Styling and rendering
At this level, we’ve discovered knowledge and cut up it into tiles. To show it into an precise map, we want a mode and a strategy to render it in an online browser.
Rendering is easy: Use Maplibre, the open-source fork of Mapbox GL. Whereas the 2 initiatives have diverged considerably for the reason that fork, familiarity with Mapbox will translate simply sufficient to Maplibre.
To render your map, you want a style:
A MapLibre model is a doc that defines the visible look of a map: what knowledge to attract, the order to attract it in, and learn how to model the info when drawing it. A mode doc is a JSON object with particular root stage and nested properties. This specification defines and describes these properties.
You’ll be able to write this JSON doc by hand, however that’s solely sensible for the best maps. A full road map could have dozens of layers.
Fortuitously, there are current kinds obtainable, freed from cost and permissively licensed. OpenMapTiles supplies each styles and a schema.
Each MapTiler and Protomaps supply cheap defaults for a map. This wildfire map and this map of areas covered by the Radiation Exposure Compensation Act use Protomaps and its basemaps library. Tileserver comes packaged with OpenMapTiles kinds.
For additional customization, there’s Maputnik, an open-source editor for MapLibre (and Mapbox) kinds. Sadly, Maputnik can’t work with PMTiles files natively, so it wants an intermediate server that generates x/y/z URLs from a PMTiles or MBTiles file.
Fonts and icons
Map kinds practically all the time want fonts and sprite sheets to show issues like place names, icons and highway shields. This a part of the stack is difficult to see by itself, and it took me longer than I anticipated to determine it out, so I’m together with a devoted part right here.
When you’re utilizing the Protomaps basemaps library, fonts and sprites come bundled (through the Protomaps basemaps-assets repo) and hosted on GitHub Pages.
Kinds from OpenMapTiles embody fonts and sprites hosted by MapTiler, however these would possibly require an API key.
If you wish to use customized fonts and icons, two instruments will generate the information you want: font-maker and spreet.
Font-maker has a web interface and a (largely undocumented) command-line interface. To see how the CLI works, take a look at the create_fonts.sh script within the Protomaps basemaps-assets repo.
Spreet has its personal tutorial and many examples.
Internet hosting and serving
When you’ve made it this far, congratulations, you need to now have geographic knowledge, became tiles, a mode to render it and perhaps customized fonts and icons. This can be a lot, and it’s a part of the rationale there are corporations able to promote you a month-to-month plan that can do all this for you. There’s no disgrace in paying for a service that saves you effort and time.
However we’ve got a map now, so the place are we going to place it?
When you’re utilizing MBTiles, you’ll want a server. Fly.io will host something you possibly can put in a Docker container.
When you generated PMTiles as an alternative, any static file server that may serve byte-range requests with CORS headers will work. The Protomaps hosting documentation lists a number of choices and recommends Cloudflare R2. A sufficiently small tileset will match on GitHub Pages, which has a complete size limit of 1 GB.
Examples
The primary time I noticed a information group exploring this method to self-hosted maps was Kevin Schaul’s work with the Washington Post.
I constructed the wildfire map above utilizing Protomaps and Svelte (for UI components). The code is on GitHub.
Simon Willison has a good demo and walkthrough of his course of mapping Half Moon Bay utilizing Protomaps. The tiles are sufficiently small to slot in a GitHub repo.
Evan Wagstaff constructed a demo workflow to generate tiles utilizing Planetiler and Fly.io for NICAR23 in Nashville.
Header picture through OpenStreetMap Americana map model by Brian Sperlongano, licensed within the public area.