TileKiln OSM vector map tiles

¦ OSM vector tiles ¦ Jura Mountains mapping

The OpenStreetMap (OSM) database aims to provide minute-level updates for a global scale tileset. Historically, the raster tiles set served at openstreetmap.org by the OpenStreetMap Carto project using the OSM database illustrate the types of map tiles that can be provided. The project's main use case however is to provide background maps for feedback to OpenStreetMap database editors using both online editors at openstreetmap.org for example or offline editors such as JOSM.

Vector tiles have many avantages over raster tiles and there appears to be considerable interest in using the OSM database for minute-level updated vector tiles. Several OSM vector demo maps are available (TileMaker, Versatiles, Geofabrik, Paul Norman, Altilunium, StreetComplete).

However, the openstreetmap.org map tiles present unique challenges. On the one hand they showcase what can be achieved as a basemap while on the other hand they need to be updated continuously so that changes in the OSM data appear as rapidly as possible, in order to motivate OSM contributors and help in quickly detecting and fixing problems.

Our JuraMap mapping focuses on local- and project-level applications of OSM technologies. With this in mind, and without going into details, there are use cases which would benefit greatly from the availability of minute-level updated vector tiles using a self-hosted OSM database, at least initially.

As demonstrated by OSM community discussion [1, 2, 3, 4], the TileKiln approach for serving OSM vector map tiles being developed by Paul Norman and others using existing technologies (primarily osm2pgsql and the ST_AsMVT functions of the PostGIS extension to the PostgreSQL database) is attracting much attention.

To make a start, we summarise below how TileKiln is used to serve OSM vector tiles. The most straightforwrd starting point is to implement Paul Norman's Street Spirit map style ("Spirit" for short).

Requirements

In our case we are using a fully updated Ubuntu 22.04.4 LTS server on a HPE Proliant DL20 Gen 10 server. For now it is assumed that the Spirit installation guide has been followed and tiles are being served. In other words, executing in a terminal:

tilekiln dev --config spirit.yaml --source-dbname spirit

results in "INFO: Application startup complete." In our case, and probably in most cases, there are requirements additional to those given in the installation guide. These will be summarised at a later stage. Briefly, our configuration is:

Apache2 (version 2.4.52) with http/2 (a2enhttp2) activated.
PostgreSQL 16.2
PostGIS = "3.4.0 0874ea3" [EXTENSION] PGSQL="160" GEOS="3.12.1-CAPI-1.18.1" PROJ="8.2.1" (note GEOS version)
Python 3.10.12

It is assumed that the Spirit github repository has been cloned to a directory of the form "/home/user/spirit" and that a virtual environment has been set up before installing TileKiln:

git clone https://github.com/pnorman/tilekiln.git
cd spirit
python3 -m venv venv
venv/bin/pip install tilekiln
source venv/bin/activate

whereby the Tilekiln command-line utilities can be executed in a terminal using commands at "/home/user/sprit" of the form:

tilekiln/bin/tilekiln xxx --option1 --option2

At this early stage, the tilekiln "dev" comand is perhaps the most important. It starts a server to live-render tiles with no caching by presenting what is called a tilejson address at "/<id>/tilejson.json" where for convience "/tilejson.json" redirects to the tileson address. The two styles in the Spirit repository are "spirit.yaml" with an id of "v1" and "shortbread.yaml" with an id of "shortbread_v1"

In our case we execute in a terminal:

tilekiln dev --bind-host 192.168.1.102 --bind-port 8000 --config spirit.yaml --source-dbname spirit --base-url https://umap.opennbs.net

which binds the TileKiln tilejson server to "https://umap.opennbs.net".

The Spirit style's tilejson is therefore available at "https://umap.opennbs.net/tilejson.json" or "https://umap.opennbs.net/v1/tilejson.json". The Shortbread style would be available at "https://umap.opennbs.net/shortbread_v1/tilejson.json" but is in fact not implemented.

Serving tiles

In brief, to serve vector tiles, websites serving "glyphs" and "sprites" (fonts and icons to most normal people) are normally needed as well as a website serving the Spirit (or Shortbread) style.

A. Glyphs (fonts)

NotoSans-Regular and NotoSans-Bold fonts are needed for the Spirit style. Noto fonts from Google's Noto global font collection can be downloaded as .ttf files from Font Squirrel and then packed and downloaded as a single .pbf file using Maplibre Font Maker. Each font is stored in a web-site directory (say "fonts") as a directory (called "noto_sans_regular" in the case of NotoSans-Regular) which contains a directory called "Noto Sans Regular" which contains the set of .pbf font files. This directory structure (fonts -> noto_sans_regular -> Noto Sans Regular) seems a bit bizarre, but perhaps there is a reason.

Setting the Apache2 web server to serve the .pbf Content-Type by adding a header ("AddType application/octet-stream .pbf" to apache2.conf) is not needed.

In our case there are some redundent web adresses for the opennbs.net domain which are used for serving the fonts (glyphs) at "https://www.opennbs.net/fonts/{fontstack}/{range}.pbf".

There remain few, if any, public facing glymph sources, even on CDN sites, so providing anything more than the bare minimum of fonts to demonstrate technologies is probably the way forward to avoid having a server overlaoded.

B. Sprites (icons)

The Spirit github repository that is cloned to install Spirit includes a directory called "sprites" that contains eight .svg files of standard icons (e.g., train.svg, subway.svg). These icons need to be consolidated into a single "sprite sheet" raster image file and served. For development and testing urposes, the Spirit installation guide suggests using the Basemaps-sprites node package (part of basemaps) combined with the fairly basic python http web server.

Basemaps-sprites is installed in a normal way. For example, with npm with the node version set using nvm:

nvm list
nvm use 18.20 (in our case)
npm install -g @basemaps/sprites

The sprites are served by executing in a new terminal:

node_modules/.bin/basemaps-sprites sprites && python3 serve.py

Basemaps-sprites by default serves the icons (glyphs) at the localhost port 8080 (i.e., at http://127.0.0.1:8080). Before executing in the new terminal, the "/home/user/spirit/server.py" file should be edited to give in our case:

http.server.test(HandlerClass=CachelessHTTPRequestHandler, port=8787, bind="192.168.1.102")

Port 8787 is then bound in the Apache2 web server to a web address ("https://potree.opennbs.net" in our case).

C1. Serving Spirit vector tiles (for editing purposes)

Finally the Spirit style must be served. In the case that the style is being edited and with the servers in place for a sprite and for glyphs, the Spirit installation guide suggests using Charites, a command-line tool for writing, manipulating and serving vector map styles (for TileKiln, Charites pre-processes several files to create a single Maplibre GL file). It is installed using:

npm install @unvt/charites @basemaps/sprites

The Spirit "style.yaml" is served by executing in a new terminal:

node_modules/.bin/charites serve style.yaml --port 8585

where the port is changed from the default port to a local port (e.g., 127.0.0.1:8585) by giving the port as a command-line option. The port is bound in the Apache2 web server to https://aoi.peterboswell.net to serve tiles in the .mvt format.

For the Spirit style, the initial map "center" and "zoom" can be specified in "spirit.yaml" (e.g., for a map centred on Andorra as "center: [1.573,42.56,8]", noting that longitude comes before latitude à la MapLibre GL). Normally the map centre will be specfied in the client used to view the map (MapLibre GL in our case, see below).

Charites uses maplibre-gl.js to read PMTiles and "/home/user/spirit/node_modules/@unvt/charites/provider/default/index.html" can be changed to the latest version ("https://unpkg.com/maplibre-gl/dist/maplibre-gl.js'") probably provided the legend and another module are removed ("shared.js" and "app.js") in "index.html".

Using Charites has been explained in some detail by Hidenori Fujimura (Vector styling with unvt/charites at Lnkd.in) and initial files for setting up the yaml structure are available. Adding 3d-terrain is demonstrated here.

To check that everything is working, change a style. For example, in "/home/user/spirits/style/food.yaml" change the "icon-size" of the first object from 1 to 100. A resturant point-of-interest on the map should become very large.

The various configurations for the Spirit files are given below. Indenting is not indicated but is needed - please see the Spirit repository versions).

For spirit.yaml (for TileKiln):

metadata:

id: v1

name: Street Spirit

version: 0.0.1

center: [1.573,42.56,8]

...........

style.yaml (for Charites):

name: Spirit

version: 8

sources:

spirit:

type: vector

url: https://umap.opennbs.net/tilejson.json

sprite: https://potree.opennbs.net/sprites

glyphs: https://www.opennbs.net/fonts/noto_sans_regular/{fontstack}/{range}.pbf

layers:

- !!inc/file style/background.yaml

......

C2. Serving Spirit vector tiles (for website)

Charites can be used to export a style.json which is then used with an index.html file on a webserver, as indicated below:

node_modules/.bin/charites build style.yaml spirit.json

TileKiln vector maps are now served https://www.opennbs.net. Please note that we are mainly interested in using high zoom levels.

index.html (at /var/www/html/tiles)

<!DOCTYPE html>

var map = new maplibregl.Map({container: 'map', style: 'spirit.json', center: [1.573,42.56], zoom: 8, hash: true, maxZoom: 18});

</script></body></html>

E. Serving Shortbread vector tiles

As pointed out by Paul Norman in the TileKiln installation guide, serving Shortbread styled vector map tiles is somewhat easier to set up than for the Spirit style. This is because glyphs (fonts) and sprites (icons) for the shortbread style are served elsewhere (initially using versatiles.org sources but this appears to be changing or discontinued) and the style is available as a file ("colorful.json") for loading by the web client (Malibre GL).

A TileKiln tilejson is served using:

tilekiln dev --bind-host 192.168.1.102 --bind-port 8000 --config shortbread.yaml --source-dbname spirit --base-url https://umap.opennbs.net

It gives a tilejson at "https://umap.opennbs.net/tilejson.json" (which redirects to "https://umap.opennbs.net/shortbread_v1/tilejson.json") .

maplibre-gl.js in a web page is used as the client. For example, at "https://www.opennbs.net/tiles" we would have a "/var/www/html/tiles/index.html" web page as indicated above and its "colorful.json" as follows:

colorful.json Shortbread settings (for MapLibre GL; our version):

"glyphs": "https://www.opennbs.net/fonts/noto_sans_regular/{fontstack}/{range}.pbf",

"sprite": "https://potree.opennbs.net/sprites",

"sources": {

"versatiles-shortbread": {

"type": "vector",

"url": "https://umap.opennbs.net/tilejson.json"

.........

colorful.json Shortbread settings (for MapLibre GL; see pnorman.github.io):

"glyphs": "https://tiles.versatiles.org/assets/fonts/{fontstack}/{range}.pbf",
"sprite": "https://tiles.versatiles.org/sprites/sprites",
"sources": {
"versatiles-shortbread": {
"type": "vector",
"url": "https://demo.tilekiln.xyz/shortbread_v1/tilejson.json"

..........

A few other demos of TileKiln-served Spirit or Shortbread vector tiles have been announced (e.g., pnorman.github.io, altilunium.github.io) but are often not working, possibly because development is underway and/or glymph and sprite sources are not available. tilejson sources on the other hand are often available (e.g., demo.tilekiln.xyz).

In the case of the "colorful.json" style that is based on the VersaTiles Colorful stylesheet, an interesting demo of composting two vector tile sources has been developed by Minh Nguyen (see maps) for which the composite colorful.json is available (the sources are indicated below).

colorful.json Shorbread settings (for MapLibre GL composite tile; see github.com/1ec5)

"glyphs": "https://tiles.versatiles.org/assets/fonts/{fontstack}/{range}.pbf",

"sprite": "https://tiles.versatiles.org/assets/sprites/sprites",

"sources": {

"openhistoricalmap": {

"type": "vector",

"tiles": ["https://vtiles.openhistoricalmap.org/maps/osm/{z}/{x}/{y}.pbf"]

"versatiles-shortbread": {

"type": "vector",

"url": "https://demo.tilekiln.xyz/shortbread_v1/tilejson.json"

.............

Next steps

Having more-or-less understood how TileKiln works, for the tile generation and rendering stack the next steps will probably requre investigating storage, caching and a Cron task scheduler to run osm2pgsql to update the Spirit database automatically and fairly frequently from the OSM (how frequently is unclear). The OSM database is managed using the usual Rails Port with a PostgreSQL 12.3 database and PostGIS 3.2 (GEOS 3.7.1). As we are interested in project- and local-level use cases we shall probably upload, at least initially, an entire OSM database to the Spirit database instead of minute updates.

7 April 2024