TileKiln OSM vector map tiles

¦ OSM vector tiles ¦ Jura Mountains mapping

NOTE 1

The JuraMap aims to represent the southwestern region of Switzerland's Jura Mountains as it was in about 1898. All likely roads, tracks and paths at the time that had utility value other than logging and moving cattle are mapped as tracks. Small parts of this 1898 network became part of the modern road network shown in the 2023 aerial image layer. Forested areas shown on the 1898 map corresponded fairly closely to those displayed in the 2023 aerial image. It is available in both vector and raster versions.

NOTE 2

For Firefox, please turn off tracking protection (Select the shield icon to the left of the URL of the webpage. A drop-down list will appear, listing the protections for the JuraMap website. Click on the button to the right of “Enhanced Tracking Protection” to turn it off.

NOTE 3

These notes were originally for TileKiln 0.4.0 (source) released in March 2024. The development function "tilkiln dev" to serve tiles with live generation now requires the command "tilekiln serve dev". Similarly probably "tilekiln serve live". This change in TileKiln 0.5 followed the removal of the psycopg_pool module that was often difficult to install.

The OpenStreetMap (OSM) database aims to provide minute-level updates for a global scale tileset. Historically, the raster tiles set served at openstreetmap.org by the OpenStreetMap Carto project using the OSM database illustrate the types of map tiles that can be provided. The project's main use case however is to provide background maps for feedback to OpenStreetMap database editors using both online editors at openstreetmap.org for example or offline editors such as JOSM.

Vector tiles have many avantages over raster tiles and there appears to be considerable interest in using the OSM database for minute-level updated vector tiles. Several OSM vector demo maps are available (TileMaker, Versatiles, Geofabrik).

However, the openstreetmap.org map tiles present unique challenges. On the one hand they showcase what can be achieved as a basemap while on the other hand they need to be updated continuously so that changes in the OSM data appear as rapidly as possible, in order to motivate OSM contributors and help in quickly detecting and fixing problems.

Our JuraMap mapping focuses on local- and project-level applications of OSM technologies. With this in mind, and without going into details, there are use cases which would benefit greatly from the availability of minute-level updated vector tiles using a self-hosted OSM database, at least initially.

As demonstrated by OSM community discussion [1, 2, 3, 4], the TileKiln approach for serving OSM vector map tiles being developed by Paul Norman and others using existing technologies (primarily osm2pgsql and the ST_AsMVT functions of the PostGIS extension to the PostgreSQL database) is attracting much attention.

To make a start, we summarise below how TileKiln is used to serve OSM vector tiles. The most straightforwrd starting point is to implement Paul Norman's Street Spirit map style ("Spirit" for short). We start by discussing serving tiles in a development mode (using a terminal command of the "tilekiln dev" form). We then discuss serving in the live mode ("tilekiln live", where tiles are generated if they are not available in storage).

We have in fact found that for three-dimensional terrain mapping (using terrain-rgb tiles) Shortbread-styled tiles perform much better than Spirit-styled tiles. Both are discussed below, but from now on JuraMap vector styles will refer to Shortbread-styled tiles.

Requirements

In our case we are using a fully updated Ubuntu 22.04.4 LTS server on a HPE Proliant DL20 Gen 10 server. For now it is assumed that the Spirit installation guide has been followed and tiles are being served. In other words, executing in a terminal:

tilekiln serve dev --config spirit.yaml --source-dbname spirit

results in "INFO: Application startup complete." In our case, and probably in most cases, there are requirements additional to those given in the installation guide. These will be summarised at a later stage. Briefly, our configuration is:

Apache2 (version 2.4.52) with http/2 (a2enhttp2) activated.
PostgreSQL 16.2
PostGIS = "3.4.0 0874ea3" [EXTENSION] PGSQL="160" GEOS="3.12.1-CAPI-1.18.1" PROJ="8.2.1" (note GEOS version)
Python 3.10.12

It is assumed that the Spirit github repository has been cloned to a directory of the form "/home/user/spirit" and that a virtual environment has been set up before installing TileKiln:

git clone https://github.com/pnorman/tilekiln.git
cd spirit
python3 -m venv venv
venv/bin/pip install tilekiln
source venv/bin/activate

whereby the Tilekiln command-line utilities can be executed in a terminal using commands at "/home/user/sprit" of the form:

tilekiln/bin/tilekiln xxx --option1 --option2

At this early stage, the tilekiln "serve dev" comand is perhaps the most important. It starts a server to live-render tiles with no caching by presenting what is called a tilejson address at "/<id>/tilejson.json" where for convience "/tilejson.json" redirects to the tileson address. The two styles in the Spirit repository are "spirit.yaml" with an id of "v1" and "shortbread.yaml" with an id of "shortbread_v1"

In our case we execute in a terminal:

tilekiln serve dev --bind-host 192.168.1.102 --bind-port 8000 --config spirit.yaml --source-dbname spirit --base-url https://umap.juramap.org

which binds the TileKiln tilejson server to "https://umap.juramap.org".

The Spirit style's tilejson is therefore available at "https://umap.juramap.org/tilejson.json" or "https://umap.juramap.org/v1/tilejson.json". The Shortbread style would be available at "https://umap.juramap.org/shortbread_v1/tilejson.json" but is in fact not implemented.

Serving tiles

In brief, to serve vector tiles, websites serving "glyphs" and "sprites" (fonts and icons to most normal people) are normally needed as well as a website serving the Spirit (or Shortbread) style.

A. Glyphs (fonts)

NotoSans-Regular and NotoSans-Bold fonts are needed for the Spirit style. Noto fonts from Google's Noto global font collection can be downloaded as .ttf files from Font Squirrel and then packed and downloaded as a single .pbf file using Maplibre Font Maker. Each font is stored in a web-site directory (say "fonts") as a directory (called "noto_sans_regular" in the case of NotoSans-Regular) which contains a directory called "Noto Sans Regular" which contains the set of .pbf font files. This directory structure (fonts -> noto_sans_regular -> Noto Sans Regular) seems a bit bizarre, but perhaps there is a reason.

Setting the Apache2 web server to serve the .pbf Content-Type by adding a header ("AddType application/octet-stream .pbf" to apache2.conf) is not needed.

In our case there are some redundent web adresses for the opennbs.net domain which are used for serving the fonts (glyphs) at "https://www.juramap.org/fonts/{fontstack}/{range}.pbf".

There remain few, if any, public facing glymph sources, even on CDN sites, so providing anything more than the bare minimum of fonts to demonstrate technologies is probably the way forward to avoid having a server overlaoded.

B. Sprites (icons)

The Spirit github repository that is cloned to install Spirit includes a directory called "sprites" that contains eight .svg files of standard icons (e.g., train.svg, subway.svg). These icons need to be consolidated into a single "sprite sheet" raster image file and served. For development and testing urposes, the Spirit installation guide suggests using the Basemaps-sprites node package (part of basemaps) combined with the fairly basic python http web server.

Basemaps-sprites is installed in a normal way. For example, with npm with the node version set using nvm:

nvm list
nvm use 18.20 (in our case)
npm install -g @basemaps/sprites

The sprites are served by executing in a new terminal:

node_modules/.bin/basemaps-sprites sprites && python3 serve.py

Basemaps-sprites by default serves the icons (glyphs) at the localhost port 8080 (i.e., at http://127.0.0.1:8080). Before executing in the new terminal, the "/home/user/spirit/server.py" file should be edited to give in our case:

http.server.test(HandlerClass=CachelessHTTPRequestHandler, port=8787, bind="192.168.1.102")

Port 8787 is then bound in the Apache2 web server to a web address ("https://potree.juramap.org" in our case).

C1. Serving Spirit vector tiles (for editing purposes)

Finally the Spirit style must be served. In the case that the style is being edited and with the servers in place for a sprite and for glyphs, the Spirit installation guide suggests using Charites, a command-line tool for writing, manipulating and serving vector map styles (for TileKiln, Charites pre-processes several files to create a single Maplibre GL file). It is installed using:

npm install @unvt/charites @basemaps/sprites

The Spirit "style.yaml" is served by executing in a new terminal:

node_modules/.bin/charites serve style.yaml --port 8585

where the port is changed from the default port to a local port (e.g., 127.0.0.1:8585) by giving the port as a command-line option. The port is bound in the Apache2 web server to serve tiles in the .mvt format.

For the Spirit style, the initial map "center" and "zoom" can be specified in "spirit.yaml" (e.g., for a map centred on Andorra as "center: [1.573,42.56,8]", noting that longitude comes before latitude à la MapLibre GL). Normally the map centre will be specfied in the client used to view the map (MapLibre GL in our case, see below).

Charites uses maplibre-gl.js to read PMTiles and "/home/user/spirit/node_modules/@unvt/charites/provider/default/index.html" can be changed to the latest version ("https://unpkg.com/maplibre-gl/dist/maplibre-gl.js'") probably provided the legend and another module are removed ("shared.js" and "app.js") in "index.html".

Using Charites has been explained in some detail by Hidenori Fujimura (Vector styling with unvt/charites at Lnkd.in) and initial files for setting up the yaml structure are available. Adding 3d-terrain is demonstrated here.

To check that everything is working, change a style. For example, in "/home/user/spirits/style/food.yaml" change the "icon-size" of the first object from 1 to 100. A resturant point-of-interest on the map should become very large.

The various configurations for the Spirit files are given below. Indenting is not indicated but is needed - please see the Spirit repository versions).

For spirit.yaml (for TileKiln):

metadata:

id: v1

name: Street Spirit

version: 0.0.1

center: [1.573,42.56,8]

...........

style.yaml (for Charites):

name: Spirit

version: 8

sources:

spirit:

type: vector

url: https://umap.juramap.org/tilejson.json

sprite: https://potree.juramap.org/sprites

glyphs: https://www.juramap.org/fonts/noto_sans_regular/{fontstack}/{range}.pbf

layers:

- !!inc/file style/background.yaml

......

C2. Serving Spirit vector tiles (for website)

Charites can be used to export a style.json which is then used with an index.html file on a webserver, as indicated below:

node_modules/.bin/charites build style.yaml spirit.json

TileKiln vector maps are now served juramap.org. Please note that we are mainly interested in using high zoom levels.

index.html (at /var/www/html/tiles)

<!DOCTYPE html>

var map = new maplibregl.Map({container: 'map', style: 'spirit.json', center: [1.573,42.56], zoom: 8, hash: true, maxZoom: 18});

</script></body></html>

E. Serving Shortbread vector tiles

As pointed out by Paul Norman in the TileKiln installation guide, serving Shortbread styled vector map tiles is somewhat easier to set up than for the Spirit style. This is because glyphs (fonts) and sprites (icons) for the shortbread style are served elsewhere (initially using versatiles.org sources but this appears to be changing or discontinued) and the style is available as a file ("colorful.json") for loading by the web client (Malibre GL).

A TileKiln tilejson is served using:

tilekiln serve dev --bind-host 192.168.1.102 --bind-port 8000 --config shortbread.yaml --source-dbname spirit --base-url https://umap.juramap.org

It gives a tilejson at "https://umap.juramap.org/tilejson.json" (which redirects to "https://umap.juramap.org/shortbread_v1/tilejson.json") .

maplibre-gl.js in a web page is used as the client. For example, at "juramap.org" we would have a "/var/www/html/tiles/index.html" web page as indicated above and its "colorful.json" as follows:

colorful.json Shortbread settings (for MapLibre GL; our version):

"glyphs": "https://www.juramap.org/fonts/noto_sans_regular/{fontstack}/{range}.pbf",

"sprite": "https://potree.juramap.org/sprites",

"sources": {

"versatiles-shortbread": {

"type": "vector",

"url": "https://umap.juramap.org/tilejson.json"

.........

colorful.json Shortbread settings (for MapLibre GL; see pnorman.github.io):

"glyphs": "https://tiles.versatiles.org/assets/fonts/{fontstack}/{range}.pbf",
"sprite": "https://tiles.versatiles.org/sprites/sprites",
"sources": {
"versatiles-shortbread": {
"type": "vector",
"url": "https://demo.tilekiln.xyz/shortbread_v1/tilejson.json"

..........

A few other demos of TileKiln-served Spirit or Shortbread vector tiles have been announced (e.g., pnorman.github.io, altilunium.github.io) but are often not working, possibly because development is underway and/or glymph and sprite sources are not available. tilejson sources on the other hand are often available (e.g., demo.tilekiln.xyz).

In the case of the "colorful.json" style that is based on the VersaTiles Colorful stylesheet, an interesting demo of composting two vector tile sources has been developed by Minh Nguyen (see maps) for which the composite colorful.json is available (the sources are indicated below).

The latest TileKiln version (0.5.1) is used with the Versatiles Shortbread colorful.json to demonstrate tile serving at juramap.org. This was implemented to make sure that there are no suprises when using the "tilekiln serve dev" command. This Versatiles Shortbread colorful.json implements most of the layers defined by the Shortbread Vector Tile Schema 1.0 which is described as a "basic, lean, general-purpose vector tile schema (definitions for what goes in vector tiles) for OpenStreetMap data .... (that) does not ... cover the full breadth and depth of OpenStreetMap tagging". The "colorful.json" style can be edited using Maputnik (as can Spirit's "spirit.json").

colorful.json Shortbread settings (for MapLibre GL composite tile; see github.com/1ec5)

"glyphs": "https://tiles.versatiles.org/assets/fonts/{fontstack}/{range}.pbf",

"sprite": "https://tiles.versatiles.org/assets/sprites/sprites",

"sources": {

"openhistoricalmap": {

"type": "vector",

"tiles": ["https://vtiles.openhistoricalmap.org/maps/osm/{z}/{x}/{y}.pbf"]

"versatiles-shortbread": {

"type": "vector",

"url": "https://demo.tilekiln.xyz/shortbread_v1/tilejson.json"

.............

F. Live serving of Spirit tiles

In order to serve TileKiln tiles "live" from storage, with a fall back to live generation of a tile if the tile is missing in storage, one needs to set up storage and use a "tilekiln live" command.

Using a termina command in the virtual environment, Postgresql database called say "tiles" is created using

createdbd tiles

The storage database is then initialised with the TileKiln "init" command:

tilekiln storage init --storage-dbname tiles --config spirit.yaml

where the Spirit conguration file (in the directory "/user/spirit/" is the same as "spirit.yaml" used for the "serve dev" server. Four tables should have been created in the "tiles" database together with a metadata entry in the "metadata" table.

The live serving of tiles is started in our case with the command:

tilekiln serve live --config spirit.yaml --bind-host 192.168.1.102 --bind-port 8000 --config spirit.yaml --source-dbname spirit --storage-dbname tiles --base-url https://umap.juramap.org

It is important to ensure that the "spirit.json" referred to by the style in the website's index.html includes the id of the "tilejson.json" since the tileson at "/<id>/tilejson.json" is not redirected to "/tilejson.json". The id ("v1") in our case is referred to in the "tiles" database metadata entry. To be clear, spirit.json in our case must include the source:

"spirit": { "type": "vector", "url": "https://umap.juramap.org/v1/tilejson.json"}

One can test tile generation with a command of the form:

tilekiln generate tiles --config spirit.yaml --source-dbname spirit --storage-dbname tiles

and then type in a tile's z/x/y coordinates (for example, type 13/1541/4974) followed by [control d] to signal the end of standard input. An entry should appear in the "tiles" database "v1" table.

Terrain mapping

At first glance there seems to be a large difference in the way Spirit and Shortbread styled vector maps respond to 3d terrain-rgb mapping. The difference is shown here:

In both cases we use the Swisstopo 0.5m Alti3d digital terrain (elevation) model and the standard terrain-rgb tile creation chain based on rio-rgbify (see BERT, Azure, Syncpoint, xyCarto, UNVT, Maplibre). Maplibre GL is used as the client in the recommended manner (see example).

Most important, for Firefox as the browser, there are single-pixel visual artifacts (see Maplibre issue) with maplibre-gl.js (see jsfiddle.net/1xwgf6pq/), although the artifacts in this case seem to disappear when the map is panned. The artifacts arise from Firefox's “Enhanced Tracking Protection” feature. To turn off tracking protection, select the shield icon to the left of the URL of the webpage. A drop-down list will appear, listing the protections for the website. Click on the button to the right of “Enhanced Tracking Protection” to turn it off.

For Chrome and Explorer, the Spirit tiles seem to interact in some way with the terrain tiles to give bands as one pans the map. Shortbread tiles do not show this effect. The artifacts do not arise from interaction between the various functionalities that are implemented using a Maplibre client.

Next steps

Having more-or-less understood how TileKiln works, for the tile generation and rendering stack the next steps will probably requre investigating caching and a Cron task scheduler to run osm2pgsql to update the Spirit database automatically and fairly frequently from the OSM (how frequently is unclear). The OSM database is managed using the usual Rails Port with a PostgreSQL 12.3 database and PostGIS 3.2 (GEOS 3.7.1). As we are interested in project- and local-level use cases we shall probably upload, at least initially, an entire OSM database to the Spirit database instead of minute updates.

30 June 2024