Tutorial: production-hosting

production-hosting

Production hosting

Production hosting is managed by the Shields ops team:

operations issues

#ops chat room

Component Subcomponent People with access
shields-io-production Full access @calebcartwright, @chris48s, @paulmelnikow
shields-io-production Access management @calebcartwright, @chris48s, @paulmelnikow
Raster server Full access as team members @paulmelnikow, @chris48s, @calebcartwright, @platan
shields-server.com redirector Full access as team members @paulmelnikow, @chris48s, @calebcartwright, @platan
Cloudflare (CDN) Account owner @espadrine
Cloudflare (CDN) Access management @espadrine
Cloudflare (CDN) Admin access @calebcartwright, @chris48s, @espadrine, @paulmelnikow, @PyvesB
Twitch OAuth app @PyvesB
Discord OAuth app @PyvesB
YouTube Account owner @PyvesB
GitLab Account owner @calebcartwright
GitLab Account access @calebcartwright, @chris48s, @paulmelnikow, @PyvesB
DNS Account owner @olivierlacan
DNS Read-only account access @espadrine, @paulmelnikow, @chris48s
Sentry Error reports @espadrine, @paulmelnikow
Metrics server Owner @platan
UptimeRobot Account owner @paulmelnikow
More metrics Owner @RedSparr0w

Attached state

Shields has mercifully little persistent state:

  1. The GitHub tokens we collect are stored in a fly.io postgres database
  2. The server keeps the resource cache in memory. It is neither persisted nor inspectable.

Configuration

To bootstrap the configuration of non-secret settings, we set a single environment variable:

NODE_CONFIG_ENV=shields-io-production

With that variable set, the server (using config) reads these files:

Secrets are supplied directly as environment vars.

Badge CDN

Sitting in front of the three servers is a Cloudflare Free account which provides several services:

  • Global CDN, caching, and SSL gateway for img.shields.io and shields.io
  • Analytics through the Cloudflare dashboard
  • DNS resolution for shields.io (and subdomains)

Cloudflare is configured to respect the servers' cache headers.

Raster server

The raster server raster.shields.io (a.k.a. the rasterizing proxy) is hosted on Heroku. It's managed in the squint repo.

Fly.io Deployment

Both the badge server and frontend are served from Fly.io. Deployments are triggered using GitHub actions in a private repo.

DNS

DNS is registered with DNSimple.

Error reporting

Error reporting is one of the most useful tools we have for monitoring the server. It's generously donated by Sentry. We bundle @sentry/node into the application, and the Sentry DSN is configured via local-shields-io-production.yml (see documentation).

URLs

The canonical and only recommended domain for badge URLs is img.shields.io. Currently it is possible to request badges on both img.shields.io and shields.io i.e: https://img.shields.io/badge/build-passing-brightgreen and https://shields.io/badge/build-passing-brightgreen will both work. However:

  • We never show or generate the img.-less URL format on https://shields.io/
  • We make no guarantees about the img.-less URL format. At some future point we may remove the ability to serve badges on shields.io (without img.) without any warning. img.shields.io should always be used for badge urls.

Monitoring

Overall server performance and requests by service are monitored using Prometheus and Grafana.

Request performance is monitored in two places: