Tutorial: code-walkthrough

code-walkthrough

High-level code walkthrough

Code inventory and testing strategy

The Shields codebase is divided into several parts:

  1. The frontend
    1. frontend
  2. The badge renderer (which is available as an npm package)
    1. badge-maker
  3. The base service classes (about 8% of the code, and probably the most important code in the codebase)
    1. core/base-service
  4. The server code and a few related odds and ends
    1. core/server
  5. Helper code for token pooling and persistence (used to avoid GitHub rate limiting)
    1. core/token-pooling
  6. Service common helper functions (about 7% of the code, and fairly important since it’s shared across much of the service code)
    1. *.js in the root of services
  7. The services themselves (about 80% of the code)
    1. *.js in the folders of services

The tests are also divided into several parts:

  1. Unit and functional tests of the badge renderer
    1. badge-maker/**/*.spec.js
  2. Unit and functional tests of the core code
    1. core/**/*.spec.js
  3. Unit and functional tests of the service helper functions
    1. services/*.spec.js
  4. Unit and functional tests of the service code (we have only a few of these)
    1. services/*/**/*.spec.js
  5. End-to-end tests for the frontend
    1. cypress/e2e/*.cy.js
  6. The service tester and service test runner
    1. core/service-test-runner
  7. The service tests themselves live integration tests of the services, and some mocked tests
    1. *.tester.js in subfolders of services
  8. Integration tests of PostgreSQL-backed persistence code
    1. core/token-pooling/sql-token-persistence.integration.js
  9. Integration tests of the GitHub authorization code
    1. services/github/github-api-provider.integration.js

Our goal is to reach 100% coverage of the code in the frontend, core, and service helper functions when the unit and functional tests are run.

Our test strategy for the service code is a bit different. It’s primarily based on live integration tests. That’s because service response formats can change, and when they do the badges break. We want our tests to fail when this happens. That way we can fix the problems proactively instead of waiting for users to report them. There’s a good discussion about this decision in #927. It’s acceptable to write mocked tests of logic that is difficult to reach using live tests, however where possible, it’s preferred to test this kind of logic through unit tests (e.g. of render() and transform() functions).

Server initialization

  1. The server entrypoint is server.js which sets up error reporting, loads config, and creates an instance of the server.

  2. The Server, which is defined in core/server/server.js, is based on the web framework Scoutcamp. It creates an http server, sets up helpers for token persistence and monitoring. Then it loads all the services, injecting dependencies as it asks each one to register its route with Scoutcamp.

  3. The service registration continues in BaseService.register. From its route property, it derives a regular expression to match the route path, and invokes camp.route with this value.

  4. At this point the situation gets gnarly and hard to follow. For the purpose of initialization, suffice it to say that camp.route invokes a callback with the four parameters ( queryParams, match, end, ask ) which is created in a legacy helper function in legacy-request-handler.js. This callback delegates to a callback in BaseService.register with three different parameters ( queryParams, match, sendBadge ), which then runs BaseService.invoke. BaseService.invoke instantiates the service and runs BaseService#handle.

Downstream caching

  1. In production, the majority of requests are served from caches, including the browser cache, GitHub’s camo proxy server, and other downstream caches.
  2. The Shields servers sit behind the Cloudflare CDN. The CDN itself handles about 40% of the HTTPS requests that come in.
  3. The remaining requests are proxied to one of the servers.
  4. See the production hosting documentation for a fuller discussion of the production architecture.

How the server makes a badge

  1. An HTTPS request arrives. Scoutcamp inspects the URL path and matches it against the regexes for all the registered routes until it finds one that matches. (See Initialization above for an explanation of how routes are registered.)
  2. Scoutcamp invokes a callback with the four parameters: ( queryParams, match, end, ask ). This callback is defined in legacy-request-handler. A timeout is set to handle unresponsive service code and the next callback is invoked: the legacy handler function.
  3. The legacy handler function receives ( queryParams, match, sendBadge ). Its job is to extract data from the regex match and queryParams, and then invoke sendBadge with the result.
  4. The implementation of this function is in BaseService.register. It works by running BaseService.invoke, which instantiates the service, injects more dependencies, and invokes BaseService.handle which is implemented by the service subclass.
  5. The job of handle(), which should be implemented by each service subclass, is to return an object which partially describes a badge or throw one of the handled error classes. "Partially rendered" most commonly means a non-empty message and an optional color. In the case of the Endpoint badge, it could include many other parameters. At the time of writing the handled error classes were NotFound, InvalidResponse, Inaccessible, InvalidParameter, and Deprecated. Throwing any other error is a programmer error which will be reported and described to the user as a shields internal error.
  6. A typical handle() function delegates to one or more helpers to handle stages of the request:
    1. fetch: load the needed data from the upstream service and validate it
    2. transform: pluck, convert, or summarize the response format into a few properties which will be displayed on the badge
    3. render: given a few properties, return a message, optional color, and optional label.
  7. When an error is thrown, BaseService steps in and converts the error object to renderable properties: { isError, message, color }.
  8. The service invokes coalesceBadge whose job is to coalesce query string overrides with values from the service and the service’s defaults to produce an object that fully describes the badge to be rendered.
  9. sendBadge is invoked with that object. It does some housekeeping on the timeout. Then it renders the badge to svg or raster and pushes out the result over the HTTPS connection.