Bootstrapping a small application¶

This section is a bit different

This section of the guidebook is a little different from the others in that it outlines the how a new "small" web application was bootstrapped from start to finish.

In this section, we're going to cover the steps which are necessary to bootstrap a new web application based on our Django boilerplate and then to host it in Google Cloud. This guide is not presented in strict chronological order but is instead grouped thematically. When possible links to individual commits or merge requests are included for context.

This section is listed as a "tutorial" in that you can follow the steps described in it and end up with a fully deployed application. Unlike other tutorials in the guidebook it's not intended to be followed line-by-line but to give examples of the work required to get to the desired outcome.

This text was written in November 2023 and reflects the state of the art at the time of writing. Things in DevOps are always moving forward so, where a future direction of travel is known, this is indicated.

The general plan¶

The application itself is conceptually simple. It is a small application which allows sufficiently authorised users to create "ballots". Each ballot has a time period when it is open, a "sharing URL" and a mapping between eligible voter CRSids and "voting URLs".

When a potential voter visits the ballot’s sharing URL, they are authenticated and, if their CRSid is in the mapping of eligible voters, they’re redirected to the voting URL.

The plan was to make a very small Django app. It didn’t need any React-based UI. Nor did it need a REST API. It would be a traditional web application which renders HTML responses for incoming requests.

How the application is developed and deployed¶

As of November 2023 the most recent version of our development and deployment process is as follows:

Releases are managed using our release automation CI template. As features/fixes are merged, a Merge Request representing the "next" release is automatically maintained. Merging this Merge Request causes a new release to be made. See, as an example, the 0.7.0 release Merge Request.
When a release Merge Request is merged, a new release is made. This includes an update to the Changelog, recording the release in the GitLab project, tagging the repository and building and pushing an appropriately tagged Docker image to the Google Artefact repository. See, as an example, the 0.7.0 release on GitLab and the corresponding CI pipeline.
Releases of the web application code itself are versioned using semantic versioning. The version number is present in the pyproject.toml file and is automatically incremented by the release automation.
Conventional commits are used to indicate which commits add new features, fix bugs or introduce backwards compatibility changes necessitating a new major version.
The terraform configuration for deployment has a list of which image tags should be deployed in which environment. So, to deploy version 0.5.0 of the web application to the "staging" environment, there is first a commit which updates the terraform configuration.
When the terraform change lands in the main branch, a CI pipeline is created which generates a terraform plan against staging, production and the development environment. Staging is applied automatically. Production and deployment can be deployed manually using the play buttons on the environment page.

Things which need other people to complete¶

This section talks about the things which will probably need to be done by others either within DevOps or without.

Creating GitLab projects¶

In reality this was done "backwards" and made good later but, conceptually, this was done by opening a Merge Request (MR) on the GitLab project factory project to create a GitLab group and two projects: one for the web application itself and one for the terraform deployment. Once merged, that MR was applied by a member of the Cloud Team.

Creating Google Cloud projects¶

Since this application is hosted via Cloud Run, it needs Google Cloud projects to deploy into. An MR was opened on the google product factory and, once it was approved and merged, it was deployed by the Cloud Team.

This created one Google Cloud project for each environment: "production", "staging" and "development". Information on the projects was written to a JSON file stored in a Google Cloud bucket and is read by our deployment boilerplate code in order to configure itself.

To create Google Cloud projects, you’ll need a cost code. This can be obtained from the budget holder or, in extremis, the Head of DevOps.

Requesting a DNS name¶

Ultimately we want our application to be hosted under https://ballots.apps.cam.ac.uk/. There is guidance on the UIS website about how to request domain names for cross-University apps. In short it requires filling out an application form and waiting for a decision. The justification used in this case for needing the domain was a) the purpose of the application, b) it being a webapp and so belonging under .apps.cam.ac.uk and c) it being a cross-University application.

DNS within the University is a complex topic. There is a dedicated section of this guidebook devoted to it.

Creating GitLab runners¶

Once the GitLab and Google Cloud projects were created, an MR was opened in the GitLab runner project to add Google Cloud-hosted GitLab runners. These runners are GitLab CI runners which have an associated Google Cloud service account identity and so can be given permission to push container images to the Google Artefact Registry. This is needed by our boilerplate CI configuration. Once applied, the MR was deployed by the Cloud Team.

The Django application¶

Our webapp boilerplate is configurable: you can specify if you want to generate a "pure" API backend for the API Gateway or if you want to have an application which hosts an API for a React-based UI. What you couldn’t do, until this application was developed, was to generate a traditional "request-render HTML response" application. From a Django point of view, this is just a pure API application with the api module removed and Django social auth put back in.

An MR was opened which tweaked the webapp boilerplate to allow generating such an application and the output from this version of the boilerplate formed the initial commit for the application.

The application itself was created following the instructions in the boilerplate project and committed to GitLab.

Boilerplate changes¶

While developing the application, a small bug was found in our boilerplate’s CI and so the next commit was porting that change.

There are also some other boilerplate changes which may end up being folded into the standard boilerplate.

Firstly, some configuration changes were added which meant that production containers would output logs in JSON format. This makes it a lot easier to search for logs in the Google Cloud console or to set up log-based metrics.

A more subtle change was to tweak where container images are pushed. Previously we’d create a new container registry for each branch in GitLab and push images to it. This is the default "out of the box" configuration for GitLab’s AutoDevOps but leads to an annoying set of "dangling" container image registries. Even with container registry expiration policies, you’ll still end up with a repository with a several hundred megabyte image in it for a branch no-one has looked at for several years.

This change was accompanied with an MR in the GitLab project factory which set a suitable container registry expiration policy.

The small tweak means that images are all pushed into the same registry. This means that "latest" tag no-longer has any meaning beyond "the most recent image built" but since we’re a) deploying by explicit image tags and b) the GitLab container registry is not the canonical source of images, this is OK.

Automated release management was added via a single commit alongside another commit which added commitlint support so that the conventional commit specification was followed.

For this application, we wanted to explore authentication methods beyond Google. As such, there is a commit which implements SAML-based sign in via Raven SAML2. This had the virtuous effect of removing the need for manual creation of Google OAuth2 credentials. As of writing, we don’t have a good self-service story for Azure Entra ID application credential creation and management. Once these credentials can be managed by terraform, we can move to using OAuth2 authentication via Azure Entra.

Aside from those tweaks, the application README and CHANGELOG files were updated.

The application itself¶

This entire section is more about the process than the application itself but it’s worth mentioning some things about the implementation of the specific application in passing.

The application’s data model is fairly traditional but it does leverage Django’s admin UI for use by non-superuser users. To that end, the admin has been customised to an unusual degree adding custom text, changing the login box to include a link to Cambridge sign in and providing a custom form for the ballot management process.

Since Raven SAML2 provides a list of Lookup group ids as assertions at sign in time, some code using django-rules was created to allow Lookup group membership to represent roles within the application. Group membership was used to determine which users had superuser roles, which had staff roles and which had "ballot manager" roles. The ballot manager role unpacks to a set of atomic permissions.

The deployment¶

The deployment configuration is pretty much just our deployment boilerplate. Since terraform fetches everything it needs about Google projects, DNS, etc at deployment time from the Google Cloud product factory config, the remaining work is to "fill in the gaps". Notable changes include the following.

Pre-deployment database migration¶

A pre-deployment job was added which runs database migrations before a new release is deployed. This replaces our older method of baking the database migrations as part of the application start up. For new Django applications, adding this pre-deploy job is essential.

Container tagging¶

Since the CI was updated to remove branch names from container image paths, the logic around selecting an image from the artefact registry to deploy was updated.

README changes¶

The README was updated to note configuration changes which differed between environments.

SAML keys and certificates¶

Terraform configuration was added to create a SAML private key and public certificate for the application. These were passed as part of the application configuration.

Lookup group-based roles¶

The application supports granting roles within the application based on Lookup group membership. The configuration was updated to allow these role-granting groups to differ between environments. The Lookup group ids themselves were passed as application settings.

DNS configuration¶

After the ballots.apps.cam.ac.uk registration was approved, it was added to the UIS-DEVOPS MZONE by hostmaster.

A DevOps user with rights to modify that MZONE performed the following actions on the ipreg interface:

On the vbox_ops page, the following vboxes were added:
- webapp.prod.rh-ballots.gcp.uis.cam.ac.uk
- webapp.test.rh-ballots.gcp.uis.cam.ac.uk
- webapp.devel.rh-ballots.gcp.uis.cam.ac.uk
On the cname_ops page, the following CNAMEs were created:
- ballots.apps.cam.ac.uk with a target of webapp.prod.rh-ballots.gcp.uis.cam.ac.uk, and
- staging.ballots.apps.cam.ac.uk with a target of webapp.test.rh-ballots.gcp.uis.cam.ac.uk.
- development.ballots.apps.cam.ac.uk with a target of webapp.devel.rh-ballots.gcp.uis.cam.ac.uk.

What is a vbox?

A "vbox" is a record in ipreg which is used to indicate that a hostname exists but is not directly managed by the ipreg system. In this case, the DNS records are managed in Google Cloud. We need to create a vbox record in order to be able to add a CNAME record pointing to it.

The DNS entries are created at 53 minutes past the hour and so there was then a delay of one hour while the DNS records were updated.

Once the records appeared in the public DNS, the terraform deployment configuration was updated with a single commit to add the domain names to the list of domain names for the Google managed certificates. Since the DNS records resolve, via the CNAME, to the Google Cloud load balancer IP, Google is happy to issue certificates without further verification.

Summary¶

This section outlined the main tasks required to bootstrap a brand new application from conception into being deployed into Google Cloud.