Cloud Run

Overview

GCP Cloud Run is an managed way to run containers. Here we are deploying the Buz image to GCP Artifact Registry. The schemas are maintained in GCP bucket and the Buz Configurations are maintained as a GCP Secret. Data processed by buz are sent to one of two Pub/Subs.

Primary Resources

• Buz (Cloud Run)
• Artifact Registry
• Secret
• Bucket
• 2 Pub/Sub

Console Deploy

Here are the steps to set up Buz via the GCP Console.

1. Create Pub/Sub topics.

Create buz-valid and buz-invalid Pub/Sub topics:

The result should look like:

Yo

It is entirely possible to only use one output topic but if you want the upside of redirecting events that fail validation out of the "happy path", two topics are necessary.

2. Upload config to Secret Manager.

For the sake of keeping your secrets a.. secret.. uploading the entire Buz config yml to Secret Manager is the easiest way forward.

YO

We've provided a working config sample that you can copy/paste to Secret Manager here.

Create Buz config as a Secret Manager secret

If all is well you'll see:

Grant the default compute service account appropriate iam access. It will need the Secret Manager Secret Accessor role:

Yo

• While this example uses the default compute service account you'll probably want to create a dedicated service user.

3. Push image to GCP Artifact Registry.

Create a Docker repository in GCP Artifact Registry if you don't have one yet:

Auth to newly-created registry

gcloud auth configure-docker us-east1-docker.pkg.dev
Adding credentials for: us-east1-docker.pkg.dev
....
Docker configuration file updated.

Pull the latest Buz image from the Github container registry:

docker pull ghcr.io/silverton-io/buz:v0.11.11 --platform linux/amd64

AMD64

• At the time of writing Google Cloud Run doesn't support ARM64-based images so you'll need to grab the AMD64 image.

Tag and push the latest Buz image to Artifact Registry:

Tag:

docker tag ghcr.io/silverton-io/buz:v0.11.11 us-east1-docker.pkg.dev/silverton-docs/registry/buz:v0.11.11

Push:

docker push us-east1-docker.pkg.dev/silverton-docs/registry/buz:v0.11.11

Use your own Artifact Registry URL

This example uses the Silverton registry url - you'll need to use your own.

It's structured as: $ARTIFACT_REGISTRY_URL/$GCP_PROJECT/$REGISTRY_NAME/buz:$VERSION

---

4. Run Buz as a Cloud Run service.

Create a new Buz service:

Verify service is running (using out-of-the-box metrics and logs):

Yo

• Log verbosity is cranked in the example configuration. You'll probably want less.
• The stdout sink is included for feedback purposes. You'll probably want to turn it off.
• The above screenshots are all GCP Cloud Run defaults. You'll probably want to tune them.

Bonus

Map a custom domain to Buz

Yo

While this step is technically optional, some Buz functionality like server-side identity cookies will not work without it.

It takes a minute to map a domain/subdomain to a GCP Cloud Run service. Here's how to do it.

Add mapping:

Follow directions to update your dns records:

Set up a GCS schema registry backend

Yo

• While this step is optional, you'll need to do it when using custom schemas.
• Buz includes an onboard schema registry that supports many cache backends, so you can just as easily use a different backend.

Create a GCS bucket for schemas:

Copy schemas to the new schema bucket using gsutil:

(From buz root)

buz ❯❯❯ gsutil cp -r schemas/*  gs://$THE_BUCKET_YOU_JUST_CREATED

Reconfigure Buz with a new schema registry backend:

registry:
  backend:
    type: gcs
    bucket: $THE_BUCKET_YOU_JUST_CREATED
    path: /

---

Push events to BigQuery using a Pub/Sub Subscription

With the announcment of BigQuery Subscriptions pushing events straight to BigQuery is easier than ever.

Terraform

Github Terraform