Cloud Run
Overview
GCP Cloud Run is an managed way to run containers. Here we are deploying the Buz image to GCP Artifact Registry. The schemas are maintained in GCP bucket and the Buz Configurations are maintained as a GCP Secret. Data processed by buz are sent to one of two Pub/Subs.
Primary Resources
- Buz (Cloud Run)
- Artifact Registry
- Secret
- Bucket
- 2 Pub/Sub
Console Deploy
Here are the steps to set up Buz via the GCP Console.
1. Create Pub/Sub topics.
Create buz-valid
and buz-invalid
Pub/Sub topics:
The result should look like:
It is entirely possible to only use one output topic but if you want the upside of redirecting events that fail validation out of the "happy path", two topics are necessary.
2. Upload config to Secret Manager.
For the sake of keeping your secrets a.. secret.. uploading the entire Buz config yml to Secret Manager is the easiest way forward.
We've provided a working config sample that you can copy/paste to Secret Manager here.
Create Buz config as a Secret Manager secret
If all is well you'll see:
Grant the default compute service account appropriate iam access. It will need the Secret Manager Secret Accessor
role:
- While this example uses the
default compute service account
you'll probably want to create a dedicated service user.
3. Push image to GCP Artifact Registry.
Create a Docker repository in GCP Artifact Registry if you don't have one yet:
Auth to newly-created registry
gcloud auth configure-docker us-east1-docker.pkg.dev
Adding credentials for: us-east1-docker.pkg.dev
....
Docker configuration file updated.
Pull the latest Buz image from the Github container registry:
docker pull ghcr.io/silverton-io/buz:v0.11.11 --platform linux/amd64
- At the time of writing Google Cloud Run doesn't support ARM64-based images so you'll need to grab the AMD64 image.
Tag and push the latest Buz image to Artifact Registry:
Tag:
docker tag ghcr.io/silverton-io/buz:v0.11.11 us-east1-docker.pkg.dev/silverton-docs/registry/buz:v0.11.11
Push:
docker push us-east1-docker.pkg.dev/silverton-docs/registry/buz:v0.11.11
This example uses the Silverton registry url - you'll need to use your own.
It's structured as: $ARTIFACT_REGISTRY_URL/$GCP_PROJECT/$REGISTRY_NAME/buz:$VERSION
4. Run Buz as a Cloud Run service.
Create a new Buz
service:
Verify service is running (using out-of-the-box metrics and logs):
- Log verbosity is cranked in the example configuration. You'll probably want less.
- The stdout sink is included for feedback purposes. You'll probably want to turn it off.
- The above screenshots are all GCP Cloud Run defaults. You'll probably want to tune them.
Bonus
Map a custom domain to Buz
While this step is technically optional, some Buz functionality like server-side identity cookies will not work without it.
It takes a minute to map a domain/subdomain to a GCP Cloud Run service. Here's how to do it.
Add mapping:
Follow directions to update your dns records:
Set up a GCS schema registry backend
- While this step is optional, you'll need to do it when using custom schemas.
- Buz includes an onboard schema registry that supports many cache backends, so you can just as easily use a different backend.
Create a GCS bucket for schemas:
Copy schemas to the new schema bucket using gsutil:
(From buz root)
buz ❯❯❯ gsutil cp -r schemas/* gs://$THE_BUCKET_YOU_JUST_CREATED
Reconfigure Buz with a new schema registry backend:
registry:
backend:
type: gcs
bucket: $THE_BUCKET_YOU_JUST_CREATED
path: /
Push events to BigQuery using a Pub/Sub Subscription
With the announcment of BigQuery Subscriptions pushing events straight to BigQuery is easier than ever.