Debugging OAuth authorization tokens from your terminal

Reading Time: Approximately 3 minutes.

Here’s a quick post about something that I don’t do often but is typically a pain when I do: debugging OAuth tokens super quickly. An Example Scenario You’ve just provisioned a shiny new Keycloak instance and want to make it the upstream identity provider (IdP) to your company’s Okta tenant because you were volun-told to do so. You’re also using an app that is configured to do single sign-on through Keycloak and will only log in users if they have admin in their access token’s groups claim. … »

How to empty an AWS Routed53 Hosted Zone with AWS CLI

Reading Time: Approximately 3 minutes.

So you’re managing an AWS VPC with Terraform or something like that, but some other pesky thing decided to add Route53 records that Terraform doesn’t know about. As a result, when you try to delete the zone with Terraform, you’re no longer able to because of this: │ Error: deleting Route53 Hosted Zone (REDACTED): HostedZoneNotEmpty: The specified hosted zone contains non-required resource record sets and so cannot be deleted. │ status code: 400, request id: REDACTED Ughh. … »

Troubleshooting Kubernetes Namespaces That Won't Delete

Reading Time: Approximately 7 minutes.

Usually, deleting Kubernetes namespaces is easy: kubectl delete ns delete-me Sometimes, however, deleting them takes way longer than expected… kubectl delete ns delete-me # still deleting, two months later... This quick “way longer than I acutally ever thought possible” post shows you a few troubleshooting tricks for dealing with this. Forget everything you know about the word “all” kubectl delete --all -n delete-me is a lie. While the kubectl delete man page suggests that “–all” means “all”: … »

Render Golang templates with a timeout

Reading Time: Approximately 2 minutes.

Situation You’re writing a Go program that renders arbitrary Go templates that users can write. Since they are arbitrary, you want to prevent users from accidentally DDoSing your program by using long-running template functions. Something like this: import ( "os" "template" ) // Perhaps this is exposed through an interface that a // third-party API implements, for example. func LongRunningFunction(s string) { time.Sleep(100000000) // This takes forever return s } func main() { tmpl := `Hello, {{ . … »

Run ARM Docker images in GitHub Actions runners!

Reading Time: Approximately 1 minutes.

This is so easy to do, I think I can describe it in less than 150 words! Problem: You want to run Docker images from Docker images that target ARM, or you want to build images for ARM platforms. Solution: Add this to .github/workflows/main.yml (or whichever YAML file you’d like to enable ARM support for): jobs: your-job-name: steps: # Add this to the top of your `steps` - name: Set up QEMU - arm if: ${{ runner. … »

Scraping without JavaScript using Chromium on AWS Lambda: The Novel

Reading Time: Approximately 14 minutes.

UPDATE: 2022-01-17 16:33 CST Forget the below. Just do this instead! UPDATE: 2022-01-15 16:43 CST It appears that Docker as configured within the runners provided by GitHub Actions do not native support building ARM images. However, you can use qemu-user-static to emulate instructions for other CPU architectures to get around this. This image uses binfmt_misc to tell the host’s Linux kernel to tell a third-party application (in this case, qemu) to execute binaries in formats that it doesn’t recognize. … »

Using a Mac and burned by Docker Desktop? Use Lima instead!

Reading Time: Approximately 4 minutes.

UPDATE 2021-12-28 16:54:00 CST If you’re getting weird networking or security errors from your Docker containers, your VM’s time might need to be updated. You can fix this by running this command: docker run --rm --privileged alpine hwclock -s You’re probably not going to want to run this manually every time your time goes out of sync. Run this automatically by adding this command to your computer’s crontab: echo '0 * * * * docker run --rm --privileged alpine hwclock -s` > \ /etc/crontabs/root In August, Docker/Mirantis has changed their licensing model to require businesses with more than 250 employees and $10M in revenue to pay for Docker Desktop. … »

How to SRE-ify your React app with Prometheus

Reading Time: Approximately 3 minutes.

I am not a JavaScript developer. However, I was given a task at work recently that forced me to enter the abyss and get good at keeping my Promises. I was asked to create a webinar on helping developers become better SREs through observability and instrumentation. The objective was to take a broken web app and add enough monitoring and logging to it to make troubleshooting its brokenness easier. (I’ll update this post with a link when we broadcast it on April 22nd! … »

A few tips on successful remote value stream maps.

Reading Time: Approximately 6 minutes.

There is no doubt that the worldwide COVID-19 crisis has been a wet blanket for digital transformation across the enterprise. However, I don’t know about you, but I’m super fortunate that this is happening in 2020’s technological landscape instead of, say, 2010’s. With video conferencing solutions that work with even the slowest and least reliable internet connections and real-time collaboration tools that scale to hundreds of people per session, many of today’s key activities that required an office only five years ago can be done from the comfort of our own homes or apartments. … »

Want to test Ansible playbooks that require systemd in Docker? Try this.

Reading Time: Approximately 2 minutes.

Kubernetes and other cloud-native strategies might be putting configuration management out to pasture, but I found myself writing a playbook recently while learning how to create infrastructure as code for Azure. I needed to start my Flask web server and Postgres database with systemd, which isn’t a pattern that’s easily supported by Docker. I got this working with Docker Compose, however, and this post will show you how! Create a Docker Compose file with the following services: version: '2. … »

Story Points Aren't Units of Time

Reading Time: Approximately 4 minutes.

They just aren’t. WHY Search for “story points agile” on Google. Try it. You don’t even have to type it into Google; click the link! You’ll get, at this time of writing, approximately 12 million results. Accouting for the 8 million results are bots promoting something that requires your wallet, that leaves four million web pages, many of which will go on to describe story points to the letter and how they aren’t about estimation. … »

SRE and BDD: The Ultimate Power Pair

Reading Time: Approximately 7 minutes.

The responsibilities of a Reliability Engineer are well understood: maintain a high degree of service availability so that customers can have a consistently enjoyable and predictable experience. How these goals are accomplished — establishing SLOs with customers, enforcing them through monitoring SLIs and exercising the platform against failure through Game Days — is also well understood. Much of the literature that exists on SRE goes into great depths talking about these concepts, and for good reason: failing to establish a contract with the customer on availability expectations for the service that they are paying for is a great way for its engineers to spend their entire careers fire-fighting. … »

SRE Communities vs SRE Centers of Excellence

Reading Time: Approximately 7 minutes.

I read Google’s Site Reliability Engineering Workbook on a flight to New York the other day. I read their original book when it came out two years ago and was curious to see how much of it mirrored my own (brief) experience as a Google SRE. Given that it’s been a while since I did pure SRE work, I wanted to keep my skills caught up, and the Workbook seemed like a more accurate reference to follow. … »

Is your Java app ready for Docker? Take this super quick test!

Reading Time: Approximately 1 minutes.

Here’s a really quick test to see if your enterprise Java app is ready for Docker. NOTE: I am not a Java developer; more like a casual observer. Get your pitchforks ready! If I can’t do this: $> docker run --rm --volume "$PWD:/app" --volume "$HOME/.m2:/root/.m2" \ --tty maven:3.6.0-jdk$WHICHEVER_VERSION-alpine mvn build $> docker run --rm --volume "$PWD:/app" --tty openjdk:$WHICHEVER_VERSION-jdk-alpine \ java -jar /path/to/war.war Then either: Your application is not 12-factor and is probably not ready for Docker, Your source code has hidden dependencies that live outside of your pom. … »

Move Fast And Retain Corporate Governance with Pull Requests

Reading Time: Approximately 7 minutes.

DevOps and change control mix like oil and water. Product and development teams want to experiment with and release ideas as quickly as their customers request them, and do so with tight, but unstructured, collaboration across organizations. On the other hand, corporate governance wants auditability, transparent risk mitigation and justification in every step of the way. Consequently, both of these sides often don’t get along with each other well, hindering development speed in the progress. … »