Goread2 - Chapter 2: Deployment Hell

This is the second in a series of posts wherein I attempt to recount the history of Goread2 as it approaches a state in which I might actually try to share it more broadly. The first in the series is here.

GoRead2 worked great on my laptop!

On August 2, fresh off the high of the multi-user transformation, I did what any sensible developer would do when they have a working local app and a Google Cloud account: I tried to put it on the internet. What followed was a masterclass in the specific, humbling genre of suffering that is “deploying a real app for the first time”, a genre characterized by problems that are obvious in retrospect, cryptic in the moment, and deeply embarrassing in the git log forever.

Roll the tape.

Hour 1: Our First Production Security Incident

We commited the multi-user stuff at 4:40pm. 38 minutes later in the git log, we made a second commit with the subject line SECURITY: Remove foo.env from repository and add to .gitignore. The commit message explained, with the kind of formal dignity one adopts when hoping nobody notices: Remove accidentally committed environment file containing OAuth secrets and add it to .gitignore to prevent future commits.

Even better: the file was called foo.env. It contained OAuth credentials. It had just been pushed to a public GitHub repository. Tell me you are not a professional software developer without telling me? And I hadn’t even tried to deploy anything yet!

The rest of August 2 was spent fighting the CI pipeline: golangci-lint didn’t like the version specification, then GitHub Actions didn’t support Go 1.23, then after downgrading to Go 1.22 the OAuth library had compatibility issues, then after downgrading the OAuth library the security scanner complained, then the security scanner itself turned out to use a deprecated action, then (and this is my favorite part) the solution to the overly aggressive security scanner was a commit titled Disable security scanning job to resolve excessive govulncheck errors. We had committed actual secrets to a public repo less than an hour ago, and now I was disabling the security scanner. Tremendous.

Day 2: Fun With Authentication

August 3 was the day I actually tried to log in to GoRead2 running on App Engine. The first result was a commit called Fix auth callback 500 error by implementing Datastore user management, which is a polite way of saying: when you clicked “Sign in with Google,” the app crashed. The OAuth flow completed successfully, Google redirected back to the app, and then the app threw an internal server error because we hadn’t actually implemented the part where it saves the user to the database.

This is a relatable oversight, right? You build an auth system, you test the middleware, you write the session management, and then you forget to handle the moment when a new user shows up for the first time. Classic amateur hour over here.

Fixed that. Tried to add a feed. The feed appeared in the sidebar. Clicked on it — no articles. Commited Fix missing articles by adding subscription verification to GetUserFeedArticles. Okay. Now there were articles. Clicked on an article. The browser console showed: TypeError: this.feeds.forEach is not a function.

The API was returning feeds. The frontend was receiving them but it was simply declining to do anything with them, because the response format had changed in the multi-user refactor and the JavaScript hadn’t caught up. Commited Fix TypeError: this.feeds.forEach is not a function. Fine.

Now, while debugging all of this, another look at app.yaml revealed: the OAuth client secret was hardcoded in the config file. Again. Not the credentials themselves this time, just the environment variable syntax ${GOOGLE_CLIENT_SECRET}, which App Engine doesn’t expand — but still, essentially a repeat mistake.

This is where the commit messages turned into poetry: More secret leakage. C'mon, Claude. (I asked Claude to review this article and its accusations, and it replied in its inimitable understated voice, “This one is addressed to me, and I find it reasonable.”)

The fix was to use App Engine’s Secret Manager integration syntax, where instead of an environment variable you write _secret:projects/PROJECT_ID/secrets/SECRET_NAME/versions/latest. This was duly committed as Fixing this secret leakage once and for all.

Immediately followed by Aaaaand the human makes a typo. The typo? {$GOOGLE_CLIENT_SECRET} instead of ${GOOGLE_CLIENT_SECRET}. Do you see it? It took me a few minutes.

A commit titled We will get this right someday switched to a shorter _secret:google-client-id format. Another titled Maybe I should have let Claude do this after all added a missing GOOGLE_REDIRECT_URL that had been omitted entirely. Are we having fun yet?

Day 3: Two Days of OAuth Debugging, Compressed

August 4 was dedicated to the question: why is the OAuth client ID coming through as an empty string in production?

The answer was another typo in app.yaml. The Secret Manager path for the client secret was _secret:project/goread-467200/secrets/.., missing an “s”, it should have been projects. Are we loving computer programming, or what? The commit that fixed it was titled, with admirable restraint: Fix typo in app.yaml: project -> projects for GOOGLE_CLIENT_SECRET.

But we didn’t know that yet. What we knew was: OAuth was broken, and we didn’t know why. The approach to solving that problem was adding logging. Like, a LOT of logging. A few of the git log greatest hits, all from August 4:

  • Add OAuth environment variable debugging.
  • Print actual OAuth environment variable values.
  • Add comprehensive startup debugging logs.
  • More debug tweakery.
  • Make OAuth environment variable debugging more explicit for empty values.
  • Simplify OAuth debugging with step-by-step logging.
  • Add GOOGLE_CLOUD_PROJECT debugging to verify project ID.

Yes, seven commits over the course of one day, each one adding more print statements to try to see what App Engine was actually loading for those environment variables. Each one requiring a full deployment to App Engine to test, because the Secret Manager integration only works in production. You cannot easily reproduce “App Engine’s Secret Manager is silently ignoring a malformed path” locally.

On August 5, we finally found the path typo and fixed it. OAuth flow worked. We removed debug logging. Commit titles became less combative: Verbiage tweakage. Then a couple of linter fixes. Then Fix GitHub Actions tests by running unit tests in short mode, which addressed the fact that the integration tests took too long to run in CI and were timing out.

And finally, GoRead2 was live.

Looking back at those 44 commits over four days, the striking thing isn’t that any individual problem was that hard. None of them were. A missing “s” in a config path, a transposed dollar sign, a response format mismatch, an overlooked foo.env. Each one was the kind of thing that takes thirty seconds to fix once you know what it is. The hard part was knowing what it was — staring at a 500 error from an App Engine log with no local reproduction path, trying to reason backward from “OAuth client ID is empty” to “you misspelled ‘projects’, you moron.”

I suppose there’s a tax you pay the first time you deploy any app: the gap between what you’ve built and what you’ve deployed turns out to contain some unpredictable number of small decisions you made incorrectly. You try to pay it all at once, over a frantic long weekend, and then you are done. The app works. The secrets are where they should be. The CI pipeline is green.

And you quietly update the .gitignore and move on.

Next up: The First Bill Arrives.