The Dogs That Didn’t Bark

In The Adventure of Silver Blaze, Sherlock Holmes solves the case because a dog doesn’t bark.

“Is there any point to which you would wish to draw my attention?”
“To the curious incident of the dog in the night-time.”
“The dog did nothing in the night-time.”
“That was the curious incident,” remarked Sherlock Holmes.

The Adventure of Silver Blaze

Today, I found myself in a spot where I missed that the dog was not barking.

I was working on a failing Lambda function, but it also did not give me useful logs about why it wasn’t failing.

The Lambda was in a VPC and called out to SecretsManager to get some credentials.

The Lambda failed, and I at first thought that I had an issue with permissions or the name was wrong for the secret; they weren’t.

I rewrote the Lambda to put in more logging and hopefully better error handling, and I still don’t get any error messages in the log.

I finally pinged a coworker and asked for a second set of eyes, and it was he who noticed that the dog was not barking, so to speak.

There was nothing wrong with my code, my permissions, or the data I was using. The call to SecretsManager was timing out, but my Lambda timed out first, so I never saw the timeout error from SecretsManager that would have shown me the issue.

If a Lambda is in a VPC, you need to add an endpoint to allow the Lambda to talk to SecretsManager.

Using an AWS Secrets Manager VPC endpoint
StackOverflow: AWS Lambda access Secrets Manager from within VPC

Inverse Hanlan’s Razor or Forest Gump Principle

Hanlan’s Razor is one of my favorite mental models.

Never attribute to malice that which is adequately explained by stupidity.

I grew up with several conspiracy-minded relatives, and this one idea did more than anything else to help me see other possible explanations beyond a secret cabal.

It also made me a far easier person to work with since it helped me see certain relationships at work as something other than adversarial relationships.

I often contemplate what most TV shows and movies would look like if a character’s best friend would, at a key moment, step in and ask them if they’d ever heard of Hanlan’s Razor.

In my fantasies, the main character understands and goes on a completely different course with more empathy for the other person. In a more realistic depiction, the best friend probably gets their head chewed off.

Today I discovered a different side of Hanlan’s Razor that might be worth calling the Forest Gump Principle.

In the movie Forest Gump, the main character Forest bumbles into important historical moments. It’s not so much that he is a genius, quite the opposite, but he does some incredible things through luck and timing.

One of my favorite examples is when Forest stays at the Watergate Hotel one night and calls the front desk to tell them about the power being out, eventually leading to Nixon’s resignation.

So my proposal for the Inverse Hanlan’s Razor is

Never attribute to good intentions that which is adequately explained by dumb luck.

It turns out that determining people’s intentions is really hard, even when it’s ourselves. I don’t know that moving entirely to consequentialist thinking is the correct answer, but heavily discounting our assumptions of people’s intentions is probably a good rule of thumb.

Resolve merge conflicts on GitHub protected branches

If you have the Require a pull request before merging branch protection rule enabled on the source branch of a pull request, you cannot resolve merge conflicts via the web interface. Unfortunately, no error alerts you to the failure.

The only solution I’ve found that allows you to resolve merge conflicts between branches with branch protection rules without disabling the branch protection rules is to create a temporary branch, locally resolve the conflicts, and then merge that into the target branch via a PR.

Galaxy and VS Code

Last week was GitHub Galaxy, where GitHub showed off many of its innovations and how its platform can help enterprise and “mid-market” customers. I’ll do a separate write-up on what I got from those presentations.

Let’s look at what got released last week.

On the quality of life side, GitHub improved the contrast in their default light and dark themes. I haven’t taken them for a spin because I have been using one of their themes targeting color-blind folks because it has better contrast. I wish their announcement did more to show the side-by-side comparisons of what changed. Still, I’m delighted they continue to make efforts to improve their accessibility.

The feature that I’ve been anticipating the most is the addition of a VS Code extension for GitHub Actions. You can get the extension here. I’ve been using it since it came out, and it has already saved me a couple of times. Previously, I depended on actionlint for early feedback on my workflows, but this new extension does an even better job. The most significant difference I’ve seen is it does a decent job of detecting what variables are in scope. It still needs improvement, but that is typical with a beta product. While I haven’t used it yet, you can also check the status of workflows associated with the repo you are in from VS Code. This feature might be a massive win for devs who aren’t creating workflows but using them.

Putting on my security and compliance hat, GitHub made a ton of improvements that I can’t wait to use.

Looking at the dependency graph, we can now export an SBOM or Software Bill of Materials. GitHub has been making many improvements to the dependency graph and allowing us to add things not found during the build. This built-in way to generate an SBOM will be a huge help for anyone who needs to worry about Executive Order 14028. They’ve also made a bunch of improvements to the UI for the dependency graph.

Another fascinating compliance change lets us see when an admin bypassed a branch protection rule to push a commit. While this is only a minor tweak to the existing functionality it helps shine a light on areas where something exceptional happened.

GitHub released even more features than this, but these are the ones that caught my eye.

This article will be my last This Week in GitHub. I’ve learned a lot by looking at their releases every week, but it’s time for me to move on to some new projects.

An abundance of caution

Usually, this is a space where I write about the features that GitHub released last week. I am still amazed at how much stuff their engineering teams release in a given week, but that is not what I’ll write about now.

GitHub announced 13 changes last week, most around their Advanced Security offering, and the rest didn’t immediately catch my eye. The possible exception is that GitHub fixed a bug where the title of private issues and PRs could be visible in their new search feature](https://github.blog/changelog/2023-03-23-fixed-bug-that-allowed-private-issues-and-pull-request-titles-to-be-shown-in-search-results/). Overall, I’m not too worried about this particular issue. There are nuances around getting a specific item to appear in the search. Realistically, there isn’t a lot of proprietary information to glean from the title of a PR or Issue. All that said, I’m hoping that in the next Availability Report, GitHub will go into more detail about how they will prevent similar issues from happening in the future.

A potentially more impactful change happened last week when GitHub changed its RSA SSH Host Key. This happened because they accidentally published the private key, and out of an abundance of caution, GitHub changed their host key. So this is not a sign that GitHub had a data breach.

If you use an RSA SSH key, you will see a message like this when you try executing git commands against GitHub.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:uNiVztksCsDhcc0u9e8BujQXVUpKZIDTMczCvj3tD2s. Please contact your system administrator. Add correct host key in ~/.ssh/known_hosts to get rid of this message. Host key for github.com has changed and you have requested strict checking. Host key verification failed.

GitHub provided instructions in the announcement for updating your ~/.ssh/known_hosts if you want to continue using the same SSH key. Alternatively, you could create a new SSK key using ECDSA or Ed25519 or get fancy and use a Yubikey.

Sticking with the overall theme of security, GitHub has also created a Secure Code Game. This game specifically targets C and Python, but many of the ideas will carry over to whatever language you work in. It follows my favorite style of coding challenge, where there is some existing code and some tests for you to make them pass. It’s been a while since I’ve written any Python, but I’m excited to give this a go.

And finally, GitHub has announced some cool stuff it’s doing with AI and even opened up a few waitlists. While it’s a little ridiculous, I love the idea of having Copilot write a summary describing the PR I’m looking at. I want to see how all this was made because I’d bet the difference between adding a summary and making a poem was so minimal it just got thrown in.

Have you started using AI in your daily work? I’d love to hear about how you are using it.

Betas for everyone!

Last week GitHub released a feature that had a coworker whooping with joy. But we’ll get to that in a moment.

First, there is a more significant thing I want to look at. GitHub is working with EU policymakers on the Cyber Resilience Act. While I’m based out of the US and work at a US-based company, I still have unpleasant memories from the year before GDPR went into effect as we scrambled to comply with some legislation we didn’t understand. I’m very excited to hear that GitHub is working on this, and I hope the policymakers listen.

I’m slowly reading through the proposals for the Cyber Resilience Act or CRA, but it is slow going. It will be worth keeping an eye on this proposal.

Also, GitHub is officially starting its rollout process to move everyone to 2FA.

Now for the releases that GitHub made last week.

GitHub announced a fix to a bug where users retained access to an org after being removed. I couldn’t find any previous mention of this bug, but apparently, they have had a manual fix since Oct. 20, 2022. My guess is GitHub has a policy of waiting for the automated fix to be in place before discussing issues. I wish I had heard about this issue in October and suddenly got an update rather than finding everything out later.

Moving to happier news, I was surprised to see how excited some coworkers were about this change. You can now comment on an entire file in PRs. This is a public beta, and they are looking for feedback, so try it out.

There is also a new beta for slash commands in discussions, issues, PRs, and projects. I’m curious to see how much this becomes a part of my workflow. Adding code blocks doesn’t knock my socks off, but easy access to saved replies and creating markdown tables has my attention.

Enterprise customers can join a public beta to display members’ IP addresses in the audit logs.

Moving from public to private betas, GitHub is looking for folks to test a tool for migrating repos from BitBucket Server to GitHub.

Moving over to Generally Available features, we have a lot more.

Projects are now on GitHub Mobile. On the one hand, using a ticketing system from my phone sounds like hell; on the other, I recognize that I’ve done that with Jira.

Discussions can now be closed as either Resolved, Outdated, or Duplicate.

Creating and managing GitHub Action Runner Groups comes to the team plan. This functionality was already available to enterprises.

In the land of GitHub Advanced Security, code scanning now only shows alerts based on the files changed in a PR. Also, the change to notifications for secret scanning has gone into effect.

We saw runner groups get moved into the team plan, and I also hope we eventually see something like this for GitHub Advanced security.

Sticking with changes to the enterprise tier, apps can now call GitHub APIs using a user-to-server token. I haven’t used this functionality, but it apparently creates parity with OAuth apps.

Finally, we get to Dependabot, who also got some love this week. Dependabot now keeps Gradle version catalogs up-to-date. And, for anyone using versioned Reusable Workflows, Dependabot can now keep those up to date too.

Well, that about wraps it up. What new features stood out to you?

2FA is Coming

I’ve talked about it a lot, but it’s here. GitHub is starting to roll out the 2FA requirements for everyone starting March 13th, the day this is published. GitHub will begin with small groups at first but progressively scale the size of those groups up over the year.

They will email folks 45 days before they will require it for an account, and if you do not set up 2FA in that time, you will get the opportunity to snooze, turning it on for up to one week the first time you log in after those 45 days are up.

This article mentions the numerous improvements they have made to 2FA this year. I have no doubt that we will see more as they learn through this rollout process.

That’s not the only big news from this week. GitHub also announced the Octernship, open to students in India, Singapore, Indonesia, Malaysia, Vietnam, Philippines, Thailand, Mexico, Nigeria, and Colombia. This is an excellent program to expand the opportunities in tech beyond the US. It is also fantastic that GitHub allows other companies to partner with them and work with students. If you’re interested in being a partner for this program, you can apply here.

None of what I’ve mentioned so far is about features that GitHub released last week, and they certainly released a lot, so let’s get into those.

The dependency graph will no longer ingest go.sum files in Go projects. This will apparently reduce the number of false positives for Depenedabot alerts. Also, go.mod files are still supported.

Sticking with the dependency-related topics dependency graph and Dependabot now support npm v9.

Security advisories now have multiple types of credits. These credits are:

  • finder
  • reporter
  • analyst
  • coordinator
  • remediation developer
  • remediation reviewer
  • remediation verifier
  • tool
  • sponsor
  • other

GitHub Enterprise Server 3.8 is now generally available. While I understand the need for self-hosted GitHub instances, I still find it amusing that some folks want to self-host GitHub. You can run both the self-hosted and cloud-based versions if you have GitHub Enterprise.

The GitHub Slack integration now supports filters on Discussion categories. I can imagine this being a great improvement for anyone subscribing to Discussion via the Slack integration. I hope we see more filters like this added to that integration in the future.

GitHub Enterprises can now run more concurrent 2-core Windows and Linux jobs. Previously, you could only run 180 jobs concurrently as a GitHub Enterprise on 2-core Windows and Linux runners. Now you can run up to 500; if that still isn’t enough, you can reach out to GitHub support to get even more.

Custom repository roles API is now generally available. There was also a breaking change that came with this. The endpoint is moving from /orgs/{org}/custom_roles to /orgs/{org}/custom-repository-roles. The now deprecated beta API will be removed on September 7th, 2023.

There will also be changes to the code search API starting April 10th. The most significant change is rate limiting, which will be set to 10 requests a minute.

Sticking with searching through code over in Advanced Security land, you can now run CodeQL queries across multiple repos at the same time. GitHub is mainly targeting this at security researchers, allowing them to look at a bunch of open-source repositories simultaneously. Hopefully, this will help make the theory that there are more eyes on open-source software true.

Also, you can now delete stale code-scanning configurations. I’ve not used this feature, so I’m not entirely sure what this is solving, but my guess is fewer false positives to look through in security scans which is always a win.

In GitHub Actions land, we see some improvements to required actions. The biggest improvement is that any branch with a required workflow automatically requires a PR.

Finally, we have GitHub Issues & Projects with a whole bunch of improvements, but I’m waiting for the Task List improvements they’ve made to come to PRs.

What new feature are you most excited about?

So many features so little time

One of the things that I am constantly amazed by at GitHub is just how much they release stuff. Now I know that GitHub has many engineers and different product areas to work on, but last week they announced 13 changes. And they didn’t release anything on Monday!
Looking at this with my engineer hat on, GitHub gives us a great view of what ship then iterate looks like. They tend to release public betas of new features to get feedback from actual system users and eventually promote them to GA or Generally Available. A lot of their changes are also minor changes.
I remain fascinated with their work on improving 2FA as they move to require it for anyone who contributes code. I remember working at a company with quarterly releases where we would batch together 3-4 months’ worth of changes from devs and release them all at once. This release cadence was jarring for the user and also prone to bugs. Instead, GitHub is releasing a bunch of small stand-alone features as they are ready. This approach gives them faster feedback on each feature, makes the whole rollback process a lot easier, and is easier for most users to adapt to the changes.

But enough talk about their engineering process; let’s look at those 13 changes.

Dependabot just got even easier to enable across your enterprise. Enterprises exist as an entity over multiple organizations. A pattern I’ve seen some companies do is to have an organization dedicated to their open source/public repositories and another to their closed source/private repos. This segregation offers a ton of value and peace of mind. Previously you could enable dependabot at the repo and organization level, but just doing this at the enterprise level is nice, especially if you ever roll out a new organization.

Secret Scanning is now generally available on all public repos. You can also enable this for all your public repos from your account’s code security and analysis settings. This functionality already existed for organizations, but seeing this as an easy feature for individuals is terrific.

The GitHub profile now allows you to specify your pronouns. When you hover over a user’s profile picture and sometimes their name, you see their profile details which now include their pronouns if they specified them.

As a paid team or enterprise, you can now enroll in a public beta for access to a 12-core x64 macOS runner for GitHub Actions. At the moment, the per-minute runner cost is

  • $0.32 per minute for the 12-core macOS
  • $0.08 for a 3-core macOS
  • $0.256 for a 64-core Linux
  • $0.512 for a 64-core Windows

GitHub Actions Importer is now Generally Available. The importer is an extension for the GitHub CLI. It supports importing workflows from

  • Azure DevOps
  • CircleCI
  • GitLab
  • Jenkins
  • Travis CI
    Initially, I was intrigued by the ability to create extensions for gh, but this is the first one I’ve seen that looks truly useful. I hope we start seeing more extensions like this from the community and GitHub. In fact, GitHub released another extension for gh this week, but I’ll get to that in a moment.

You can now try rich diffs for Jupyter Notebooks in PRs. You can enable this under Feature preview, one of the few places you can’t deep link to in GitHub. This feature is another excellent example of how GitHub ships than iterates. There was a private beta for this feature; if you were part of it, you might need to re-enable it. Also, it does not allow you to make comments in the rich preview for the moment. I know that in the past, many places I’ve worked would have held back a feature until it was “perfect,” but this way, GitHub is getting it out faster to those who care about it.

GitHub admins can now bypass environment protection rules in GitHub Actions. It is possible to disable this, but by default, admins can skip those rules. This bypass is for a break-glass scenario when the normal approver is unavailable. I was initially skeptical about this, but then I realized I had previously added the admin group to environments to create this very option. I haven’t looked at the audit logs, but I’m betting this will make it easier to check when an admin bypassed the standard process rather than my current strategy, which looks like a normal approval.

The Projects team seems to take a slightly different approach from the other teams and gives weekly updates on what they release. It is great to see an example of a company providing a team the autonomy to do what works for them rather than trying to force every team into the same pattern. The new GitHub Projects has more improvements. Also, continuing the theme of extensions to the GitHub CLI, projects now has an extension.

I’m sure at some point, you’ve heard someone go on a moralistic rant about the importance of accessibility in what we create. GitHub is walking the talk and has made many accessibility improvements to the contribution graph. It is now keyboard navigable and more screen reader friendly.

The 2FA improvements keep coming. You can now enable both SMS and TOTP as a second factor at the same time. This has removed SMS as a fallback method since it is now a first-class second factor.

GitHub Desktop got some love, and you can now preview PRs. I’ve been using GitKraken for years, but I’ll give GitHub Desktop another try.

And last but not least, there were some changes to GitHub Advanced Security. You can now control code scanning for all repos in an organization from the Security Coverage page. This is a public beta, and currently, it only supports JavaScript, Python, and Ruby but more languages will come. Also, notifications for secret scanning are moving to an opt-in model; you’ll need to make changes before March 16th if you wish to continue receiving notifications.

There was certainly a lot of stuff released last week by GitHub. What feature are you most excited about?

GitHub is Continuing to Improve 2FA

This is the year that GitHub will require anyone who contributes code to enable 2FA, and we continue to see them improve how we use 2FA this week. As I previously mentioned, the changes we see out of GitHub this year will be an excellent opportunity for us to learn how to do 2FA better.

You can now disassociate your email address from your locked GitHub account. As GitHub mentions when enabling 2FA, if you lose access to your second factor and recovery codes, there is no way to regain access to your account. With this latest enhancement, if you find yourself in that situation, you can disconnect your email from that account to create a new account.

While creating a new account would change the read-only email account that GitHub generates for you, if you use your actual email address, all your commits would transfer over to that new account.

GitHub also made it easier to set your preferred 2FA method. This will hopefully make it easier for you should you need to log in from another device or use sudo mode. I didn’t see the point of this change at first. Then I remembered a brief time when GitHub decided I should always use the mobile app to verify myself even though I had my security key already connected to my computer.

Moving away from 2FA, if you use JetBrains Gateway and are a .NET developer, you’ll be excited to hear that CodeSpaces now has a beta for Rider. I would love to hear how this goes for anyone using it.

Also, code search and code view are now in a public beta. I had some initial issues with code search, but over the past few weeks, I’ve not run into any, so hopefully, they have gotten all the kinks worked out as they enter into the public beta.

Were there any GitHub announcements I missed that caught your eye?

You can’t win them all

Most weeks, I find a few things that GitHub releases that catch my eye. Unfortunately, last week was not one of those weeks. In fact, what caught my eye this week was a project getting canceled.
The book Accelerate popularized the four key metrics of DevOps or the DORA metrics. These metrics offer a great deal of insight and can help guide teams to find out how they can improve.
GitHub had a feature on the roadmap to show Deployment Frequency and Lead Time. These are the two metrics that GitHub is ideally situated to provide, but there are a lot of challenges in collecting these metrics.
While this was not assigned to anything other than future development, I was very excited about this feature, and I am sad to see that it will not be worked on. That said, I understand the challenges of trying to collect this data.
How can you tell from your repository that code was released? Does merging to your main branch represent releasing? What about creating a tag? If you use GitHub Actions for your deployment pipeline, this is much easier since it is the mechanism that releases your code.
Unfortunately, we are left with many other tough questions, like how do you calculate the lead time for an individual item? Is this something that can only be calculated if you use GitHub issues? What if you use conventional commits or provide a changelog?
I’ve also found that each repo, let alone each team has a different release process. Creating a truly generic solution to these various problems would be very challenging.
So, while I am sad that GitHub will not build in a feature, I understand why they won’t.
Do you track any of the DORA metrics? Have you automated their collection?