In The Adventure of Silver Blaze, Sherlock Holmes solves the case because a dog doesn’t bark.
“Is there any point to which you would wish to draw my attention?”
The Adventure of Silver Blaze
“To the curious incident of the dog in the night-time.”
“The dog did nothing in the night-time.”
“That was the curious incident,” remarked Sherlock Holmes.
Today, I found myself in a spot where I missed that the dog was not barking.
I was working on a failing Lambda function, but it also did not give me useful logs about why it wasn’t failing.
The Lambda was in a VPC and called out to SecretsManager to get some credentials.
The Lambda failed, and I at first thought that I had an issue with permissions or the name was wrong for the secret; they weren’t.
I rewrote the Lambda to put in more logging and hopefully better error handling, and I still don’t get any error messages in the log.
I finally pinged a coworker and asked for a second set of eyes, and it was he who noticed that the dog was not barking, so to speak.
There was nothing wrong with my code, my permissions, or the data I was using. The call to SecretsManager was timing out, but my Lambda timed out first, so I never saw the timeout error from SecretsManager that would have shown me the issue.
If a Lambda is in a VPC, you need to add an endpoint to allow the Lambda to talk to SecretsManager.
Using an AWS Secrets Manager VPC endpoint
StackOverflow: AWS Lambda access Secrets Manager from within VPC
Dan Cook 