By Hyuntaek Park
Senior full-stack engineer at Twigfarm
Monitoring or getting an alert when unexpected happens is essential to keep our system reliable. System faults should be notified to appropriate person as early as it can be. This can be achieved easily with CloudWatch.
In this post, I create a simple lambda function and a couple of tests with a normal condition and one with an error condition. Once the error condition happens SMS text message is sent to my cell number all the way from North Virginia (us-east-1) to Seoul, Korea (ap-northeast-2).
The idea is simple:
Go to Lambda Functions and click Create function. Then create a simple function with any name you like.
The lambda code does simple division. Our goal is to get notified if DivideByZeroException occurs.
Let’s create another lambda function, called sms-sender, which gets triggered if the CloudWatch log contains certain pattern. For now this lambda function does nothing but prints out the input so that we are able to identify what the CloudWatch input format is like.
Now CloudWatch Subscription Filter connects following two lambda functions:
Go to CloudWatch –> Logs –> Log groups –> /aws/lambda/cloudwatch-alert-test
Then click on Subscription filters tab then choose Create Lambda subscription filter
Input as following:
Click on Start streaming button to finish CloudWatch subscription filter setup.
This means that sms-sender lambda function gets invoked, if and only if the CloudWatch log contains string DivideByZeroException.
We created two tests for cloudwatch-alert-test. Run each of them and check the CloudWatch log for sms-sender to see if the function is invoked only when DivideByZeroException is happened.
What do you see in sms-sender CloudWatch log? Was the function triggered only on error test case? Probably yes. Now take a look at the event parameter value in the log. You would see some data value like the following:
What is it? We have to unzip to decode the data.
Following code takes care of unzipping the input from CloudWatch.
Variable stringResult shows stringified version of the input, which looks like:
If your alert method requirement is webhook notifications, then you can just add a few lines of code for HTTP POST call in the lambda function. End of story.
Since SMS text message is a means of alert requirement, one more setup is required.
As of December 2021, mobile text messaging (SMS) feature is not supported in Seoul region (ap-northeast-2). So I changed my region to complete text messaging setup in SNS.
Supported Regions are listed here: https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html
After changing region to us-east-1, go to Amazon SNS –> Mobile –> Text messaging (SMS). If Text messaging (SMS) is now showing in your region, it is likely that your region is not supporting this feature.
Under the Sandbox destination phone numbers, click Add phone number for adding recipients’ phone numbers. By default, you are in a sandbox mode which has a few restrictions such as:
Note: you can always request limit increase here: https://aws.amazon.com/premiumsupport/knowledge-center/sns-sms-spending-limit-increase/
AWS support will get you out of sandbox as well upon the limit request.
After you add a recipient’s phone number for testing, now let’s go back to sms-sender lambda function in order to add some code for text messaging.
Following is the complete lambda code for sms-sender, in which message sending feature is added.
Almost done. One last adjustment is left. Our lambda role needs a permission to send text messages.
Let’s move to IAM role for sms-sender lambda function.
In the lambda role, click Attach policies –> check AmazonSNSFullAccess –> click Attach policy button.
Now all of our components are set up and ready. Go back to cloudwatch-alert-test lambda function and run error test case. Wait a few seconds, you should receive a text message if you set up everything correctly. You can modify the message to any way that you like to. Note that if your message is longer than 160 characters, they are split into separate messages.
You can design your alert system in various ways by applying different log patterns without much hassle utilizing CloudWatch subscription filters.
Getting alerts if something goes wrong is very useful. But it might be annoying if our phones vibrate too frequently than it should be. Yes, it is very tricky to get the balance right. I know DivideByZeroException is not an appropriate case to get SMS text messaging alert :)
Each system has different reliability requirements. When to fire alarm should be designed carefully according to the requirements. Avoiding false alarms is crucial point to consider as well.
Happy Christmas and to our reliable systems!