AWS Lambda with SQS trigger using .NET Core-Part 2

Welcome back!!!

During Part - 1 of this article we learned how to create Lambda project in .NET Core and how to use MsBuild to build, test and package the Lambda function.

During Part-2 we are going to learn how to deploy the Lambda function to AWS and how the deployment can be automated and also test the Lambda function with SQS trigger from AWS Console.

So let’s begin. Lambda function can be deployed two ways.

1. By manually uploading the package via AWS Console.
2. By automating the deployment process.

There are various ways Lambda deployment can be automated and each of the way would require different set of tools and technologies.

The approach we will use here would use MsBuild, AWS CLI commands and Terraform. We already have used Msbuild and AWS CLI command in the first part of the article. Terraform is the new word here.

Following is the one-line introduction of Terraform.

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently.

More details about Terraform available at (Here).

For the lambda function deployment, Terraform needs to know from where to get the deployment package. Terraform can locate the deployment package from an S3 bucket. We will take that route. So before we can start the deployment using Terraform, we need to make sure that the specific bucket is created on S3 and also the package is available at the certain S3 bucket.

Let’s create a bucket on AWS using AWS CLI S3 commands. Switch to the Terminal or command window and type run following command to create a bucket on S3.

aws s3api create-bucket --bucket lambdawithsqs --region ap-southeast-1 --create-bucket-configuration LocationConstraint=us-east-1

You can use your preferred bucket name instead of lambdawithsqs. If the bucket is created successfully, command should return following output.

{
  "Location": "http://lambdawithsqs.s3.amazonaws.com/"
}

We need to change our MsBuild script to upload the package to an S3 bucket. We will use AWS CLI S3 commands from MsBuild to upload the build package to S3.
Let’s add a new property BucketName to the script. Add following line of code below <PackageName> property.

<BucketName>lambdawithsqs</BucketName>

Add following snippet to the proj file below “Package” target.

<Target Name="UploadBuildToS3">
  <Message Text="Uploading build to S3" />
  <Exec WorkingDirectory="$(ProjectDirectory)" Command="aws s3 cp $(OutputDirectory)$(PackageName) s3://$(BucketName)/$(PackageName)">
  </Exec>
</Target>

As you can see aws s3 cp command is executed as part of this target and it is using values of OutputDirectory, PackageName and BucketName properties of the script.

To create a new target which will also execute the newly created target above add following line after target BuildTestPackage.

<Target Name="BuildTestPackageUpload" DependsOnTargets="Compile;UnitTests;Package;UploadBuildToS3" />

Let’s test these changes and see if the package is being uploaded to the bucket which we created earlier. Run following command.

dotnet msbuild ./Build/lambdawithsqs.proj /t:BuildTestPackageUpload

To check if build is uploaded to S3 or not, run following command.

aws s3api list-objects --bucket lambdawithsqs

So we are now ready to create Terraform script to perform automated deployment of Lambda function. Create a new directory Terraform under the solution directory and create a new file deploylambda.tf under Terraform directory.

mkdir Terraform
cd Terraform/
touch deploylambda.tf

Open deploylambda.tf file and enter following contents to it.

provider "aws" {
  region = "us-east-1"
}

variable "lambda_role" {
  default = ""
}

resource "aws_iam_role" "lambda_role" {
  name = "lambda_role"

  assume_role_policy = <<EOF
  {
    "Version": "2012-10-17",
    "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }]
  }EOF
}

resource "aws_iam_role_policy_attachment" "lambda_role_sqs_policy" {
  role       = "${aws_iam_role.lambda_role.name}"
  policy_arn = "arn:aws:iam::aws:policy/AmazonSQSFullAccess"
}

resource "aws_iam_role_policy_attachment" "lambda_role_cloudwatch_policy" {
  role       = "${aws_iam_role.lambda_role.name}"
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess"
}

resource "aws_sqs_queue" "lambdaqueue" {
  name                      = "lambdaqueue"
  delay_seconds             = 90
  max_message_size          = 2048
  message_retention_seconds = 86400
  receive_wait_time_seconds = 10
  visibility_timeout_seconds = 150
}

resource "aws_lambda_function" "lambdasample" {
  function_name = "lambdasample"

  s3_bucket = "lambdawithsqs"
  s3_key    = "lambdawithsqs.zip"

  handler = "lambdawithsqs::lambdawithsqs.Function::FunctionHandler"
  runtime = "dotnetcore2.1"
  publish = true
  role = "${aws_iam_role.lambda_role.arn}"
  timeout = 120
}

resource "aws_lambda_event_source_mapping" "lamdasqsmapping" {
  batch_size        = 10
  event_source_arn  = "${aws_sqs_queue.lambdaqueue.arn}"
  enabled           = true
  function_name     = "${aws_lambda_function.lambdasample.arn}"
}

The above contents of deploylambda.tf file explains what resources to be created on AWS with what specific property values associated with them.

Following are the explanations about few of the important parts of the file.

1. Line 1 configures the cloud provider name (aws in this case) where the resources to be created. Terraform supports multiple cloud providers such as AWS, Microsoft Azure, Google Cloud etc. to name a few.
2. Line 5 a variable declared with name lambda_role and default value set to empty string. This variable is used Line 27.
3. Line 9 a new IAM role to be created with name lambda_role.
   1. Line 12 the role is configured to be assumed by Lambda function on AWS.
4. Line 27 role create above is assigned a policy which will allow full access to Amazon SQS service.
5. Line 32 the role is assiged a policy which will allow full access to Amazon CloudWatch Logs.
6. Line 37 a new SQS queue to be created with name lambdaqueue and other properties set.
7. Line 46 a new Lambda Function to be created with name lambdasample.
   1. Line 49 package is available at S3 bucket with name lambdawithsqs
   2. Line 50 the deployment package name under the bucket is lambdawithsqs.zip
   3. Line 52 the handler of the lambda function. This value is fully qualified name of the function which is the entry point of the Lambda function.
   4. Line 55 the ARN of the role (lambda_role) which is created above.
8. Line 59 a event source for the Lambda function.
   1. Line 61 ARN of the source of the event which will be the ARN of the queue which will be created as per the configuration at `Line 9.
   2. Line 63 ARN of the Lambda function created as per the configuration at Line 35.

Now we have Terraform script ready, let’s integrate it with the MsBuild script so that we can deploy the Lambda function as part of MsBuild.

Add following snippet after UploadBuildToS3 target in lambdawithsqs.proj file.

<Target Name="Deploy">
  <Message Text="Testing Terraform" />
  <Exec Command="terraform --version" />
  <Exec WorkingDirectory="..\Terraform" Command="terraform init" />
  <Exec WorkingDirectory="..\Terraform" Command="terraform plan -out=tfplan -input=false" />
  <Exec WorkingDirectory="..\Terraform" Command="terraform apply -input=false -auto-approve" />
</Target>

The above target executes three Terraform commands. terraform init, terraform plan and terraform apply.

1. terraform init command initializes the working directory and gets ready. (More…)
2. terraform plan command determines all the Terrafom files under the current directory and creates an execution plan. (More…)
3. terraform apply executes the changes which are determined as part of the terraform plan command above. (More…)

Add following snippet below the “BuildTestPackageUpload” in lambdawithsqs.proj file.

<Target Name="BuildPackageUploadDeploy" DependsOnTargets="Compile;Package;UploadBuildToS3;Deploy" />

Now we have everything ready. Let’s test this entire setup. Run following command.

dotnet msbuild ./Build/lambdawithsqs.proj /t:BuildPackageUploadDeploy

If everything goes well you should be seeing following message.

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Now let us verify if the lambda deployed successfully and works as expected.

Login to AWS Console using your preferred browser and navigate to Lambda services. Select Functions from the left side menu. The page should list a function with name lambdasample. Clicking on name of the function should display the details page of the function which look somewhat like following.

Now let us check if the SQS queue trigger works properly and if Lambda function is invoked with that trigger.

Navigate to SQS service on AWS Console. The queue with name lambdaqueue should be listed over there. Select the queue and select Send a Message from the Queue Action menu located above the list.Type any random message content (such a Hello World!!!) in Message Body and click Send Message.

This should cause SQS Trigger to invoke the lambda function. Lambda function we created just receives the message and writes the message contents to the AWS CloudWatch logs. So if the trigger and Lambda function are working fine, we should be able to see the message contents on the CloudWatch logs.

Navigat to CloudWatch service on AWS Console and click on Logs from the left side menu. The list of log groups should display the log group with name /aws/lambda/lambdasample. Click on the log group name. It should display log stream. Click on the latest log stream. You can determine the latest log stream by looking at values of Last Event Time column.

The log stream should display the log messages logged by the lambda function. The message should have one entry which displays the message which we have posted on SQS Queue.

So this is how you create, build, package and deploy AWS Lambda function with SQS trigger in .NET Core.