AWS Lambda with SQS trigger using .NET Core-Part 3

Welcome back!!!

This is the third part of the article where we will learn about how to do increamental deployments to AWS Lambda functions.

During Part - 1 of this article we learned how to create Lambda project in .NET Core and how to use MsBuild to build, test and package the Lambda function.

During Part - 2 we are going to learn how to deploy the Lambda function to AWS and how the deployment can be automated and also test the Lambda function with SQS trigger from AWS Console.

Today in Part - 3 we are going to learn how to do use Terraform to deploy AWS Lambda function when we make change to code.

In the previous part we created Terraform script to create required resources on AWS and deploy Lambda Function. At the last of the Part-2 we ran following command to build, package and deploy the Lambda function.

dotnet msbuild ./Build/lambdawithsqs.proj /t:BuildPackageUploadDeploy

Now let’s make some change to the code of Function.cs. As part of this change we will modify ProcessMessageAsync method to write couple of more statements to the log. After this change the method looks like as following.

private async Task ProcessMessageAsync(SQSEvent.SQSMessage message, ILambdaContext context)
{
    context.Logger.LogLine("Entering method ProcessMessageAsync");
    context.Logger.LogLine($"Processed message {message.Body}");
    context.Logger.LogLine("Exiting method ProcessMessageAsync");
    
    await Task.CompletedTask;
}

And run the MsBuild command again. The outcome of the command should have following two lines in it.

No changes. Infrastructure is up-to-date.
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

The command output says that nothing is changed on AWS. If you observe the S3 bucket of the build, the package file should display as it is recently updated. But there is nothing changed in terms of Lambda. Sending new message lambdaqueue would also confirm that the Lambda function still logs the old messages.

The reason behind this behavior is the fact that there is nothing changed as per Terraform’s perspective. There is no change in the Terraform script. The SQS queue, the IAM role and its mappings, the Lambda function everything remained the same even when we made code change and rebuild the solution.

Terraform maintains the state of the infrastructure locally and compares it against the new changes while creating plan. And since there is no changes in the script Terraform did not perform any changes to the AWS resources.

So it is clear that the Terraform needs to see a change in the Terraform script perform the changes. In the Lambda function we are working with we are only changing code. So we need to figure out a way to tell Terraform that there is a change.

The changes made in the code are reflected in build package. Hence we need to tell Terraform that the build package is changed. The simplest way to do this would be to create a build package with different name each time the code is changed and provide the new name to the Terraform.

So let’s change the MsBuild script to generate build package with different name each time it is build. For example the different name of the package may look like lambdawithsqs.{somerandomstring}.zip.

To generate the radonm string we can use System.Guid and use string function SubString to fetch first 8 characters of it and use it to create final build package name.

Add following snippet before PackageName property.

<NewGuid>$([System.Guid]::NewGuid())</NewGuid>

Change PackageName property declaration as following.

<PackageName>lambdawithsqs.$(NewGuid.Substring(0,8)).zip</PackageName>

Now we have the new name for build package being generated each time we run MsBuild command. We need to pass this package name to the Terraform. Currently Terraform has the build package name hard coded as lambdawithsqs.zip.

Terraform makes this easy for with it support for input variables. We can pass the value for the variable while executing Terraform plan and Terraform apply commands. Change the target Deploy as following.

<Target Name="Deploy">
  <Message Text="Testing Terraform" />
  <Exec Command="terraform --version" />
  <Exec WorkingDirectory="..\Terraform" Command="terraform init" />
  <Exec WorkingDirectory="..\Terraform" Command="terraform plan -out=tfplan -input=false -var packagename=$(PackageName)" />
  <Exec WorkingDirectory="..\Terraform" Command="terraform apply -input=false -auto-approve -var packagename=$(PackageName)" />
</Target>

We are now passing the value of PackageName property of MsBuild script, which is now generated at run time and includes some random string in it, as value of packagename variable to the Terraform.

We need to declare a variable with name packagename in Terraform script to represent the package name and use it in Lambda function configuration to set value of s3_key property of it.

Declare a variable in deploylambda.tf file at line 5 as following.

variable "packagename" {
  default = "lambdawithsqs.zip"
}

And change the Lambda function resource configuration as following.

resource "aws_lambda_function" "lambdasample" {
  function_name = "lambdasample"

  s3_bucket = "lambdawithsqs"
  s3_key    = "${var.packagename}"

  handler = "lambdawithsqs::lambdawithsqs.Function::FunctionHandler"
  runtime = "dotnetcore2.1"
  publish = true
  role = "${aws_iam_role.lambda_role.arn}"
  timeout = 120
}

Notice Line 5 is using value of variable packagename in above code snippet.

Now when we run the MsBuild command with target BuildTestPackageUploadDeploy it should generate deployment package with different name and that name will be passed to the Terraform script. Terraform will detect the change as the package name is different than the previous one, it will go ahead and deploy the Lambda using the new package file.

Let’s give it a try.

dotnet msbuild ./Build/lambdawithsqs.proj /t:BuildPackageUploadDeploy

The outcome of this execution should be as following.

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

1 changed indicates that one resource on AWS is changed by Terraform. As per the changes we made above we are sure that nothing is changed except the deployment package name. And since Terrform detects that the deployment package name is changed, it does its job and deploys the new package to the Lambda function.

To verify if the Lambda function works as per the new code changes, send a message to the SQS queue and observe the CloudWatch Logs. The logs should include log messages such as Entering method ProcessMessageAsync and Exiting method ProcessMessageAsync.

So this way, everytime you make a code change and run above MsBuild command, you get new build package created, uploaded to S3 and deployed to Lambda function.

Later if there is anything changed such as SQS queue name or the S3 bucket name, all you need to do is change the bucket name and queue name in .proj file and deploylambda.tf file and you are ready to go. MsBuild would still build, package and upload the package to the new S3 location and Terraform will create a new SQS queue and mapping between the queue and the Lambda function and deploy the build from the new S3 location.