Cloud cost management: lazy loading in AWS lambda using Python

Auteur
Ioannis Petrousov
Datum

It is a known fact that outsourcing your services to the cloud brings a lot of benefits such as reduced operational activity and almost unlimited and on-demand resources. However, those benefits come with additional billing costs which can increase dramatically if not enough attention is paid to it. One way to reduce the cost of cloud infrastructure is by making the software that runs on top of it intelligent and efficient. There is a plethora of design patterns which - when applied to our software designs - can result in a better utilization of hardware resources. One of such design patterns is called Lazy Loading and in this technical focused blogpost I will examine an approach on how to create a Lambda which utilizes that technique.

drew-patrick-miller-u1iFdXU1tl0-unsplashPhoto by Drew Patrick Miller on Unsplash

Introduction

One of our client’s implementations contained a Lambda with 128MB memory allocated to it. This function loads Amazon Application ELB logs from an object, stores them into S3 and memory and finally the function converts the logs into a UTF-8 string while keeping copies of both forms in memory. This specific function was operating fine under normal conditions. However, during traffic peaks I noticed that it was being killed by AWS with a RuntimeExceptionError. Looking further into the Logs using CloudWatch, I discovered that it was trying to consume 129MB of memory which was the tipping point. Instead of just increasing the memory allocated to that Lambda, I decided to investigate how I could make it smarter and more efficient using the Lazy loading approach.

So what is Lazy loading?

According to wikipedia, Lazy loading is a technique in which the initialization of an object is being held off until the data of the object is needed in the program. In other words, during execution, a program should keep in memory only the data that is necessary for the commands it executes.

Why this is important?

According to the pricing model, not taking into account any side calls, whenever a Lambda is executed, you are charged for the number of requests and the duration. However, the price tier depends on the amount of memory the Lambda uses during its execution. This means that when the memory tier is higher, the pricing tier for the Lambda is also higher. So, optimizing your functions to require less memory and run faster, results into reducing your bill.

lazyloading1

Implementing Lazy loading in Python

In our case, whenever a new object is stored in S3, it creates a CloudWatch event which triggers our Lambda.

lazyloading2

The event variable contains all the necessary data and is passed into the handler function. All we need to do is extract that information from the dictionary.

Get the object

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    source_bucket_name = event['Records'][0]['s3']['bucket']['name']
    key_name = event['Records'][0]['s3']['object']['key']
    response = s3.get_object(Bucket=source_bucket_name, Key=key_name)

The get_object() returns a dictionary which contains metadata related to the object that was just added into S3. The response[“Body”] allows us to open a stream to that object. A stream is a file-like object which allows us to read data from the object. If interested in going further, you can read the documentation from Amazon:

Open the stream using TextIOWrapper()

Having created the stream to the object, we finally open it in binary mode rb and decode it into a UTF-8 string by wrapping the gzip.GzipFile() into the TextIOWrapper(). The last one, not only allows us to read the stream into a string but also does it in an efficient way by decoding it in chunks thus making our program more efficient in terms of memory consumption and execution time.

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    source_bucket_name = event['Records'][0]['s3']['bucket']['name']
    key_name = event['Records'][0]['s3']['object']['key']
    response = s3.get_object(Bucket=source_bucket_name, Key=key_name)
    # Open the stream in binary mode [objects in s3 are binary]
    # TextIOWrapper decodes in chunks.
    stream_content = io.TextIOWrapper(gzip.GzipFile(None, 'rb', fileobj=response['Body']))

If interested in knowing more about the specific implementation of GzipFIle() and TextIOWrapper(), I suggest reading their documentation:

Reading data from the stream

At this point, we read the data from the stream with a simple “for loop”. The specific Lambda function was trying to filter the logs for HTTP 460 responses from the ELBs.

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    source_bucket_name = event['Records'][0]['s3']['bucket']['name']
    key_name = event['Records'][0]['s3']['object']['key']
    response = s3.get_object(Bucket=source_bucket_name, Key=key_name)
    stream_content = io.TextIOWrapper(gzip.GzipFile(None, 'rb', fileobj=response['Body']))

# Count HTTP 460s per targetgroup in AWS ELB [dict]    

http_460s_per_targetgroup = {}
    for each_line in stream_content:
    if (" 460 " in each_line) and ("targetgroup" in each_line):
        # Do some regex magic to find "targetgroup"
        targetgroup_name =\
        return_target_group_from_string(each_line)

        increment_targetgroup_460_counter(\
            http_460s_per_targetgroup,\
            targetgroup_name)
    else:
        logging.info("False HTTP 460 log does not contain a target group")

Comparison results

By making the Lambda smarter, I noticed a big difference in its execution time. I’ll let you figure out where the change in the code was applied ;)

lazyloading3Execution time over a period of 10 days

Unfortunately, CloudWatch does not have memory consumption charts and so, I could only pull out several log messages. Moreover, the retention period for the logs was configured to 7 days, so these are the last logs I could find. I had to remove the info messages.

lazyloading4a

lazyloading4bSignificantly improved statistics

To make sure that the new function is safe, I doubled the amount of memory (265MB) it had initially allocated. This is a safe measure I believe was logical to prevent it from failing.

Conclusion

There are multiple blogposts and guides online which describe how to read/write data from/to an object in S3 bucket. However, only a few describe how you can make your functions more efficient.

In my case, improving the Lambda probably didn’t save a lot of cost, however, if implemented at scale or in a serverless environment, it can result in a substantial cost reduction and also make the system faster.

From a technical perspective, it’s preferred to read the contents of file objects using generators instead of dumping everything into memory.

You can download the final code in gist https://gist.github.com/gpetrousov/0b1b831c516db43574f2e11b970d4f72

Thanks for reading!


References:

Tags

DevOps Cloud Design patterns Development