If you would like to see the TLDR; version of this article please click TLDR;
Building a lambda Layer for Python
AWS has now provided away for lambda functions to share depencies. This provides the advantage to not only ensure our lambda functions are using the same base code, but also provides a way to save storage space in Lambda. Currently, lambda has a per region limit of 75GB for function and layer storage. That may sound like a lot of storage, but I have seen this storage exhausted.
Using layers will not provide additional storage to the 50 MB zipped /250 MB unzipped lambda function size limit. A lambda function can have up to 5 layers attached.
What is a Lambda Layer?
A lambda layer is a compressed file that will include dependencies for your application that can be shared across lambda functions. This can include third party libraries installed by the pip package manager.
While it is possible to build a some lambda layers in Windows, many packages will require the use of third party dependencies that may not function when the layer is built in windows. It is recommended to use the AWS Linux AMI or a Cloud 9 environment to build your lambda packages.
Building the Lambda Layer.
For this example, we are going to use the third party Python library called pymysql. This package is used as a MySQL client used to connect to MySQL databases.
To continue you will Python and pip installed on your local machine. For Linux you will need the zip command line utility installed. To compresse the package in Windows you will need 7zip or another third party compression utility. While using the default compression utility in Windows may work, I often hear from users that this utility caused issues.
Let’s start by taking a look at the following commands one line at a time.
We start by creating a new working folder for our lambda Layer. Inside we create a second folder named python.
mkdir lambda_layer cd lambda_layer mkddir python cd python
Next, we use the pip package manager to install the package into our working folder. The “-t” argument passed to pip, instructs pip to install the package in the “Target” location instead of at a system location.
pip install pymysql -t ./
The directory structure for the lambda layer should look like this.
lambda_layer python pymysql pyMySQL-0.9.3.dist-info
Now it is time to compress these files into our Lambda layer. This is where things will become a little different for Windows and Linux/Mac users
For Linux/Mac, the zip command can be used. Please note the “.” at the end of this command. The “.” is used as a wild card for the zip command.
cd .. zip -r /tmp/lambda_layer.zip .
For Windows, navigate file explorer to the root of the lambda_layer folder. Right click the python folder and choose “7-zip -> add to python.zip”. A new zip file will be created in the lambda_layer directory.
Navigate to the Lambda portion of the AWS Web Console. From the left menu choose layers and click “Create Layer.
Follow the prompts to create the lambda layer.
For Windows, choose the local zip file created.
For Linux/Mac/Cloud 9, choose "upload from S3". Provide the bucket and file name. Example: s3://geektopia-lambda-layers/lambda_layer.zip
Select all of the runtimes you would like this layer to be made available for. When finished, click “Create” at the bottom of the screen
Navigate back to the lambda function. Push the “Layers” button in the center of the console. Scroll down and choose the lambda layer from the drop down list. Choose version 1 to the right.
Be sure to click the save button for the layer to be added to the lambda function.