Reducing Clojure Lambda Cold Starts Part 1

Clojure Lambdas have some pretty abysmal memory footprints and cold start times compared to most other languages, especially Go, Node.js, and Python. Loading the Clojure core library and application code makes this worse. Regardless, I've found Clojure to be a great language for many data processing tasks and I've used it extensively for event-driven flows using Lambda. For many of the workloads I've run with Clojure Lambdas in the past, they have been periodic ETL jobs where the cold start times haven't been much of an issue as far as workflow time requirements, although the extra expense of having to use larger instances to accommodate the JVM/Clojure memory footprint along with the additional runtime caused by the cold starts has been concerning. Recently, however, I have been working on some near-real-time data flows for which the cold starts are a major issue.

To hack around the problem and eliminate most cold starts, we've had to use provisioned concurrency, which introduces huge additional expense to provision compute that is being wasted most of the time. One of the things I love about Lambda is the efficiency. It allows you to reduce your costs and carbon footprint by only using the compute you need. Having to provision concurrency on Clojure Lambdas negates a lot of this efficiency. This has created an existential crisis for me. Do I need to totally rethink using Clojure and perhaps switch to Rust? Do I need to rethink Lambda and switch Clojure workloads to ECS?

In the past, I have done some experiments with switching Clojure Lambdas to ClojureScript Lambdas running on the Node.js runtime. This has seemed to greatly reduce the cold start times, but requires some significant rewrite of code that calls AWS SDKs and I suspect the compiled JavaScript will be significantly slower for some of the heavy computation my current workloads require. This is the first in series of posts where I will be doing performance comparisons for Lambdas running Clojure on the JVM, Clojure on GraalVM, ClojureScript on Node.js, and Rust. I will start with gathering the baseline cold starts for a JVM Lambda.

Setting Up

First, I need to create a new project, which I will call tax-engine-experiment, as I will be doing comparisons on capital gains tax calculations:

>> mkdir tax-engine-experiment
>> cd tax-engine-experiment
>> git init

Previously, for Lambda, I have typically used Leiningen because of the Clojure CLIs lack of support for building uberjars, which I typically use for Lambda. Recently, however, such support has been added with tools.build, so I will try that. This requires that I add a build.clj at the root application level:

>> emacs build.clj

Then add the following code:

(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'taxbit/tax-engine)
(def version "0.1.0")
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/%s-%s-standalone.jar" (name lib) version))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src" "resources"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src"]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'tax.core}))

I also need a deps.edn file at the root level for the Clojure CLI:

>> emacs deps.edn

With the following code:

{:paths ["src/clj"]
 :aliases {:build {:deps {io.github.clojure/tools.build {:tag "v0.7.2" :sha "0361dde"}}
                   :ns-default build}}}

And I need a basic Lambda handler:

emacs src/clj/tax/core.clj

(ns tax.core
  (:gen-class
   :methods [^:static [calculationsHandler [Object] Object]]))

(defn -calculationsHandler [event]
  (prn "HELLO EVENT" event)
  event)

Now I can compile the project to uberjar:

>> clj -T:build uber

Now I need my Serverless Application Model (SAM) template.yml:

AWSTemplateFormatVersion: "2010-09-09"

Transform:
- "AWS::Serverless-2016-10-31"

Resources:

  RunCalculationsQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${AWS::StackName}-run-calcs-queue"
      VisibilityTimeout: 5400

  RunCalculations:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Sub "${AWS::StackName}-run-calcs"
      Handler: tax.core::calculationsHandler
      Runtime: java11
      CodeUri: target/tax-engine-0.1.0-standalone.jar
      Timeout: 900
      MemorySize: 2048
      Policies:
        - AWSLambdaBasicExecutionRole
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt RunCalculationsQueue.Arn
            BatchSize: 1

Outputs:

  QueueUrl:
    Value: !Ref RunCalculationsQueue

And use the SAM CLI to verify everything works:

>> sam local invoke
...
START RequestId: 8e2f1a01-11fe-4ff1-9b35-4cc8f5f152fa Version: $LATEST
"HELLO EVENT" {}
END RequestId: 8e2f1a01-11fe-4ff1-9b35-4cc8f5f152fa
REPORT RequestId: 8e2f1a01-11fe-4ff1-9b35-4cc8f5f152fa    Init Duration: 1.13 ms    Duration: 30247.04 ms    Billed Duration: 30248 ms    Memory Size: 2048 MB    Max Memory Used: 2048 MB

Looks good. Now I can deploy it to my AWS sandbox:

>> sam package --s3-bucket larrys-cool-bucket-name
>> sam deploy --stack-name tax-engine-experiments --s3-bucket larrys-cool-bucket-name --capabilities CAPABILITY_IAM

Now I can invoke the Lambda to make sure it deployed:

>> aws lambda invoke --function-name tax-engine-experiments-run-calculations results.txt

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

So far so good. Now I need to generate as many cold starts as I can by triggering current invocations. I'll throw a bunch of messages on the queue. I'll add AWS dependencies and a 'profile' alias:

>> emacs deps.edn

{:paths ["src/clj"]
 :aliases {:build {:deps {io.github.clojure/tools.build {:tag "v0.7.2" :sha "0361dde"}}
                   :ns-default build}
           :profile {:extra-paths ["dev/clj"]
                     :deps {software.amazon.awssdk/sqs {:mvn/version "2.17.100"}
                            software.amazon.awssdk/sso {:mvn/version "2.17.100"}}}}}

Note that I needed the 'sso' dependency because I use 'aws configure sso' since I use SSO to access my sandbox account. Now I need to add the code to push the messages:

>> emacs dev/clj/tax/profile.clj

(ns tax.profile
  (:import (software.amazon.awssdk.services.sqs SqsClient)
           (software.amazon.awssdk.services.sqs.model SendMessageRequest)))

(defn profile []
  (let [sqs (-> (SqsClient/builder) (.build))
        req (-> (SendMessageRequest/builder)
                (.queueUrl <<queue URL copied from deploy output>>)
                (.messageBody "Hello Calculations")
                (.build))]
    (dotimes [i 1000]
      (.start (Thread. (fn [] (.sendMessage sqs req)))))))

Once the queue has drained and the logs have finished populating, I'll use CloudWatch Log Insights to get the average:

Screen Shot 2021-12-18 at 5.53.19 PM.png

So it looks like the average cold start time with minimal code and no extra dependencies (although the package size is still 4.5 MB!) is about 2.6567039 seconds across 85 cold starts. Unfortunately I was only able to trigger 85 cold starts across the 1000 invocations, but that should be good enough for now. I'll expand upon this baseline with different memory sizes and more realistic dependencies and calculation code in later posts.

Reducing Clojure Lambda Cold Starts Part 1 - Baseline

Setting Up