Using eStargz to reduce container startup time on Amazon EKS

Photo by Alexandar Todov / Unsplash

A container image bundles executable code, library, and configuration. Images contain everything an application needs to run. It is a best practice to exclude any file that’s unnecessary for the application packaged in the image. A smaller image means that when you create a container, the container runtime (dockerd or containerd) will have to download fewer bits from the container registry, which will result in faster startup time.

There are cases when the container image becomes significantly large (>500 MB). A common scenario is machine learning workloads. ML workload containers usually package model data necessary for the application. The image size for these containers can easily span in to multiple GBs. As a result, these applications have a slower startup time when the runtime has to pull the image. In fact, research shows that pulling packages accounts for 76% of container start time, but only 6.4% of that data is read.


There are two open source projects designed to improve container startup time. Stargz and SOCI are containerd plugins that reduce the cold start time by providing a way to run containers without downloading the entire image. They introduce the concept of lazy pulling, a technique that allows the runtime to download the bits from the container registry as needed.

Using lazy pulling significantly reduces the application startup time. eStargz is a lazily-pullable image format that is compatible with OCI runtimes and standard container registries like DockerHub, GitHub Container Registry.


The eStargz image format is based on stargz image format by Container Registry Filesystem (CRFS) open source project. CRFS is a read-only FUSE filesystem that lets you mount a container image, served directly from a container registry, without pulling it all locally first. The project introduces Seekable tar.gz format, which makes tar.gz files seekable using an index.

Stargz Snapshotter plugin

Stargz snapshotter is implemented as a proxy plugin daemon (containerd-stargz-grpc) for containerd. When containerd starts a container, it queries the rootfs snapshots to stargz snapshotter daemon through a unix socket. This snapshotter remotely mounts queried eStargz layers from registries on the node and provides these mount points as remote snapshots to containerd. The plugin uses FUSE to mount eStargz layers directly from the container registry.


Running eStargz images on Amazon EKS

eStargz images are a little different from the images you’d build using docker build. In order to create an image that supports lazy pulling, you'll need an eStargz-aware image builder or a converter.

I am going to build my eStargz image using nerdctl, which is an eStargz-aware image builder. Since image size is not an issue, I will use Debian Jessie as the base image to demo. To make the image size artificially large, I will include twenty 50 MB files.

Lets generate large files containing random text:

mkdir files
for i in {0..20}; do base64 /dev/urandom | head -c 50000000 > files/file${i}.txt; done

Create a DockerFile:

cat > Dockerfile <<EOF
FROM debian:jessie
RUN apt-get update && apt-get install -y \
COPY files .


I am going to store my image in Amazon ECR. I’ll create an ECR repository:

ECR_URI=$(aws ecr create-repository \
  --repository-name estargz-demo \
  --query 'repository.repositoryUri' \
  --output text)

Use nerdctl to create container image:

sudo nerdctl build -t ${ECR_URI}:1 .

The image is not in eStargz formatted right now, I’ll have to convert it:

nerdctl image convert --estargz --oci \
  ${ECR_URI}:1 ${ECR_URI}:1-esgz

The resulting images:

nerdctl images
REPOSITORY                                                   TAG       IMAGE ID        CREATED           PLATFORM       SIZE       BLOB SIZE    1         798b85a131ed    31 minutes ago    linux/amd64    1.2 GiB    828.8 MiB    1-esgz    81b0ffd2a4a3    36 seconds ago    linux/amd64    0.0 B      832.3 MiB

Login to ECR and push the image to ECR:

aws ecr get-login-password | sudo nerdctl login \
   --username AWS \
   --password-stdin \
nerdctl push ${ECR_URI}:1-esgz

The image is now available in ECR and I can start running it in my EKS cluster.

Create a managed node group that uses containerd

EKS nodes currently don’t enable containerd by default. I’ll create a node group that uses containerd.

Create environment variables for AWS Region and EKS cluster name:

AWS_REGION=<Your AWS Region>
CLUSTER_NAME=<Your EKS cluster's name>

First, I need to retrieve the id of the EKS optimized AMI in my region:

aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.23/amazon-linux-2/recommended/image_id --region us-west-2 --query "Parameter.Value" --output text
cat > containerd-mng.yaml <<EOF
kind: ClusterConfig

  region: $AWS_REGION

  - name: containerd-mng
    minSize: 1
    maxSize: 1
    desiredCapacity: 1
    instanceType: m5.xlarge
    ami: $AMI_ID
    overrideBootstrapCommand: |
      /etc/eks/ Socrates --container-runtime containerd

Preparing EKS nodes to use eStargz

Next, I need to install eStargz snapshotter plugin on my worker node. I use AWS Systems Manager on my nodes, so I will connect to the containerd node.

aws ssm start-session --target <Instance ID of the worker node>

Connect to the node (using ssh or Systems Manager) and back up the current containerd config file (/etc/containerd/config.toml) and replace it with:

sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.bak
cat > config.toml <<EOF
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

address = "/run/containerd/containerd.sock"

default_runtime_name = "runc"
snapshotter = "stargz"
disable_snapshot_annotations = false

sandbox_image = ""

runtime_type = "io.containerd.runc.v2"

SystemdCgroup = true

bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"
sudo mv config.toml /etc/containerd/config.toml

Install FUSE:

sudo yum install fuse -y
sudo modprobe fuse
sudo bash -c 'echo "fuse" > /etc/modules-load.d/fuse.conf'

Install the snapshotter from its GitHub repository:

sudo tar xvzf stargz-snapshotter-v0.12.1-linux-amd64.tar.gz -C /usr/local/bin
sudo wget -O /etc/systemd/system/stargz-snapshotter.service
sudo systemctl enable --now stargz-snapshotter

Finally, restart the containerd and kubelet:

sudo systemctl restart containerd
sudo systemctl restart kubelet

Test lazy-pulling

I will now run the pod using my eStargz formatted image. Create a manifest for a new pod and replace the node name with the newly created node:

cat > estargz-pod.yaml <<EOF
apiVersion: v1
kind: Pod
  name: stargz-demo
  - name: stargz-demo
    image: ${ECR_URI}:1-esgz
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 30; done;" ]

The pod started in 2 seconds:

k get pods
NAME                            READY   STATUS        RESTARTS      AGE
stargz-demo                     1/1     Running       0             2s

To compare, I ran the same pod on another node that didn’t use containerd. It took 45 seconds to start the same image. That’s a lot of improvement in pod startup time!!

Image pull time without eStargz:


Image pull with eStargz:


Prefetching files

What if you wanted the runtime to always download a file and disable lazy-pulling?

eStargz supports prefetching of files. This mitigates runtime performance drawbacks caused by the on-demand fetching of each file.

The example below always pulls ls and bash files before starting the container:

$ cat <<EOF > /tmp/record.json
{ "path" : "/usr/bin/bash" }
{ "path" : "/usr/bin/ls" }
$ nerdctl image convert --estargz --oci \
    --estargz-record-in=/tmp/record.json \
    ubuntu:21.04 ubuntu:21.04-ls

Seekable OCI (SOCI)

Seekable OCI (SOCI) is a technology open sourced by AWS that enables containers to launch faster by lazily loading the container image. SOCI works by creating an index (SOCI Index) of the files within an existing container image. This index is a key enabler to launching containers faster, providing the capability of extracting an individual file from a container image before downloading the entire archive.

SOCI borrows some of the design principles from stargz-snapshotter, but takes a different approach.

A SOCI index is generated separately from the container image, and is stored in the registry as an OCI Artifact and linked back to the container image by OCI Reference Types. This means that the container images do not need to be converted, image digests do not change, and image signatures remain valid.

Most OCI registries like DockerHub and ECR do not currently support the "referrers" feature. So you cannot use SOCI unless you run a local ORAS registry.


If you are looking to reduce container start time for your workloads, eStargz snapshotter is definitely worth a look. You’ll have to change your existing container build pipelines to add a step to convert images though. When SOCI support is available in OCI registries, you’ll be able to lazily-pull images without converting them first.

You’ll also have to install eStargz snapshotter on your nodes and configure containerd to use the plugin. Creating a custom AMI or using AWS Systems manager will be the best way to do that. You could also use a DaemonSet to configure the system, but keep in mind that you’ll need to account for restarting the containerd and kubelet.