Docker on The Edge Cases

Building Docker Images on M1 Mac for Other Platforms

Sat, 19 Sep 2026 02:10:03 +0000

In order to build Docker images for Lambda functions with CDK on ARM or other non-Intel platforms, but want the Lambda to be on an x86_64 platform, use the platform parameter with the from_image_asset() function.

_lambda.DockerImage_function(
    ...
    code=_lambda.DockerImageCode.from_image_asset(  
        directory="./lambda",  
        file="./myservice/Dockerfile",  
        platform=ecr_assets.Platform.LINUX_AMD64,  
    ),
    ...
)

ecr_assets is a CDK package that also needs to be imported from the main aws_cdk package.

Docker Run Issues - Unable to Start Container

Thu, 22 Feb 2024 15:40:37 +0000

Sometimes you won’t be able to run a Docker container because some .so file can’t be found. This is how to fix it:

sudo apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl restart docker
Rebuild and re-run image

Error message

Running a Jupyter Notebook From a Docker Container Running on EC2

Wed, 22 Nov 2023 21:54:06 +0000

Given a Dockerfile on an EC2 instance that has, at the very least, these commands at the end:

EXPOSE 8888
ENTRYPOINT ["/bin/bash"]

Build the image

docker build -t <image name> .

Run the container

docker run -it -p 8888:8888 --ip=0.0.0.0 <image name>

If you want to persist data from the host to the container, include bind mounts:

docker run -it -p 8888:8888 -v /abs/path/to/data:/app/data --ip=0.0.0.0 <image name>

Within the container, run Jupyter Lab (note, specifying the port isn’t necessary if you’re using the default port 8888):

jupyter lab --port=8888 --ip=0.0.0.0 --allow-root

Troubleshooting

If you try to open Jupyter Lab in your browser, and you get a “refused to connect” error, ensure that the SSH config you have for your connection to your EC2 instance (or for SSH in general if you don’t have a specific entry for EC2) looks like this:

Working Within a Running Docker Container

Wed, 22 Nov 2023 02:34:18 +0000

First, build your image (assuming your Dockerfile is in the current directory):

docker build -t myimg .

Then call run:

docker run -it myimg

Using Bind Mounts to Speed Up Docker Development

Tue, 06 Jun 2023 20:41:50 +0000

When using docker run, the --mount parameter allows you to specify a directory to recreate within the Docker container so that you can edit your code locally and then run it within the Docker container without rebuilding the image.

Consider whether coding within a running container fits your use case better.

Example

docker run -it --mount type=bind,source="$(pwd)/tests",target=/tests image_tag

This uses the tests directory within the working directory as the source and puts the contents within a directory /tests in the container whenever you call docker run.

Remote Development with VS Code

Tue, 06 Jun 2023 20:27:18 +0000

Developing with Docker

To use Docker for developing in VS Code, you can use the Docker, Dev Containers and Remote Explorer extensions. Here’s a rough step-by-step guide:

Build an image if it hasn’t been built already from VS Code
Right click the image and select “Run interactive”
Then in the running container, right-click and run “Attach VS Code”
A new VS Code window will open and you can select a directory within the container to open You can also click the green icon in the lower left corner of the window to access the same options.

Developing on a Remote Server with SSH

To connect to a remote server and run VS Code on it, use the same Remote Explorer extension along with Remote - SSH:

Invalid Cross-Device Link when Deploying with CDK

Tue, 06 Jun 2023 20:20:58 +0000

When deploying with CDK, I started out of nowhere getting this error:

OSError: [Errno 18] Invalid cross-device link: '/tmp/pip-target-e_4qns87/lib/python/numpy' -> '/asset-output/numpy'

This seems to be caused by using gRPC FUSE for file sharing in Docker. To fix this, change the setting for file sharing implemnentations from gRPC FUSE to VirtioFS.

If this doesn’t work, it could imply a corrupted virtual environment. Build a new one and try again with that. This was what did the trick for me - but only for one deployment. Updating the few Python 3.8 lambda runtimes to 3.9 fixed it.

Logging from Custom Docker Code in Sagemaker

Tue, 10 Jan 2023 18:05:43 +0000

Unlike in AWS Lambda code, in Sagemaker training jobs using custom Python code in Docker containers logging won’t work unless a stream handler is specifically added to the logger.

logger = logging.getLogger("__main__")
stream_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stream_handler)

SAGEMAKER_PROGRAM in Custom Dockerfile

Tue, 10 Jan 2023 03:47:47 +0000

At the end of a Dockerfile that defines a container for a Sagemaker training job, you have to define the SAGEMAKER_PROGRAM environment variable. It needs to point directly to the script that acts as the entry point as it exists within /opt/ml/code, rather than the full path.

ENV SAGEMAKER_PROGRAM train.py

Deploying an Updated Image to Lambda Function

Wed, 04 Jan 2023 18:24:48 +0000

Redeploying a CDK stack with a Lambda function that relies on a Docker image in ECR won’t necessarily update the function with a newly pushed image. Instead, use the Lambda console’s “Deploy new image” button to push the ECR image to the function.

Troubleshooting Slow Docker Image Builds

Wed, 05 Oct 2022 14:31:11 +0000

Slow Build During Python Dependency Installation

This could be due to Docker attempting to build the dependency from the code rather than using a prebuilt wheel binary. This happens with older versions of dependencies, and especially hard-to-build dependencies like scikit-learn.

Installing Scipy with Docker

Tue, 04 Oct 2022 19:19:35 +0000

Certain packages can be troublesome to install in a Docker container when they’re defined in a requirements.txt file, such as scipy. If you’re running into issues installing these, which are typically problems with installing or building dependencies, try running pip install from the Dockerfile itself, then installing from requirements.txt.

Example

RUN pip install --no-cache-dir --upgrade pip && \  
    pip install numpy scipy cython && \  
    pip install -r requirements.txt --target ${FUNCTION_DIR}

Including Assets from Parent Directories in Dockerized Lambda Functions with CDK

Tue, 04 Oct 2022 17:31:19 +0000

If you have a Docker based Lambda that is defined in some directory and there are assets that the Lambda needs in the parent directory, from_image_asset() takes directory and file parameters, so you can select the parent directory as well as the specific Dockerfile that describes the environment for the Lambda.

Here is an example folder structure:

CDK_infrastructure/
    lambdas/
        lambda_one.py
        lambda_two.py
        lambda_pkg/
            ...
        docker_lambda/
            Dockerfile

And here is how you would set this up in your stack definition:

Addressing Docker No Space Left on Device Errors

Mon, 15 Nov 2021 23:03:06 +0000

When you build Docker assets with CDK, they are still built locally and aren’t removed when the process is complete. You may eventually hit this error: #8 29.62 ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device. This is because the images and containers are sitting in the Docker disk image that is of a fixed size. To fix this, you can prune all stopped containers, images, and volumes with docker system prune --all, volumes only docker system prune --volumes. To prune all images: docker prune images -af

To Build a Lambda from a Dockerfile in CDK

Wed, 27 Oct 2021 19:24:41 +0000

Creating the Dockerfile

To set up a Dockerfile for a Lambda function that relies on non-AWS base images, you have to include the Python package awslambdaric in your requirements and install it. Then, do the following in the Dockerfile:

ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]  
CMD ["lambda_code.function_name"]

Example

ARG FUNCTION_DIR="/function"
  
FROM tensorflow/tensorflow:2.4.3 as DEV  
ARG FUNCTION_DIR  
RUN mkdir -p ${FUNCTION_DIR}
## copies python code into container's FUNCTION_DIR
COPY lambda_code.py ${FUNCTION_DIR} 
## copies lambda_requirements.txt into container's requirements.txt
COPY lambda_requirements.txt ./requirements.txt  
## copies dataset into container's FUNCTION_DIR/dataset/
COPY dataset ${FUNCTION_DIR}/dataset/  
RUN pip install --no-cache-dir --upgrade pip && \  
    pip install -r requirements.txt --target ${FUNCTION_DIR}  
WORKDIR ${FUNCTION_DIR}  
ENTRYPOINT [ "python", "-m", "awslambdaric" ]  
CMD ["lambda_code.function_name"]

Setting Up the Lambda in CDK

To create a Lambda from a Dockerfile in CDK, use cdk.aws-lambda.DockerImageFunction when defining your Lambda: