Adding conda efs env

README.md

@@ -12,7 +12,9 @@ This repository contains examples of Docker images that are valid custom images
 - [r-image](examples/r-image) - This example contains the `ir` kernel and a selection of R packages, along with the AWS Python SDK (boto3) and the SageMaker Python SDK which can be used from R using `reticulate`
 - [rapids-image](examples/rapids-image) - This example uses the offical rapids.ai image from Dockerhub. Use with a GPU instance on Studio
 - [scala-image](examples/scala-image) - This example adds a Scala kernel based on [Almond Scala Kernel](https://almond.sh/).
-- [tf2.3-image](examples/tf23-image) - This examples uses the official TensorFlow 2.3 image from DockerHub and demonstrates bundling custom files along with the image.
+- [tf2.3-image](examples/tf23-image) - This example uses the official TensorFlow 2.3 image from DockerHub and demonstrates bundling custom files along with the image.
+- [conda-efs-image](examples/conda-efs-image) - This example creates a custom image in Amazon SageMaker Studio which will load a `custom` conda environment from a users [Elastic File System](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-manage-storage.html) (EFS) home folder.
+
 #### One-time setup
 
 All examples have a one-time setup to create an ECR repository

examples/conda-efs-image/Dockerfile

@@ -0,0 +1,52 @@
+FROM public.ecr.aws/ubuntu/ubuntu:20.04
+
+ARG NB_USER="sagemaker-user"
+ARG NB_UID="1000"
+ARG NB_GID="100"
+ARG NB_ENV="custom"
+
+######################
+# OVERVIEW
+# 1. Creates the `sagemaker-user` user with UID/GID 1000/100.
+# 2. Ensures this user can `sudo` by default.
+# 5. Make the default shell `bash`. This enhances the experience inside a Jupyter terminal as otherwise Jupyter defaults to `sh`
+######################
+
+# Setup the "sagemaker-user" user with root privileges.
+RUN \
+    apt-get update && \
+    apt-get install -y sudo wget nano && \
+    useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
+    chmod g+w /etc/passwd && \
+    echo "${NB_USER}    ALL=(ALL)    NOPASSWD:    ALL" >> /etc/sudoers && \
+    # Prevent apt-get cache from being persisted to this layer.
+    rm -rf /var/lib/apt/lists/*
+
+# installing miniconda
+RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+RUN bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda
+ENV PATH=$PATH:/miniconda/condabin:/miniconda/bin
+
+# install libraries for the base environment
+RUN conda install ipykernel
+
+# update kernelspec to load our custom environment strored on EFS
+RUN rm -rf /miniconda/share/jupyter/kernels/python3
+# write new kernel which initialises uses conda run for custom env
+# see: https://github.com/ipython/ipykernel/issues/416
+COPY custom_kernel_spec/ /miniconda/share/jupyter/kernels/python3
+
+# Make the default shell bash (vs "sh") for a better Jupyter terminal UX
+ENV SHELL=/bin/bash \
+    NB_USER=$NB_USER \
+    NB_ENV=$NB_ENV \
+    NB_UID=$NB_UID \
+    NB_GID=$NB_GID \
+    HOME=/home/$NB_USER
+
+# Set the conda envs path to map to EFS (without of requiring .condarc)
+ENV CONDA_AUTO_ACTIVATE_BASE=false \
+    CONDA_ENVS_PATH=/home/$NB_USER/.conda/envs
+
+WORKDIR $HOME
+USER $NB_UID

examples/conda-efs-image/README.md

@@ -0,0 +1,95 @@
+## Conda EFS Image
+
+### Overview
+
+This example creates a custom image in Amazon SageMaker Studio which will load a `custom` conda environment from a users [Elastic File System](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-manage-storage.html) (EFS) home folder.
+
+The Dockerfile is built from `ubuntu:20.04` and installs the latest [Minconda3](https://docs.conda.io/en/latest/miniconda.html) distribution.
+
+### Building the image
+
+Build the Docker image and push to Amazon ECR.
+```
+# Modify these as required. The Docker registry endpoint can be tuned based on your current region from https://docs.aws.amazon.com/general/latest/gr/ecr.html#ecr-docker-endpoints
+REGION=<aws-region>
+ACCOUNT_ID=<account-id>
+
+# Build the image
+IMAGE_NAME=custom-conda-efs
+aws --region ${REGION} ecr get-login-password | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom
+docker build . -t ${IMAGE_NAME} -t ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME} --build-arg NB_ENV=custom
+```
+
+NOTE: The `NB_ENV` maps to the conda environment which will need to be created on EFS (see below)
+
+```
+docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME}
+```
+
+### Using it with SageMaker Studio
+
+Create a SageMaker Image with the image in ECR. 
+
+```
+# Role in your account to be used for the SageMaker Image
+ROLE_ARN=<role-arn>
+
+aws --region ${REGION} sagemaker create-image \
+    --image-name ${IMAGE_NAME} \
+    --display-name "Conda EFS env" \
+    --role-arn ${ROLE_ARN}
+
+aws --region ${REGION} sagemaker create-image-version \
+    --image-name ${IMAGE_NAME} \
+    --base-image "${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME}"
+
+# Verify the image-version is created successfully. Do NOT proceed if image-version is in CREATE_FAILED state or in any other state apart from CREATED.
+aws --region ${REGION} sagemaker describe-image-version --image-name ${IMAGE_NAME}
+```
+
+Create an AppImageConfig for this image.
+
+```
+aws --region ${REGION} sagemaker create-app-image-config --cli-input-json file://app-image-config-input.json
+```
+
+Create a Domain, providing the SageMaker Image and AppImageConfig in the Domain creation. Replace the placeholders for VPC ID, Subnet IDs, and Execution Role in `create-domain-input.json`.
+
+```
+aws --region ${REGION} sagemaker create-domain --cli-input-json file://create-domain-input.json
+```
+
+If you have an existing Domain, you can use the `update-domain` command.
+
+```
+DOMAIN_ID=<domain-id>
+aws --region ${REGION} sagemaker update-domain --domain-id ${DOMAIN_ID} --cli-input-json file://update-domain-input.json
+```
+
+### Create your custom conda environment on EFS
+
+Use the [Amazon SageMaker Studio Launcher](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-launcher.html) to open a **System terminal** and run the following commands to setup your custom environment:
+
+```bash
+mkdir -p ~/.conda/envs
+conda create -p ~/.conda/envs/custom
+conda activate custom # or ". activate ~/.conda/envs/custom"
+conda install ipykernel
+```
+
+### Verifying your installation
+
+Use the [Amazon SageMaker Studio Launcher](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-launcher.html) to launch a **Notebook** for your *Conda EFS env* SageMaker Image and run the following command in the first cell to verify you are now using a `custom` conda environment:
+
+```
+!conda info
+```
+
+### Installing additional libraries
+
+Use the [Amazon SageMaker Studio Launcher](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-launcher.html) to launch an **Image terminal** and run the following commands to install new libraries in your custom environment such as `numpy`:
+
+```bash
+. activate /home/$NB_USER/.conda/envs/$NB_ENV
+conda install numpy
+```

examples/conda-efs-image/app-image-config-input.json

@@ -0,0 +1,16 @@
+{
+    "AppImageConfigName": "custom-conda-efs-env-config",
+    "KernelGatewayImageConfig": {
+        "KernelSpecs": [
+            {
+                "Name": "python3",
+                "DisplayName": "Python 3 (custom EFS env)"
+            }
+        ],
+        "FileSystemConfig": {
+            "MountPath": "/home/sagemaker-user",
+            "DefaultUid": 1000,
+            "DefaultGid": 100
+        }
+    }
+}

examples/conda-efs-image/create-domain-input.json

@@ -0,0 +1,19 @@
+{
+    "DomainName": "domain-with-conda-efs-image",
+    "VpcId": "<vpc-id>",
+    "SubnetIds": [
+        "<subnet-ids>"
+    ],
+    "DefaultUserSettings": {
+        "ExecutionRole": "<execution-role>",
+        "KernelGatewayAppSettings": {
+            "CustomImages": [
+                {
+                    "ImageName": "custom-conda-efs",
+                    "AppImageConfigName": "custom-conda-efs-env-config"
+                }
+            ]
+        }
+    },
+    "AuthMode": "IAM"
+}

examples/conda-efs-image/custom_kernel_spec/kernel.json

@@ -0,0 +1,9 @@
+{
+  "argv": [
+    "bash",
+    "-c",
+    "conda run -n $NB_ENV python -m ipykernel_launcher -f '{connection_file}'"
+  ],
+  "display_name": "Conda EFS env",
+  "language": "python"
+}

examples/conda-efs-image/update-domain-input.json

@@ -0,0 +1,12 @@
+{
+    "DefaultUserSettings": {
+        "KernelGatewayAppSettings": {
+            "CustomImages": [
+                {
+                    "ImageName": "custom-conda-efs",
+                    "AppImageConfigName": "custom-conda-efs-env-config"
+                }
+            ]
+        }
+    }
+}