In my daily role I am a software engineer at Union focusing on backend development. We are the core maintainers, and the most frequent contributors, to the open-source Flyte project. Per our documentation it is described as “The Workflow Automation Platform for Complex, Mission-Critical Data and Machine Learning Processes at Scale”. Basically, it is a framework to abstract execution of complex data workflows and the coinciding cloud infrastructure management. This enables teams to efficiently and effectively scale data processing.

Using k3d For Local Development / Testing

Being a cloud-native application, Flyte currently requires a k8s cluster for executing individual workflow tasks and therefore, the core components are typically deployed in k8s. While this is a battle-proven production solution, it can be cumbersome for local development, where testing implementation changes must be (1) compiled into a docker image, (2) pushed to a remote docker registry, and (3) deployed to the k8s cluster. Thankfully, this is a common problem and there are well designed solutions.

k3d is a tool which constructs and manages localized k8s development environments. I can not overstate how useful this is for my development cycle. It uses docker to spawn individual k3s containers to provide lightweight k8s servers and agents. Additionally, it handles all of the internal networking including k8s port forwarding, volume mounts, etc.

Using this we can quickly stand up and tear down k8s testing clusters locally. An example command which I frequently use to create a testing cluster is provided below:

hamersaw@ragnarok:~$ k3d cluster create -p "30081:30081@server:0" -p "30084:30084@server:0" -p "30087:30087@server:0" --image rancher/k3s:v1.21.8-k3s2 -v /dev/mapper/nvme0n1p3_crypt:/dev/mapper/nvme0n1p3_crypt flyte
WARN[0000] No node filter specified
INFO[0000] Prep: Network
INFO[0000] Re-using existing network 'k3d-flyte' (cae674a191e6540d864e0df3af767a8392f3495937595c0d637e374955858e93)
INFO[0000] Created image volume k3d-flyte-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-flyte-tools'
INFO[0001] Creating node 'k3d-flyte-server-0'
INFO[0001] Creating LoadBalancer 'k3d-flyte-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] HostIP: using network gateway 172.28.0.1 address
INFO[0001] Starting cluster 'flyte'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-flyte-server-0'
INFO[0006] All agents already running.
INFO[0006] Starting helpers...
INFO[0006] Starting Node 'k3d-flyte-serverlb'
INFO[0012] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap...
INFO[0014] Cluster 'flyte' created successfully!
INFO[0014] You can now use it like this:
kubectl cluster-info

Similarly, we can delete the cluster using the command below:

hamersaw@ragnarok:~$ k3d cluster delete flyte
INFO[0000] Deleting cluster 'flyte'
INFO[0001] Deleting 2 attached volumes...
WARN[0001] Failed to delete volume 'k3d-flyte-images' of cluster 'flyte': failed to find volume 'k3d-flyte-images': Error: No such volume: k3d-flyte-images -> Try to delete it manually
INFO[0001] Removing cluster details from default kubeconfig...
INFO[0001] Removing standalone kubeconfig file (if there is one)...
INFO[0001] Successfully deleted cluster flyte!

Hosting Private Repositories with the Docker Registry Image

The Flyte system architecture is designed on microservices, which provide improved development speeds and scalability compared to a monolithic design. However, this does mean that we are forced to download a variety of container images every time we initialize Flyte deployment. In total, these may be on the order of 100s of MB which for some Internet connections is asking a lot. Unfortunately there does not seem to be a way to reuse local docker images within a k3d deployment. In my understanding this has to do with image format incompatibility.

Fortunately docker provides an image to host private repositories, namely the registery:2 image on dockerhub. This image enables us to start a fully-compatible docker registry locally.

For our use-case we being by creating a docker volume to use as persistent storage. We could have alternatively used a local filesystem path as well.

hamersaw@ragnarok:~$ docker volume create registry-flyte
registry-flyte

Then we run the image using the REGISTRY_PROXY_REMOTEURL environment variable to use this container as a remote proxy to the specified URL.

hamersaw@ragnarok:~$ docker run -d --publish target=5000,published=127.0.0.1:5000 --name registry-flyte.local --env REGISTRY_PROXY_REMOTEURL=http://cr.flyte.org --volume registry-flyte:/var/lib/registry registry:2
6fd4dee8702ec82f4eb70086d460d73df87999d50bd09f1158a61a4183aa4977

Now we have a local docker registry serving on the address 127.0.0.1:5000 which proxies to http://cr.flyte.org and stores images in the docker volume registry-flyte.

Enable the Pull-Through Registry Within k3d

To setup k3d to use our local docker registry proxy as a pull-through cache we need to inform k3d about which images should use the registry. This configuration requires a single yaml file, in our case registries.yaml. The syntax defines a mirror for the ‘cr.flyte.org" repository to redirect to the endpoint of our local registry. There is ample documentation about using private / local registries within k3d and specifically on the syntax of the configuration file. An example is provided below:

mirrors:
  "cr.flyte.org":
    endpoint:
      - "http://registry-flyte.local:5000"

Then we can modify our k3d cluster creation command to include the registry config with the --registry-config flag.

hamersaw@ragnarok:~$ k3d cluster create -p "30081:30081@server:0" -p "30084:30084@server:0" -p "30087:30087@server:0" --image rancher/k3s:v1.21.8-k3s2 -v /dev/mapper/nvme0n1p3_crypt:/dev/mapper/nvme0n1p3_crypt --registry-config ~/documents/union/registries.yaml flyte
WARN[0000] No node filter specified
INFO[0000] Prep: Network
INFO[0000] Re-using existing network 'k3d-flyte' (cae674a191e6540d864e0df3af767a8392f3495937595c0d637e374955858e93)
INFO[0000] Created image volume k3d-flyte-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-flyte-tools'
INFO[0001] Creating node 'k3d-flyte-server-0'
INFO[0001] Creating LoadBalancer 'k3d-flyte-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] HostIP: using network gateway 172.28.0.1 address
INFO[0001] Starting cluster 'flyte'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-flyte-server-0'
INFO[0006] All agents already running.
INFO[0006] Starting helpers...
INFO[0006] Starting Node 'k3d-flyte-serverlb'
INFO[0012] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap...
INFO[0014] Cluster 'flyte' created successfully!
INFO[0014] You can now use it like this:
kubectl cluster-info

And finally, we need to ensure that the local docker registry image is accessible through the k3d network by manually connecting it.

hamersaw@ragnarok:~$ docker network connect k3d-flyte registry-flyte.local

Now we have a k3d cluster that will use our local docker registry and a pull-through cache for images from cr.flyte.org!

Testing

To test this setup we can easily run a recent flytepropller image (just opening a shell which we will exit from immediately). This requires k3d to pull the remote image locally.

hamersaw@ragnarok:~$ kubectl run -it --rm --image cr.flyte.org/flyteorg/flytepropeller:v0.16.44 test-docker-cache sh
If you don't see a command prompt, try pressing enter.
/ $ exit
Session ended, resume using 'kubectl attach test-docker-cache -c test-docker-cache -i -t' command when the pod is running
pod "test-docker-cache" deleted

Using the registry REST API we first check the catalog contains the flyteorg/flytepropeller image and then validate the tags listed.

hamersaw@ragnarok:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["flyteorg/flytepropeller"]}
hamersaw@ragnarok:~$ curl 127.0.0.1:5000/v2/flyteorg/flytepropeller/tags/list
{"name":"flyteorg/flytepropeller","tags":[..., "v0.16.43","99acef135bf44fdce8dcb6351c128628fcea0ed8","v0.16.44","eef8c265f4859f6dc0a0b745ac0339f5d2cbb1eb","v0.16.45","157cb56dd5c9faf0978f12bae3535a5636645789","v0.16.46","latest","d3bef3c3c01127b9321e075ae84c2b27d598cd8a","v0.16.47"]}

If that is not enough we can check the sizes of all docker volumes to ensure that our registry-flyte volume has data, specifically the aforementioned image.

hamersaw@ragnarok:~$ docker system df -v
Images space usage:

REPOSITORY                   TAG                                        IMAGE ID       CREATED         SIZE      SHARED SIZE   UNIQUE SIZE   CONTAINERS
...
rancher/k3s                  v1.21.8-k3s2                               e664680dc6d9   3 months ago    169.7MB   0B            169.7MB       1
..

Containers space usage:

CONTAINER ID   IMAGE                            COMMAND                  LOCAL VOLUMES   SIZE      CREATED         STATUS         NAMES
d4e2d9d42101   ghcr.io/k3d-io/k3d-proxy:5.4.1   "/bin/sh -c nginx-pr…"   1               1.44kB    3 minutes ago   Up 3 minutes   k3d-flyte-serverlb
0894c31c2a96   rancher/k3s:v1.21.8-k3s2         "/bin/k3d-entrypoint…"   5               5.32kB    3 minutes ago   Up 3 minutes   k3d-flyte-server-0
6fd4dee8702e   registry:2                       "/entrypoint.sh /etc…"   1               0B        4 minutes ago   Up 4 minutes   registry-flyte.local

Local Volumes space usage:

VOLUME NAME                                                        LINKS     SIZE
...
registry-flyte                                                     1         86.82MB

Build cache usage: 0B

CACHE ID   CACHE TYPE   SIZE      CREATED   LAST USED   USAGE     SHARED

Now when we deploy Flyte all our images from cr.flyte.org are locally cached in the registry-flyte docker volume. Similarly, you can easily add additional pull-through registries for docker.io, query.io, etc by standing up a docker container with the correct proxy URL, adding the registry to the k3d registries configuration, and connection the docker image to the k3d network.

13 day(s) offloaded in the 100DaysToOffload challenge.