Thursday, 9th March 2023
While working at LeapYear I was charged with managing our production Artifactory instance. We used infrastructure as code, but also wanted things to be as simple as possible to backup/restore/migrate/deploy/destroy as possible. Instead of just creating a monolithic doc to list all the console commands to do these things, we decided to script them (and then create a doc with the script commands).
First this was written in Bash, however Bash isn’t great for anything that starts to become complex. We though about moving the script to Python, however Python isn’t the best at running system commands. So, I decided to write Calligraphy as a solution.
Calligraphy is a hybrid programming language (really it’s just a transpiler under the hood) which allows you to write Bash inline in your Python scripts. For example, you can write:
#! /usr/bin/env calligraphy
# type: ignore
# This file will get the resources for all containers and initContainers present in pods
# in your currently active Kubernetes cluster/namespace
# It will then print them out neatly so you can verify what is set on each pod/container
import json
pods = ?(kubectl get pods | tail -n +2 | awk '{{print $1}}').split('\n')[:-1]
for pod in pods:
print(pod)
pod_data = json.loads(?(kubectl get pod {pod} -o json))
print(' containers')
for container in pod_data['spec']['containers']:
if 'limits' in container["resources"].keys():
print(f' limits : {container["resources"]["limits"]}')
if 'requests' in container["resources"].keys():
print(f' requests : {container["resources"]["requests"]}')
if 'initContainers' in pod_data['spec'].keys():
print(' init containers')
for init_container in pod_data['spec']['containers']:
if 'limits' in init_container["resources"].keys():
print(f' limits : {init_container["resources"]["limits"]}')
if 'requests' in init_container["resources"].keys():
print(f' requests : {init_container["resources"]["requests"]}')
Now you could just use the kubernetes Python package, however kubectl is already installed and we know the commands there to do what we want already. So, we grab the pods via Bash commands for convenience and us the results in the Python code that follows. While this is a toy example, the power of this becomes apparent in more complex situations. For example, take the following build script example:
#! /usr/bin/env calligraphy
# type: ignore
# This file builds either locally or via Docker depending on the release mode
# It's a modified version of the studfile commands in Velocimodel
def clean():
# Remove any existing build artifacts
rm -r dist || true
rm -r release || true
rm -r src/frontend/ui-dist || true
def build_docker(services: list) => None:
"""Build docker images of various services
Args:
services (list): List of services to build
"""
clean()
version = $(cat VERSION)
if 'frontend' in services:
# Compile the frontend with web-source-compiler
wsc compile
# Copy the Monaco editor to a usable location
cp -r src/frontend/ui/node-modules/monaco-editor src/frontend/ui-dist/static/js/monaco-editor
if 'service-manager' in services:
# Copy over service-manager config
cp -n data/service-manager/config.json.bak data/service-manager/config.json || true
cp -n data/service-manager/secrets.json.bak data/service-manager/secrets.json || true
# Loop through and build all services
for service in services:
docker build -t {service}:{version} -f src/{service}/Dockerfile .
print('Done!')
def build_local(services: list) => None:
"""Build local binaries of various services
Args:
services (list): List of services to build
"""
clean()
mkdir dist
for service in services:
# Compile each service and place it in correct directory
cd src/{service}
env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -v -o {service}
mkdir ../../dist/{service}
mv {service} ../../dist/{service}/{service}
cp launch.sh ../../dist/{service}/launch-{service}.sh
chmod +x ../../dist/{service}/launch-{service}.sh
cd ../..
if 'service-manager' in services:
# Copy over service-manager specific items
cp -n data/service-manager/config.json.bak data/service-manager/config.json || true
cp -n data/service-manager/secrets.json.bak data/service-manager/secrets.json || true
cp -r data/service-manager dist/service-manager/data
if 'frontend' in services:
# Compile the frontend
wsc compile
# Copy over required files
cp -r src/frontend/ui-dist/templates dist/templates
cp -r src/frontend/ui-dist/static dist/static
cp -r src/frontend/ui/node-modules/monaco-editor dist/static/js/monaco-editor
if 'auth-manager' in services:
# Copy over auth-manager config
cp -r data/auth-manager dist/auth-manager/data
print('Done!')
all_services = [
'api-server',
'asset-manager',
'auth-manager',
'frontend',
'model-manager',
'service-manager'
]
# Determine which services we want to build
services = env.BUILD_SERVICES.split(',')
if 'all' in services:
services = all_services
if env.BUILD_MATURITY == 'dev':
build_local()
elif env.BUILD_MATURITY == 'release':
build_release()
You can now see how much easier file management is and build commands are much easier to run.
But how did we make this happen?
To explain that, we need to dive a bit into the code. Calligraphy has a CLi which allows it to be run similar to Python (i.e. calligraphy <script name>). In addition, for convenience, we added the ability to print our the source code being fed in with Bash/Python/combined lines printed in different colors so the inner detection can be visualized. We also allowed for printing the intermediate Python so it can be verified. Past that we start to get into the parser and transpiler.
For the parser to work, we need to remove line breaks inside the code. To do this, we search for brackets, braces, quotes, and parenthesis and check for any newlines within those groupings. Upon finding one, we replace it with <CALLIGRAPHY_NEWLINE> to reduce the code to one line while allowing us to break it back up to multiple later.
Once we’ve done this, we can now determine the language of each line. To do this, we first have to parse all imports and functions in the script. This allows us to understand that os.<whatever> is from a Python package and as such is Python code. We then go through each line and check for a few conditions:
If a comment is detected, it’s recorded as such, and if either of the other two conditions are met then it’s a Python line. If none of those match then it’s a Bash line. However there’s a bit more nuance. Calligraphy also allows for combined lines (e.g. pods = ?(kubectl get pods | tail -n +2 | awk '{{print $1}}').split('\n')[:-1]). To detect this, we look for ?(...) and $(...) patterns (The leading ? denotes “don’t print output to stdout”). Upon finding those we mark the line as combined and note the location of the Bash.
Now that we know what line is what, we can replace all Bash commands with a call to a calligraphy-internal function with preserves ENV state as well as running the commands and returning output. We then dump the now pure Python to a runner which executes the script.
This project ended up being super useful for our Artifactory management, and it has become the language of my make alternative, stud. It’s proven quite useful and I recommend trying it out for yourself. You can find the repo at (https://github.com/jfcarter2358/calligraphy)[https://github.com/jfcarter2358/calligraphy]. Let me know what you think!