Photo by Jan Baborák on Unsplash
Engineering project
Automation and high-performance computing for astronomy
Automate the build and deployment of C# astronomical-analysis applications to a high-performance computing cluster using GitLab, Docker and Kubernetes.
The project
In a European Space Agency project, C# applications processed astronomical observations through two separate analyses. One concerned redshift and velocity estimation. The other used image subtraction to search for new celestial objects. Comparing an observation with a reference image highlighted differences for examination as candidate sources. My contribution covered GitLab CI/CD pipelines and application deployment to a Kubernetes high-performance computing (HPC) cluster.
Engineering approach
Docker images package the C# code and its dependencies into an environment that runs on the compute nodes. GitLab automates image creation and deployment, while Kubernetes runs the containers. Application updates therefore pass through the CI/CD pipeline before they are used to process astronomical observations.
Work delivered
An operational CI/CD workflow for deploying astronomical-analysis applications to HPC resources, with project scripts maintained in GitHub.
My contribution
- Configuration of GitLab CI/CD pipelines to build the C# applications and package them as Docker images
- Automation of container deployment to the Kubernetes cluster used for scientific computation
- Version control of project scripts in GitHub
Functional overview
GitLab automates Docker image builds and C# application deployment to Kubernetes. The HPC cluster runs two independent scientific analyses. One studies redshift and velocities. The other uses image subtraction to identify candidate sources in the search for new celestial objects.
C#
Docker
GitHub
GitLab
Kubernetes
Components and flows
- Scripts versioned in the GitHub repo; CI/CD pipelines in GitLab
- Build of Docker images containing the C# astronomical-analysis applications
- Deploy to the Kubernetes cluster and execute HPC workloads
- Redshift analysis and velocity estimation in one scientific workstream
- Subtraction of a reference image from an observation, followed by examination of candidate sources in a separate scientific workstream
Discuss your project
A discussion of your project’s design, development and production deployment.