HPCC Systems Terraform CI with GitHub Actions

This project was completed by a student accepted on to the 2024 HPCC Systems Intern Program.

Project Description

In order for the HPCC Systems project to effectively support Continuous Integration (CI) and Continuous Delivery (CD) pipelines with GitHub Actions, the capability of automatically deploying an Azure Kubernetes Service (AKS) cluster with HPCC Systems using the GitHub Actions platform is required. Therefore, the main objective of this project idea is to design and implement a GitHub Actions workflow that implements this capability.

An initial implementation of this project idea is already available in GitHub (see the repository in the "important resources" section below) and leverages Terraform modules to automate the deployment of the AKS cluster with HPCC Systems. This initial implementation now needs to be ported and tested using GitHub Actions. A GitHub Actions workflow with such capability could be leveraged in several areas under the HPCC Systems project such as smoke testing, performance testing, placements testing and various other deployment, configuration, and regression testing.

To accomplish this, the following requirements must be considered:

• The deployment of the AKS cluster with HPCC Systems should use the Terraform modules available in this repository: GitHub - hpcc-systems/terraform-azurerm-hpcc

• Azure blob storage will be needed to host various data files, such as the Terraform state files, and output data files from each of the Terraform modules. The output data files are in JSON format and provide data to the storage, logging,
  AKS, and HPCC modules.

• Beyond the deployment of the AKS cluster with HPCC Systems, the workflow should also support automated tests, such as ECL tasks to verify the HPCC Systems build. The workflow developed should be optimized to minimize costs in Azure.

• The implementations may include following items (2,3,4 are only required for experienced students such as senior or master/PhD student):

1. Write a single action to include everything: azure login such as using Azure Service Principal, deploy AKS, test ECL,
   destroy AKS

2. Write action for azure login and publish to GitHub Marketplace (optional)

3. Divide 1) to 3 actions: (optional)
   a. Deploy AKS and make it to stop state
   b. Start existing AKS and perform ECL Tests and stop AKS
   c. Destroy AKS and related resources

4. Add htpasswd security functionality for exposed HPCC services. (optional)

Completion of this project involves:

• A complete functional GitHub Actions workflow for HPCC Systems build CI/CD integration.
  A complete GitHub project with code and documentation.

• A blog, a recorded presentation, and a poster artifact about your project (see examples from previous years here).

• Wishlist: Extend the GitHub Actions workflow to support additional HPCC Systems testing, such as deployment,
  configuration and regression testing.

By the mid term review, we would expect you to have:

• Learned Azure Kubernetes Service (AKS) and Azure Service Principal

• Learned Azure Blob Storage

• Learned HPCC Systems Terraform modules for AKS

• Learned GitHub Actions Workflow

• An initial GitHub Actions workflow supporting the deployment of an AKS cluster and basic ECL testing.