/
HPCC Systems Terraform CI with GitHub Actions

HPCC Systems Terraform CI with GitHub Actions

Thisย project wasย completed by a student accepted on to the 2024 HPCC Systems Intern Program.

Project Description

In order for the HPCC Systems project to effectively support Continuous Integration (CI) and Continuous Delivery (CD) pipelines with GitHub Actions, the capability of automatically deploying an Azure Kubernetes Service (AKS) cluster with HPCC Systems using the GitHub Actions platform is required. Therefore, the main objective of this project idea is to design and implement a GitHub Actions workflow that implements this capability.

An initial implementation of this project idea is already available in GitHub (see the repository in the "important resources" section below) and leverages Terraform modules to automate the deployment of the AKS cluster with HPCC Systems. This initial implementation now needs to be ported and tested using GitHub Actions. A GitHub Actions workflow with such capability could be leveraged in several areas under the HPCC Systems project such as smoke testing, performance testing, placements testing and various other deployment, configuration, and regression testing.

To accomplish this, the following requirements must be considered:

  • The deployment of the AKS cluster with HPCC Systems should use the Terraform modules available in this repository: GitHub - hpcc-systems/terraform-azurerm-hpcc

  • Azure blob storage will be needed to host various data files, such as the Terraform state files, and output data files from each of the Terraform modules. The output data files are in JSON format and provide data to the storage, logging,
    AKS, and HPCC modules.

  • Beyond the deployment of the AKS cluster with HPCC Systems, the workflow should also support automated tests, such as ECL tasks to verify the HPCC Systems build. The workflow developed should be optimized to minimize costs in Azure.

  • The implementations may include following items (2,3,4 are only required for experienced students such as senior or master/PhD student):

  1. Write a single action to include everything: azure login such as using Azure Service Principal, deploy AKS, test ECL,
    destroy AKS

  2. Write action for azure login and publish to GitHub Marketplace (optional)

  3. Divide 1) to 3 actions: (optional)
    a. Deploy AKS and make it to stop state
    b. Start existing AKS and perform ECL Tests and stop AKS
    c. Destroy AKS and related resources

  4. Add htpasswd security functionality for exposed HPCC services. (optional)

Completion of this project involves:

  • A complete functional GitHub Actions workflow for HPCC Systems build CI/CD integration.
    A complete GitHub project with code and documentation.

  • A blog, a recorded presentation, and a poster artifact about your project (see examples from previous yearsย here).

  • Wishlist: Extend the GitHub Actions workflow to support additional HPCC Systems testing, such as deployment,
    configuration and regression testing.

By the mid term review, we would expect you to have:

  • Learned Azure Kubernetes Service (AKS) and Azure Service Principal

  • Learned Azure Blob Storage

  • Learned HPCC Systems Terraform modules for AKS

  • Learned GitHub Actions Workflow

  • An initial GitHub Actions workflow supporting the deployment of an AKS cluster and basic ECL testing.

Mentor

Godji Fortil Godji.Fortil@lexisnexisrisk.com

Backup Mentor:ย 
Xiaoming Wang Xiaoming.Wang@lexisnexisrisk.com

Skills needed
  • General knowledge of Linux and shell (bash)

  • Self-motivated to learn new technologies such as HPCC Systems, ECL, Git, GitHub Actions, Kubernetes, Azure, Terraform

  • Resilience to perform several rounds of troubleshooting and testing.

  • Technical writing skills for developing documentation.

Deliverables

Midterm

  • A GitHub project with a workflow that successfully deploys an AKS cluster using the HPCC
    Systems Terraform modules.

End of project

  • A complete GitHub project with fully working workflow including ECL testing and documentation.

  • A blog, a recorded presentation, and a poster artifact about your project.

Important resources



Related content

Develop an automated ECL Watch Test Suite
Develop an automated ECL Watch Test Suite
Read with this
General GitOps for Kubernetes deployment and management
General GitOps for Kubernetes deployment and management
More like this
Investigate Frameworks and Best Practices for HPCC Systems Cloud Native
Investigate Frameworks and Best Practices for HPCC Systems Cloud Native
More like this
Gagana Premnath - 2024 Poster Contest Resources
Gagana Premnath - 2024 Poster Contest Resources
More like this
Explore using Spot Instances on Azure and AWS
Explore using Spot Instances on Azure and AWS
More like this
How To... Videos - HPCC Systems Cloud Native Platform
How To... Videos - HPCC Systems Cloud Native Platform
More like this

All pages in this wiki are subject to our site usage guidelines.