Browse Poster Wiki: 2024 Poster Award Winners, Awards Ceremony (Watch Recording), Posters by 2024 HPCC Systems Interns, Posters by Academic Partners, Poster Judges, /wiki/spaces/hpcc/pages/86966553, 2024 Poster Contest Home Page, Poster Contest Previous Years

Hi, I am Gagana Premnath, a computer science graduate from Syracuse University, specialized in Application development, DevOps, and cloud computing. I have over 5 years of full time experience as a Full Stack Developer, having worked on diverse projects ranging from e-commerce platforms to enterprise-level applications with tech stacks in Java, Python, JavaScript and its frameworks (React, Next, Angular), Azure, AWS, NodeJS and its frameworks, APIs development and integrations. My experience primarily lies in JavaScript and Java programming. During my studies, I completed an internship where I worked extensively with proprietary languages, gaining valuable experience in specialized software development environments. Currently, I am looking for full-time job opportunities in the software development domain, particularly in roles focused on Cloud and Backend (Java/Python). Alongside this, I volunteer as a mentor with GHC, helping students in their job search. Outside of work, I'm passionate about traveling, painting, and contributing to tech communities like GHC through mentorship and blogging. I'm always eager to connect with like-minded professionals and explore new opportunities to innovate and grow.

Poster Abstract

In modern software development, continuous integration (CI) and continuous deployment (CD) have become essential practices to ensure efficient, reliable, and repeatable deployment processes. Terraform, an open-source infrastructure as code (IaC) tool by HashiCorp, has gained widespread adoption for its ability to provision and manage infrastructure across various cloud platforms. HPCC Systems, known for its high-performance computing cluster (HPCC) technology, leverages Terraform for infrastructure management, enabling the definition and deployment of HPCC clusters programmatically.

This abstract outlines the integration of HPCC Systems Terraform CI with GitHub Actions, a popular CI/CD platform provided by GitHub. The integration aims to streamline the deployment process of HPCC clusters, ensuring consistency, reducing manual intervention, and improving deployment efficiency.

Introduction

HPCC Systems, developed by LexisNexis Risk Solutions, provides a comprehensive data lake management platform with robust tools for big data processing and analytics. Terraform, with its declarative configuration language, facilitates the automation of infrastructure provisioning, making it an ideal tool for managing HPCC cluster deployments. HPCC has four Terraform modules to be deployed: vnet, storage, aks, and HPCC. Each of these modules is deployed sequentially due to dependencies. GitHub Actions offers seamless automation of software workflows directly within GitHub repositories, enabling developers to define custom workflows for building, testing, and deploying code.

Methodology

The integration involves the following key steps:

Terraform Configuration: Four Terraform modules—vnet, storage, aks, and hpcc—are used for creating the infrastructure.

GitHub Actions Workflow Definition: Create a GitHub Actions workflow file (e.g., main.yml) in the repository. Define the jobs required for installing packages like Terraform, AzureCLI, and kubelogin, logging into Azure, and deploying each module using action files. This file specifies the CI/CD pipeline, including steps to execute Terraform commands (e.g., terraform init, terraform plan, terraform apply) to deploy the HPCC clusters.

Application Registration: Create and register an application with Microsoft Entra and use the credentials within GitHub Actions to access AzureCLI via the command line.

Data Persistence: Create storage accounts to maintain data files between each module deployment, ensuring that the accounts persist between jobs.

State Management: Create storage accounts for maintaining Terraform state files.

Secrets Management: Securely store sensitive information, such as cloud provider credentials, using GitHub Secrets. These secrets are referenced within the GitHub Actions workflow to authenticate Terraform operations and log into AzureCLI.

Triggering Workflows: Configure the workflow to trigger on specific events, such as pushes to the repository, pull requests, or manual triggers. This ensures that any changes to the Terraform configurations automatically initiate the CI/CD pipeline, leading to the deployment or update of HPCC clusters.

Regression testing of ECL test suit after each deployment to check for the validation of deployments.

Results and Conclusion:

Integrating HPCC Systems Terraform CI with GitHub Actions results in a robust and automated deployment process. The key benefits include:

Automation: HPCC clusters are deployed automatically based on the Terraform configurations, reducing manual intervention and deployment time.

Consistency: The use of version-controlled Terraform scripts ensures consistent infrastructure deployment across different environments.

Efficiency: GitHub Actions workflows streamline the deployment process, enabling quick and reliable infrastructure provisioning.

Collaboration: Storing Terraform configurations in a GitHub repository fosters collaborative development and peer reviews, improving code quality.

Presentation

In this Video Recording, Gagana provides a tour and explanation of her poster content.

Integration of HPCC Systems Terraform CI with GitHub Actions

Click on the poster for a larger image.