Migrate regression testing for ECL bundles to GitHub Actions

This project is available as a student work experience opportunity with HPCC Systems. Curious about other projects we are offering? Take a look at our Ideas List.

Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community. 

Find out about the HPCC Systems Summer Internship Program.

Project Description

The HPCC Systems Platform team has an automated test system called OBT – Overnight Build and Test systems (see repo listed in the table below) that is currently utilized to perform regression testing on ECL bundles, such as GNN and Machine Learning bundles, whenever a change to the HPCC Systems source code is made. However, more modern DevOps approaches suggest performing regression testing as part of automated workflows such as GitHub actions. GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production. Therefore, the first goal of this project is to migrate the regression testing for ECL bundles from scripts contained within the OBT to GitHub Actions.

Also, the HPCC Systems user documentation and the HPCC Systems GitHub README files usually contain hyperlinks to the HPCC Systems website and other external sources that are constantly changing. This can result in the famous error 404 (i.e., page not found). While manual testing by a human can still be employed to verify and update the hyperlinks, this is not the most reliable way. Therefore, the second goal of this project is to develop an automated testing of hyperlinks in HPCC Systems user documents and GitHub README files using GitHub Actions.

To accomplish this, the following requirements must be considered:

• The solution should be implemented using GitHub Actions as the CI/CD framework.
• For the first goal, the solution should be capable of testing all the ML and supported bundles (see ECL bundle repository in the resources area below).
• For the second goal, a script will need to be developed to parse these user documentation and README source files and extract all the hyperlinks to be tested (see documentation repositories in the resource area below - source files are available in either XML format or markdown).

Completion of this project involves:

There are two kinds of deliverables:

• Code, provided in GitHub Pull Requests from cloned repositories. (The original codes are executed regularly to support our development work, so they are untouchable directly by this project.)
• Documentation