Page Comparison

Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community. 

Find out about the HPCC Systems Summer Internship Program.

The project proposal application period for 2020 summer internships is now closed. Check back in the Fall for details about applying to join our 2021 program.This project was completed by a student accepted on to the 2021 HPCC Systems Intern Program

Project Description

An increasing number of companies are moving to cloud environments. However, taking full advantage of the cloud infrastructure, particularly to get the job done while minimizing cost, remains a challenge. This project will try to explore how to use AWS EC2 Spot Instances  for some of our explore setup Kubernetes cluster, CI/CD build and test with spot instance on Azure and AWS.

For Kubernetes setup we currently use Azure Kubernetes Service (AKS) on Azure and Elastic Kubernetes Service (EKS) on AWS. Both should support spot instances but need some special treatment. This has high priority than following CI/CD tasks. We hope there is measurement (AZ/AWS client program) can estimate possibility if we can get the instances. Also it will be nice to either query AZ/AWS or write the function to get trend of spot instances.  

For CI/CD build: Current HPCC Systems development works, such as nightly builds, some function tests, etc.  are done through Jenkins with AWS instance. Spot Instances are much cheaper than their on-demand counterparts.  However, most spot instances will not have the pre-we are not sure pre-build AWS AMI  and Azure VHD which have pre-requisites suitable for HPCC Systems Development , such as the compute powerwill be able to adopted to spot instances. Thus, we may need to provision the spot instance before it can be used for our development purposes. As another option, we can create a Docker image with all of the prerequisite packages. 

We This project requires knowledge of AWS (EC2,  Client API, S3, etc),  of  AZ/AWS  CLI, AKS and EKS, Docker, Unix Shell, Jenkins, maybe Python and also HPCC Systems knowledge.  The students may not have all these skills but should be eager to learn the required tools or technologies.If you are interested in this project, please contact Gordon Smith. 

Completion of this project involves:

...

• Deep understand how spot instance work in Azure and AWS

• Create AZ/AWS CLI scripts to estimate possibility, monitor trend and calculate money saved.

• create AZ/AWS CLI scripts to setup Kubernetes cluster. Do Azure first.

• Enable spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 20.04 or CentOS 7. It will be on AWS.

• Use Jenkins and AWS Spot Instances  for some of HPCC Systems Development work, at least nightly build

• Documentation

• Explore the potential spot instance usage for HPCC Systems related jobs

By the mid term review we would expect you to have:

• <What must be completed to pass the evaluation and continue on to complete the project>

...

Xiaoming Wang
Contact Details

Backup Mentor: Godson Fortil
Contact Details

• Enabled spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 18.04 or CentOS 7. It will be on AWS.