Versions Compared

Version Old Version 19 New Version Current
Changes made by

Xiaoming Wang

Hugo Watanuki

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The proposal application period for 2021 internships is now closed. The proposal period for 2022 internships will open in the Fall.

...

This project was completed by a student accepted on to the 2021 HPCC Systems Intern

...

Project suggestions must be relevant to HPCC Systems and of benefit to our open source community. 

Find out about the HPCC Systems Summer Internship Program.

Project Description

An increasing number of companies are moving to cloud environments. However, taking full advantage of the cloud infrastructure, particularly to get the job done while minimizing cost, remains a challenge. This project will try to explore setup Kubernetes cluster, CI/CD build and test with spot instance on Azure and AWS.

...

We This project requires knowledge of  AZ/AWS  CLI, AKS and EKS, Docker, Unix Shell, Jenkins, maybe Python and also HPCC Systems knowledge.  The students may not have all these skills but should be eager to learn the required tools or technologies.If you are interested in this project, please contact Gordon Smith. 

Completion of this project involves:

  • Deep understand how spot instance work in Azure and AWS

  • Create AZ/AWS CLI scripts to estimate possibility, monitor trend and calculate money saved.

  • create AZ/AWS CLI scripts to setup Kubernetes cluster. Do Azure first.

  • Enable spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 20.04 or CentOS 7. It will be on AWS.

  • Use Jenkins and AWS Spot Instances  for some of HPCC Systems Development work, at least nightly build

  • Documentation

  • Explore the potential spot instance usage for HPCC Systems related jobs

By the mid term review we would expect you to have:

  • Enabled spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 18.04 or CentOS 7. It will be on AWS.

Mentor

Godson Fortil
Godson.Fortil@lexisnexisrisk.com

Backup Mentor: Xiaoming Wang Xiaoming.Wang@lexisnexis.com

Skills needed

  • General Cloud Environment knowledge, particularly AKS and EKS

  • Azure/AWS  Client API (shell or Python), S3, Docker, Jenkins, Packer

  • Unix Shell, Python

  • Ability to build and test the HPCC system (guidance will be provided).

  • Ability to write test code. Knowledge of ECL is not a requirement since it should be possible to re-use existing code with minimal changes for this purpose. Links are provided below to our ECL training documentation and online courses should you wish to become familiar with the ECL  language.

Deliverables

Midterm

  • A script to estimate the  possibility of getting desired spot instance.

  • Basic scripts to create AKS cluster and apply to HPCC Cloud testing

  • Enable spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 18.04 or CentOS 7. I will be on AWS.

End of project

  • Complete scripts for spot instance estimating possibility, monitoring trend and calculating money saved.

  • Complete scripts for AKS and EKS creation.

  • Use Jenkins and AWS Spot Instances  for some of HPCC Systems Development work, at least nightly build

  • Documentation

  • A github project to host code and documentation

  • Explore the potential spot instance usage for HPCC Systems related jobs

Other resources