Versions Compared

Version Old Version 18 New Version 19
Changes made by

Lorraine Chapman

Xiaoming Wang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Enabled spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 18.04 or CentOS 7. It will be on AWS.
Mentor

Xiaoming Wang
Contact DetailsGodson Fortil
Godson.Fortil@lexisnexisrisk.com

Backup Mentor: Godson Fortil
Contact Details Xiaoming Wang Xiaoming.Wang@lexisnexis.com


Skills needed
  • General Cloud Environment knowledge, particularly AKS and EKS
  • Azure/AWS  Client API (shell or Python), S3, Docker, Jenkins, Packer
  • Unix Shell, Python
  • Ability to build and test the HPCC system (guidance will be provided).
  • Ability to write test code. Knowledge of ECL is not a requirement since it should be possible to re-use existing code with minimal changes for this purpose. Links are provided below to our ECL training documentation and online courses should you wish to become familiar with the ECL  language.
Deliverables

Midterm

  • A script to estimate the  possibility of getting desired spot instance.
  • Basic scripts to create AKS cluster and apply to HPCC Cloud testing
  • Enable spot instances for  our nightly build on  at least one supported Linux distro, such as Ubuntu 18.04 or CentOS 7. I will be on AWS.

End of project

  • Complete scripts for spot instance estimating possibility, monitoring trend and calculating money saved.
  • Complete scripts for AKS and EKS creation.
  • Use Jenkins and AWS Spot Instances  for some of HPCC Systems Development work, at least nightly build
  • Documentation
  • A github project to host code and documentation
  • Explore the potential spot instance usage for HPCC Systems related jobs
Other resources