Project Description
Provide helm chart examples CSI driver support on various cloud providers and storage types, for example Azure Azurefile, AWS EFS and FSx Lustre, etc.
HPCC Helm char currently has basic EFS CSI driver support but AWS update the configuration and documentation.
Azurefile with CSI driver: https://docs.microsoft.com/en-us/azure/aks/azure-files-csi. Current HPCC Helm Chart example for Azure File (without CSI driver): https://github.com/hpcc-systems/helm-chart/tree/master/helm/examples/azure
AWS EFS with CSI driver: https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html. Current HPCC Helm Chart example for AWS EFS (without CSI driver): https://github.com/hpcc-systems/helm-chart/tree/master/helm/examples/efs
AWS FSx Lustre with CSI driver: https://docs.aws.amazon.com/eks/latest/userguide/fsx-csi.html. It is optional
Student can based on current Azure File and EFS examples and adapt them to CSI driver.
Update: Azure File CSI support is already done so student will focus on AWS EFS and FSx. There are possible three variations of each type:
1) Automatically creating CSI PV/PVC when HPCC Systems started. The life cycle will HPCC Systems cluster which means when HPCC Systems destroyed the PV/PVC will be automatically deleted.
2) Start CSI PV/PVC first with Helm Chart before starting HPCC Systems. The PV/PVC will be still live after HPCC Systems cluster destroyed. But delete Kubernetes cluster will destroy the PV/PVC. So the life cycle will be Kubernetes cluster.
3) Start CSI PV/PVC first with Helm Chart before starting HPCC Systems. But the PV/PVC can be persistent even the Kubernetes cluster destroyed. The life cycle is beyond Kubernetes cluster. The CSI PV/PVC can be reused by following Kubernetes and HPCC System clusters. It is very useful for data processed once but will used by various following applications
HPCC-Platform/helm/examples/azure at master · hpcc-systems/HPCC-Platform (github.com) implemented all 3 cases for Azure File
HPCC-Platform/helm/examples/efs at master · hpcc-systems/HPCC-Platform (github.com) implementede 2)
Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community.
This project is mainly for community user instead of HPCC Systems internal users.
Azure and AWS account will be provided for the student of project.
There are probably other similar storage solutions in public clouds. We already have one for EFS/CSI. More information coming soon.
If you are interested in this project, please contact our intern program mailbox.
Completion of this project involves:
- Create Azure File with CSI and AWS EFS with CSI examples in https://github.com/hpcc-systems/helm-chart including documentation (README.md)
By the mid term review we would expect you to have:
- Understand CSI driver and finish basic Azure File with CSI implementation for HPCC Systems Cluster Deployment
| Mentor | Xiaoming Wang Backup Mentor: Godson Fortil |
Skills needed |
|
Deliverables | Midterm End of project |
Other resources |
|