Get Started Using the HPCC Systems Cloud Native Platform
- Lorraine Chapman
- Hugo Watanuki
- Christopher Lo
- Jim DeFabia
Find out more about our Cloud Native platform:
Resource | Description | Author |
|---|---|---|
This Helm chart can be used to deploy an entire HPCC environment to a K8s cluster. | HPCC Systems Platform Team | |
HPCC Local (host) storage | HPCC Systems Platform Team | |
Includes examples for a number of different areas such as Azure Storage, Cert Manager, Logging, Metrics, Secrets Storage Planes and more. | HPCC Systems Platform Team | |
If you are a PC user and want to run the HPCC Systems Platform, the simplest and most natural environment may be to use a Hyper-V Virtual Machine. Hyper-V is standard on many versions of Windows, and tends to work better than most add-on virtualization systems. It also has good support for multiple CPUs, and is quite reliable and performant. | Lili Xu | |
Learn about the most efficient ways to track and trend cloud usage costs, as well as how to set configuration parameters. | Shamser Ahmed | |
Optimising the Cloud Operation Cost of HPCC Systems using Azure Spot Instances | Processing data on the cloud using Virtual Machines can be expensive. Read about the results of some research carried out by Roshan Bhandari about how to save a considerable amount of money by running your VM using Azure Spot instances. | Roshan Bhandari |
In today’s world, security in our schools is critical for a healthy learning environment. The need for that security has inspired a High School student to design a facial recognition interface using HPCC Systems GNN bundle, to identify these potential threats by utilizing the camera built into the school’s security robot. | Carina Wang | |
Getting Started with the HPCC Systems Cloud Native Platform Using Terraform and Azure | Includes step-by-step instructions, showing how to deploy the HPCC Systems Cloud Native Platform into Azure using Terraform open source and additional modules from the LexisNexis Risk Solutions Terraform open-source registry. | Godson Fortil |
Configuring a HashiCorp Vault as a Certificate Manager on HPCC Systems | The importance of certificate management and instructions for setting up and configuring a Hashicorp Vault. | Nikita Jha |
The method of defining storage on our Cloud Native platform has been rationalized and simplified in HPCC Systems 8.2.0.  This blog takes you through the new way storage is defined in the values.yaml file, providing some examples of how it can be used. | Gavin Halliday | |
Walk through the steps required to setup a default cluster on Microsoft Azure | Jake Smith | |
How to persist your data and get your data on to your cluster in a cloud native environment. | Gavin Halliday | |
Configuring Storage in the Cloud Native HPCC Systems Platform | This blog provides more details about configuring storage on a cloud native HPCC Systems cluster. | Gavin Halliday |
This blog continues the theme of storing and persisting data, focusing on how to use storage planes. | Gavin Halliday | |
Exporting Data from an HPCC Systems Bare Metal platform to our Cloud Native Platform | A tutorial style blog providing the steps to take to export files to a Kubernetes system using azure blob storage. | Gavin Halliday |
Can we use a service mesh for internode communication within HPCC Systems? This blog provides a tutorial style guide to the answer to this question, focusing on installing a Service Mesh (specifically Istio) with HPCC Systems and making it work. | Manish Kumar Jaychand | |
In this tutorial style blog, Nandhini shares the steps needed to install HPCC Systems with Linkerd on Microsoft Azure. Find out more about Linkerd, which is a service mesh for Kubernetes. It makes running services easier and safer by providing runtime debugging, observability, reliability and security without requiring any changes to code. | Nandhini Velu | |
Log visualizations help identify, track and predict important events and trends on HPCC Systems clusters, by spotting interesting patterns and giving you visual clues which are easier to interpret than reading through the log file itself. Starting in HPCC Systems 8.0.0, a mechanism will be provided to deploy a lightweight Elastic Stack instance as a simple solution for the logs on-cloud processing challenges. In this blog, Rodrigo Pastrana walks through how to get started, view the logs, apply filter conditions and configure the Elastic Stack components. | Rodrigo Pastrana | |
This book focuses on containerized deployments. The first section is about using Docker containers and Helm charts locally. The second part uses the same techniques in the cloud. | Jim DeFabia | |
Interns Contributing to the HPCC Systems Cloud Native Platform | Some of our Class of 2020 interns have been working on cloud specific projects, while others have added new items to their project task list which focus on setting up and testing HPCC Systems using different Cloud platforms. This blog shares the details of the valuable contributions our interns are making to this ongoing development project. | Lorraine Chapman |
Yash Mishra is studying for a Masters in Computer Science at Clemson University. This summer, he joined the HPCC Systems intern program to work on a project that contributes to our ongoing cloud native platform development project. In this blog, Yash shares the results of his work focusing on cost considerations, storage and troubleshooting. | Yash Mishra | |
Our 'Going Cloud Native' blog series continues with this demonstration, showing how to setup and deploy an HPCC Systems Platform Cluster on the AWS Elastic Kubernetes Service (EKS). This tutorial style blog is a truly collaborative effort featuring contributions from Lucas Varella who is an intern studying at the University of Santa Catarina (UFSC) in Brazil, our LexisNexis Risk Solutions Group colleagues, Hugo Watanuki and Xiaoming Wang and Akash Gheewala a Solutions Architect from Amazon Web Services. | Lucas Varella Hugo Watanuki Xiaoming Wang Akash Gheewala | |
A series of short videos providing some quick tips and tricks for completing specific tasks. | Various HPCC Systems Developers |
2021 HPCC Systems Community Day Presentations about our Cloud Native Platform | Description | Presenter(s) |
|---|---|---|
Focuses on the approach and methods used in a custom Azure cloud deployment, based on our own company experience. This includes how to perform robust deployments of HPCC Systems environments on the cloud, with a focus on security, maintainability and supportability, and using the precepts of zero touch and zero trust. | Jon Burger | |
HPCC Systems Logging in the Cloud and an Elastic Stack Solution | As HPCC Systems continues its journey to the cloud, one major challenge faced is the ephemeral nature of log data and the accessibility of distributed application-level logs. This presentation discusses these challenges, the HPCC Systems logging architecture, and a simple Elastic Stack-based solution to the challenge. We'll demonstrate in detail the end-to- end solution, which includes Helm-based deployment, Kibana configuration, HPCC Systems log exploration, querying, and filtering. We'll also discuss an advanced topic that improves log data query performance by utilizing Elastic Search Ingest Pipelines. Finally, we'll touch on other possible solutions such as Azure Log Analytics. | Greg Panagiotatos & Rodrigo Pastrana |
A Simple HPCC Systems Cloud Deployment for Open-Source Users | How to setup and implement a basic HPCC Systems cluster in the cloud using Azure Kubernetes. We will walk through the deployment configuration leveraging Terraform, GitOps/Flux2 and storage settings. Note: This talk is intended for the wider open source HPCC Systems community. It is advised to check with your organization for any specific security protocols | Xiaoming Wang & Godson Fortil |
Includes a demo where you can learn how to start up and use a containerized single-node HPCC Systems Platform cluster on a local machine running Docker Desktop or Minikube to use for ECL Development, testing, or training. This provides a platform for hands-on experience with HPCC Systems to experiment with and even create real-world data analytics applications--all on your desktop or laptop PC. | Jim DeFabia & Christopher Lo | |
Terasort with HPCC Systems on Azure Kubernetes Service and High Performance Storage | Challenges, AKS considerations and storage options are discussed, including a demo covering the setup and configuration of HPCC Systems on AKS with Blob NFS 3.0 and performing a Terasort. | Shrikrishna Khose & Steve Griffith |
Design Considerations for Migrating Your HPCC Systems Data Lake to the Cloud | Lessons learned and design best practices through our own cloud migration experience are shared. The beginning of our presentation is a simple installation of our cluster on Azure using the community helm charts. During this demo we hit topics such as how the HPCC Systems platform differs between the Kubernetes cluster that we are deploying and the bare metal installations that community members are familiar with. Dive into helm for HPCC Systems, the value of .yaml files and a few different ways that the cluster can be configured and explain storage in the cloud compared to bare metal. Then learn about ROXIE and Thor usage in the cloud. Krishna covers some details about getting query lists, suspended queries, and doing package file deployments. Michael expands on basic security features that end users will want to enable in the cloud, including encryption in transit and at rest in a cloud environment such as Azure. | Krishna Turlapathi & Michael Gardner |
Covers a discussion of some current and future technology enhancements around HPCC Systems platform security, with a primary focus on cloud deployments. This session will include a look at support for mutual transport layer security between internal components within an HPCC Systems environment, external facing TLS for securing access into the HPCC Systems environment, using cert-manager to generate TLS certificates for HPCC Systems services and components, installing externally created TLS certificates, secrets management, and a look at our plans for future support of OAUTH2 based authentication and authorization, with an initial focus on support for OAUTH2 integration with Azure Active Directory services. | Russ Whitehead, Tony Fishbeck & Mark Kelly | |
With the advent of Cloud and Kubernetes over the years, machines are no longer considered as attached to a data center. Machines are more ephemeral than ever before. The traditional architecture of the HPCC Systems environment harnessed the physical storage of each node and that in turn gave certain performance benefits. But with cloud, it is no longer necessary to have a fixed machine for a process. The true power of cloud can be harnessed only when we treat them as ephemeral. Therefore, a Thor worker node which is always on in a traditional HPCC Systems environment is spun up only when it is required in a cloud environment. With the latest cloud native version of HPCC Systems, we now have the flexibility to spin up the clusters only when required. In this session, we will cover how the latest cloud native platform is different from the bare metal version, explain service mesh and how it fits into the HPCC Systems scheme of things, and a comparison of service mesh Istio and Linkerd. | Manish Kumar Jaychand | |
An update on the new features in the latest release including data handling in the cloud native version | Gavin Halliday | |
The Microsoft team will discuss the strong collaboration between Microsoft and LexisNexis Risk Solutions. See how HPCC Systems can be the foundation for a Data Lake and explore Azure services integration possibilities. Bonus: high level overview on Azure certification paths. | Shrikrishna Khose, Senior Cloud Solution Architect, and Steve Griffith, Principal Technical Specialist |
HPCC Systems Tech Talk Summer Series - Path to the Cloud Interviews | Who's in the room |
|---|---|
An interview with Richard Chapman - Part 1 | Flavio Villanustre and Richard Chapman |
An interview with Jake Smith | Flavio Villanustre and Jake Smith |
An interview with Gavin Halliday | Flavio Villanustre and Gavin Halliday |
An interview with Ming Wang | Flavio Villanustre and Ming Wang |
An Interview with Shamser Ahmed Shamser talks about Cloud Provisioning and the latest future development work planned for the Workunit Analysis Tool. | Flavio Villanustre and Shamser Ahmed |
An Interview with Godji Fortil Godji talks about the different types of testing taking place on our cloud native platform, including configuration, functional and performance testing. | Flavio Villanustre and Godson Fortil |
An Interview with Anthony Fishbeck Tony talks about streamlining the configuration and deployment of ESP for cloud deployment. | Flavio Villanustre and Anthony Fishbeck |
An Interview with Lorraine Chapman Lorraine talks about the contribution our Class of 2020 interns are making to our Cloud Native development project and introduces the HPCC Systems Roadmap. | Flavio Villanustre and Lorraine Chapman |
An Interview with Richard Chapman - Part 2 Richard completes this series with an update providing the latest information about our new Cloud Native platform. | Flavio Villanustre and Richard Chapman |
All pages in this wiki are subject to our site usage guidelines.