Get Started Using the HPCC Systems Cloud Native Platform
Find out more about our Cloud Native platform:
Resource | Description | Author |
---|---|---|
This Helm chart can be used to deploy an entire HPCC environment to a K8s cluster. | HPCC Systems Platform Team | |
HPCC Local (host) storage | HPCC Systems Platform Team | |
Includes examples for a number of different areas such as Azure Storage, Cert Manager, Logging, Metrics, Secrets Storage Planes and more. | HPCC Systems Platform Team | |
If you are a PC user and want to run the HPCC Systems Platform, the simplest and most natural environment may be to use a Hyper-V Virtual Machine. Hyper-V is standard on many versions of Windows, and tends to work better than most add-on virtualization systems. It also has good support for multiple CPUs, and is quite reliable and performant. | Lili Xu | |
Learn about the most efficient ways to track and trend cloud usage costs, as well as how to set configuration parameters. | Shamser Ahmed | |
Optimising the Cloud Operation Cost of HPCC Systems using Azure Spot Instances | Processing data on the cloud using Virtual Machines can be expensive. Read about the results of some research carried out by Roshan Bhandari about how to save a considerable amount of money by running your VM using Azure Spot instances. | Roshan Bhandari |
In today’s world, security in our schools is critical for a healthy learning environment. The need for that security has inspired a High School student to design a facial recognition interface using HPCC Systems GNN bundle, to identify these potential threats by utilizing the camera built into the school’s security robot. | Carina Wang | |
Getting Started with the HPCC Systems Cloud Native Platform Using Terraform and Azure | Includes step-by-step instructions, showing how to deploy the HPCC Systems Cloud Native Platform into Azure using Terraform open source and additional modules from the LexisNexis Risk Solutions Terraform open-source registry. | Godson Fortil |
Configuring a HashiCorp Vault as a Certificate Manager on HPCC Systems | The importance of certificate management and instructions for setting up and configuring a Hashicorp Vault. | Nikita Jha |
The method of defining storage on our Cloud Native platform has been rationalized and simplified in HPCC Systems 8.2.0. This blog takes you through the new way storage is defined in the values.yaml file, providing some examples of how it can be used. | Gavin Halliday | |
Walk through the steps required to setup a default cluster on Microsoft Azure | Jake Smith | |
How to persist your data and get your data on to your cluster in a cloud native environment. | Gavin Halliday | |
Configuring Storage in the Cloud Native HPCC Systems Platform | This blog provides more details about configuring storage on a cloud native HPCC Systems cluster. | Gavin Halliday |
This blog continues the theme of storing and persisting data, focusing on how to use storage planes. | Gavin Halliday | |
Exporting Data from an HPCC Systems Bare Metal platform to our Cloud Native Platform | A tutorial style blog providing the steps to take to export files to a Kubernetes system using azure blob storage. | Gavin Halliday |
Can we use a service mesh for internode communication within HPCC Systems? This blog provides a tutorial style guide to the answer to this question, focusing on installing a Service Mesh (specifically Istio) with HPCC Systems and making it work. | Manish Kumar Jaychand | |
In this tutorial style blog, Nandhini shares the steps needed to install HPCC Systems with Linkerd on Microsoft Azure. Find out more about Linkerd, which is a service mesh for Kubernetes. It makes running services easier and safer by providing runtime debugging, observability, reliability and security without requiring any changes to code. | Nandhini Velu | |
Log visualizations help identify, track and predict important events and trends on HPCC Systems clusters, by spotting interesting patterns and giving you visual clues which are easier to interpret than reading through the log file itself. Starting in HPCC Systems 8.0.0, a mechanism will be provided to deploy a lightweight Elastic Stack instance as a simple solution for the logs on-cloud processing challenges. In this blog, Rodrigo Pastrana walks through how to get started, view the logs, apply filter conditions and configure the Elastic Stack components. | Rodrigo Pastrana | |
This book focuses on containerized deployments. The first section is about using Docker containers and Helm charts locally. The second part uses the same techniques in the cloud. | Jim DeFabia | |
Interns Contributing to the HPCC Systems Cloud Native Platform | Some of our Class of 2020 interns have been working on cloud specific projects, while others have added new items to their project task list which focus on setting up and testing HPCC Systems using different Cloud platforms. This blog shares the details of the valuable contributions our interns are making to this ongoing development project. | Lorraine Chapman |
Yash Mishra is studying for a Masters in Computer Science at Clemson University. This summer, he joined the HPCC Systems intern program to work on a project that contributes to our ongoing cloud native platform development project. In this blog, Yash shares the results of his work focusing on cost considerations, storage and troubleshooting. | Yash Mishra | |
Our 'Going Cloud Native' blog series continues with this demonstration, showing how to setup and deploy an HPCC Systems Platform Cluster on the AWS Elastic Kubernetes Service (EKS). This tutorial style blog is a truly collaborative effort featuring contributions from Lucas Varella who is an intern studying at the University of Santa Catarina (UFSC) in Brazil, our LexisNexis Risk Solutions Group colleagues, Hugo Watanuki and Xiaoming Wang and Akash Gheewala a Solutions Architect from Amazon Web Services. | Lucas Varella Hugo Watanuki Xiaoming Wang Akash Gheewala | |
A series of short videos providing some quick tips and tricks for completing specific tasks. | Various HPCC Systems Developers |
2021 HPCC Systems Community Day Presentations about our Cloud Native Platform | Description | Presenter(s) |
---|---|---|
Focuses on the approach and methods used in a custom Azure cloud deployment, based on our own company experience. This includes how to perform robust deployments of HPCC Systems environments on the cloud, with a focus on security, maintainability and supportability, and using the precepts of zero touch and zero trust. | Jon Burger | |
HPCC Systems Logging in the Cloud and an Elastic Stack Solution | As HPCC Systems continues its journey to the cloud, one major challenge faced is the ephemeral nature of log data and the accessibility of distributed application-level logs. This presentation discusses these challenges, the HPCC Systems logging architecture, and a simple Elastic Stack-based solution to the challenge. We'll demonstrate in detail the end-to- end solution, which includes Helm-based deployment, Kibana configuration, HPCC Systems log exploration, querying, and filtering. We'll also discuss an advanced topic that improves log data query performance by utilizing Elastic Search Ingest Pipelines. Finally, we'll touch on other possible solutions such as Azure Log Analytics. | Greg Panagiotatos & Rodrigo Pastrana |
A Simple HPCC Systems Cloud Deployment for Open-Source Users | How to setup and implement a basic HPCC Systems cluster in the cloud using Azure Kubernetes. We will walk through the deployment configuration leveraging Terraform, GitOps/Flux2 and storage settings. Note: This talk is intended for the wider open source HPCC Systems community. It is advised to check with your organization for any specific security protocols | Xiaoming Wang & Godson Fortil |
Includes a demo where you can learn how to start up and use a containerized single-node HPCC Systems Platform cluster on a local machine running Docker Desktop or Minikube to use for ECL Development, testing, or training. This provides a platform for hands-on experience with HPCC Systems to experiment with and even create real-world data analytics applications--all on your desktop or laptop PC. | Jim DeFabia & Christopher Lo | |
Terasort with HPCC Systems on Azure Kubernetes Service and High Performance Storage | Challenges, AKS considerations and storage options are discussed, including a demo covering the setup and configuration of HPCC Systems on AKS with Blob NFS 3.0 and performing a Terasort. | Shrikrishna Khose & Steve Griffith |
Design Considerations for Migrating Your HPCC Systems Data Lake to the Cloud | Lessons learned and design best practices through our own cloud migration experience are shared. The beginning of our presentation is a simple installation of our cluster on Azure using the community helm charts. During this demo we hit topics such as how the HPCC Systems platform differs between the Kubernetes cluster that we are deploying and the bare metal installations that community members are familiar with. Dive into helm for HPCC Systems, the value of .yaml files and a few different ways that the cluster can be configured and explain storage in the cloud compared to bare metal. Then learn about ROXIE and Thor usage in the cloud. Krishna covers some details about getting query lists, suspended queries, and doing package file deployments. Michael expands on basic security features that end users will want to enable in the cloud, including encryption in transit and at rest in a cloud environment such as Azure. | Krishna Turlapathi & Michael Gardner |
Covers a discussion of some current and future technology enhancements around HPCC Systems platform security, with a primary focus on cloud deployments. This session will include a look at support for mutual transport layer security between internal components within an HPCC Systems environment, external facing TLS for securing access into the HPCC Systems environment, using cert-manager to generate TLS certificates for HPCC Systems services and components, installing externally created TLS certificates, secrets management, and a look at our plans for future support of OAUTH2 based authentication and authorization, with an initial focus on support for OAUTH2 integration with Azure Active Directory services. | Russ Whitehead, Tony Fishbeck & Mark Kelly | |