HPCC Systems Community Day 2024

Watch recordings of all presentations made during our Community Day Summit held during October 2024. Find out more about this event and read our blog review of the event.

In addition to these presentations, we also held a Technical Poster Competition for students to present work they have completed on projects leveraging HPCC Systems. See our poster contest wiki to find out more about our 2024 Poster Contest participants and winners.

Resources

Presentation Tracks and Content

Plenary Sessions

Join us for the opening and closing sessions from the 11th annual HPCC Systems Community Virtual Summit.

Welcome & Plenary Keynotes
Gavin Halliday, SVP and Head of Platform Engineering, LexisNexis® Risk Solutions/ Michael Stefanick, Managing Director, EY
Watch as Gavin Halliday kicked off the 11th annual HPCC Systems Community Virtual Summit, as well as hear from our community keynote speaker Michael Stefanick, who presented, AI + Human: How AI will Disrupt Work and Allow Workers to Thrive.

Awards Ceremony & Plenary
Trish McCall, Sr Director Program Management, Hugo Watanuki, Manager Community Tech Programs, George S Foreman, Business Analyst II, Bob Foreman, Software Engineering Lead LexisNexis Risk Solutions
Join us for the long-awaited announcement of the 2024 Community Awards and Poster Competition winners, followed by our keynote honoring our academic community and engagement.

Platform Evolution

Take a look at the latest improvements and innovative features in the platform.

The Latest Advancements in HPCC Systems Observability
Rodrigo Pastrana & Mark Kelly, LexisNexis Risk Solutions
Learn about the latest advancements in HPCC Systems observability, focused on the Open telemetry-based instrumentation framework and how this feature can help you streamline your ECL query workload and ensure your HPCC Systems deployments are functioning at full strength!
How to Secure Your Containerized HPCC Systems Platform Using Terraform
Godji Fortil, LexisNexis Risk Solutions
In our presentations over the last couple of years, we showed you how to deploy the containerized HPCC Systems Platform. As a following step, knowing how to secure your cluster over the internet is very important. This presentation demonstrated how that can be done in the best ways possible using Terraform, htpasswd, or Azure Active Directory.
Parquet Plugin Usage / Test Suite for the Parquet Plugin
Jack del Vecchio & Ilhan Gelle, LexisNexis Risk Solutions
The HPCC Systems Parquet Plugin has received many performance improvements and bug fixes over the past year. In this two-part session, Jack highlighted the improvements compared to native file formats and went into detail about the usage of the Plugin and how to leverage the Parquet file format. Ilhan then provided an overview of the robust test suite for the HPCC Systems Parquet plugin and how it helps enhance the reliability and efficiency of Parquet integration within HPCC Systems.
What’s New – ECL Watch and IDEs
Kunal Aswani, LexisNexis Risk Solutions
Catch up on the latest changes to the interface and appearance in the world of ECL development.

Data DNA

See how these projects enhanced the integrating processes and capabilities of the platform to streamline, enrich and visualize data.

Streamlining Business Tasks with HPCC Systems Clusters and Tombolo
Yadhap Dahal & Matthew Fancher, LexisNexis Risk Solutions
Introducing Tombolo, an innovative open-source project designed to enhance the already robust features, speed, and capabilities of HPCC Systems clusters. Our aim is to introduce a user-friendly web application that caters to both technical and non-technical users, enabling seamless interaction with HPCC Systems clusters. During our presentation, we’ll explore the capabilities of Tombolo, including creating workflows and monitoring assets. Additionally in this session, we’ll highlight the application’s ability to proactively send timely notifications. Moreover, we’ll provide a comprehensive overview of Tombolo’s intuitive dashboard.
Power BI Integration with HPCC Systems
Harsh Raj & Srinivasan Kothandam, LexisNexis Risk Solutions
Learn more about this integration package that enables analysts and business users to read HPCC Systems native files directly from Power BI.
Using NLP++ to Build a Brazilian Address Cleaner in HPCC Systems
Guilherme da Silva, LexisNexis Risk Solutions
NLP++ is a new programming language specially designed to build deep text parsers. The main objective of this approach is to build a Brazilian address analyzer and cleaner that is capable of improving the current cleaning process, with the advantage of being a transparent process with easy problem identification and correction, demonstrating great potential for future use in production.
Enhancing Legal Assistance Through Data Enrichment with HPCC Systems
Nihar Mandahas, Skanda P R, Manvith L B, Pratheek Rao MP, Arya Hariharan, & Dr. Jyoti Shetty,RVCE
The proposed application enhances legal research efficiency and accuracy by using NLP for keyword extraction and leveraging HPCC Systems for rapid data retrieval, ensuring quick and relevant reference searches. Among those who will benefit from this application are lawyers who wish to simplify the task of finding relevant legal references, academics and law students who usually conduct extensive research, and legal organizations overall.
Integrating Microsoft Fabric and HPCC Systems for Security Analytics
Sowmya Myneni & Kushi Kiran, LexisNexis Risk Solutions
Integrating HPCC Systems with Microsoft Fabric and Power BI offers a seamless workflow for transforming, linking, and visualizing data. HPCC Systems, with its user-friendly ECL language, efficiently handles complex data transformations, which can then be imported into Microsoft Fabric. From there, Power BI can be used to create interactive and insightful visualizations and reports. This integration simplifies the process of turning raw data into actionable insights, making it an effective solution for data analysis and presentation.

Productivity Tools

These projects highlighted tools and best practices for maximizing efficiency.

We’ve Come a Long Way From README.txt – Improvements in the Platform Documentation
Jim DeFabia, LexisNexis Risk Solutions
Keeping documentation clear, concise, and user-friendly is crucial for a smooth user experience. We are always working to improve our HPCC Systems® Platform documentation and have recently implemented a few new features to increase use and improve the user experience.
LLM for ECL Code Generation Using Llama3
Connor Davis, DataSeers
Learning and exploring a new programming language can be very challenging, especially when your resources are limited. To help overcome this challenge and improve developers’ productivity, this project proposes the creation of an AI assistant that can help with ECL learning and code development tasks. To achieve this aim, the LLM Llama3 and prompt engineering techniques were leveraged to create a chat bot that can be run locally in VSCode to assist with the generation of ECL code. This is a work in progress development, but the preliminary results are promising and there is a large potential for further expansion and improvements.
Implementing Conditional Cleanup after Regression Testing in HPCC Systems
Goutami Sooda, Arya Vinod, Ahana Patil, & Chandana S, RVCE
The cleanup module designed and developed during this project allows users to choose the cleanup mode and when enabled can effectively delete thousands of workunits generated by regression tests across different clusters, including Thor, Roxie, and hThor. This feature has yielded tangible benefits, including reduced operational costs, improved resource utilization, and enhanced overall efficiency of the regression test engine. By preventing overload on both the cluster and the Dali component, we have significantly enhanced cost-effectiveness and streamlined resource management within the HPCC Systems environment. Overall, this project focuses on contributing to the extensive codebase of the regression test engine.
Data 360° View Using HPCC Systems
S Dhanush & Shreyas Shankar, RVCE
In today's data-driven landscape, organizations face significant challenges in managing and leveraging large volumes of data across diverse platforms. This session presents a cohesive solution developed by RVCE team for ProfitOps Inc., a startup company based out of Cumming, GA, USA, using HPCC Systems to achieve comprehensive data integration and transformation, ensuring seamless connectivity across MySQL, MongoDB, AWS, FTP, and other platforms. HPCC Systems serves as the core technology for this solution, facilitating streamlined data ingestion, synchronization, transformation, and versioning processes. Whether ingesting data from the HPCC Systems landing zone to MySQL or vice versa, the solution supports bidirectional data flows with robust mechanisms for incremental updates, deletions, and version control across various data formats.
Testing Best Practices of the HPCC Systems Platform (Now and in the Future)
Christopher Lo, LexisNexis Risk Solutions
Over the past two years we have been developing a new build system that is more open and reviewable. In this lecture we will discuss; changes to our build machine image generation, leveraging vcpkg for library dependencies, our builds on Github Actions and how to utilize our build workflow code to generate your own custom builds.
Navigating the Platform Build System
Michael Gardner, LexisNexis Risk Solutions
Join us to hear about our current and future testing philosophies and how they are used to validate and test our platform. We'll talk about our current set of testing procedures on various environments and source code repository. We'll also touch on how we perform our testing and how you can replicate these tests on your own environments.
From In-House to Open Source: The Journey of PyHPCC
Amila de Silva & Rohith Podugu, LexisNexis Risk Solutions
PyHPCC is a Python package and wrapper built around the HPCC Systems web services that facilitates communication between Python and HPCC Systems. It was originally developed as a LexisNexis Risk Solutions internal-only tool to automate repetitive work done on HPCC Systems. Since the evangelization of PyHPCC began in 2022, there has been growing interest across the organization and the broader HPCC Systems community to adopt PyHPCC.

Fueling Success

Here are the real world use cases and success stories leveraging HPCC Systems.

Building an NLP Pipeline for Electronic Health Records and Brain MRI Classification
Vishalakshi Prabhu, Eshaan Mathur, Nikhil Vasu, & Prashant Ronad, RVCE
A medical record can includes a variety of types of "notes" entered over time by healthcare professionals, such as observations and administration of drugs/therapies, test results, X-rays, reports, etc. Accordingly, one of the biggest challenges in healthcare is the unavailability of data standardization models. At the same time, most doctors rely on their own knowledge and limited patient data when making decisions. Therefore, accessing the knowledge of many medical professionals would potentially benefit patient care.
This work aims to develop an effective disease identification/classification system using NLP for Electronic Health Records (EHR). Further, it builds a knowledge base for future reference, allowing for querying patient/disease details and for pattern finding. BioBERT is used for text embedding in this project. The selected approach has leveraged pre-processing in such a way that the symptom, duration, gender and affected organ of the patient are labeled/displayed when the whole text is given as input.
Learned Cache Size Setting for Roxie Clusters
Yifan Wang, University of Hawaii
Internal node cache of the index has a non-trivial impact on the Roxie system component. Proper setting of node cache size will result in a significant speedup on data access. However, optimal internal cache sizes depend on various factors, including the access patterns within the index, the compression ratio of data, the disk IO speed, and the time needed to decompress the data.
In this talk, I present our research on setting best cache size using machine learning methods. I first introduce the background and latest works of learning-based knob tuning which aims at predicting the best configurations for data systems. I then present our research in two parts: (1) simulation of Roxie system and (2) learning method over the simulation results.
I hope this talk can provide a basic overview about the knob tuning in AI for database area, as well as help inspire other researchers and developers to improve HPCC Systems using the emerging AI techniques.
This talk is targeting researchers and developers of HPCC Systems to provide insight on facilitating the system using AI techniques and inspire new direction to further improve the platform.
Machine Learning and Cybersecurity Analytics Using the NSL-KDD Dataset
Zularbine Kamal, Kennesaw State University
HPCC Systems currently supports several machine learning algorithms, both supervised and unsupervised, and makes them available via machine learning bundles. In this project, we have leveraged all the algorithms that have data classification capability to detect network intrusion with the dual goal of getting the highest accuracy possible for the trained model while ensuring efficiency during its training. We will also demonstrate the use of the “Myriad Interface”, which can perform multiple independent machine learning tasks within a single interface invocation. Invoking the activities in parallel allows them to be distributed across the nodes in the cluster, thereby maximizing the performance while minimizing the run time. Lastly, we will also cover how the ML_Core.Preprocessing bundle can be used for data preprocessing including label encoding, scaling, and one-hot-encoding.
The content of this presentation is aimed at individuals working with machine learning, cybersecurity specialists, and HPCC Systems users and technology enthusiasts overall.
Model Inversion Attacks with the HPCC Systems Platform
Andrew Polisetty, Kennesaw State University
In this world of machine learning, feeding the model with inputs and training the model is one thing, and securing that model is another. Since many companies leverage machine learning models for decision-making using sensitive data, attackers can target these data-sensitive models, and one of the biggest threats to these models is MIA (Model Inversion Attack). Mainly, MIA is a technique that can be leveraged to reconstruct sensitive information such as financial data. Attackers can access these models and gather predicted data, which can be used as input for training a new model similar to the original one. The attackers can then infer the sensitive data from the original model to reconstruct the training data or build a comparable model. 
In this project, we have leveraged a public loan dataset and utilized the HPCC Systems platform to perform black-box attacks and to design solutions to prevent them. We first developed the machine learning model and then utilized it to build the original, threat and defender models. For the original model, we chose a credit risk assessment scenario consisting of a person’s loan and personal details. We developed the original model by using the learning trees algorithm from the HPCC Systems machine learning bundle. In this scenario, the attackers would access the inputs and outputs of the model through querying, where they can analyze the data points to perform a black-box attack. In the attack model, we simulate an attack by querying the original model and we train the attack model using its output. For the prevention model, we are currently exploring different approaches, such as adding noise to the output of the original model to manipulate the attacker. So far, the raw accuracy obtained from the learning trees algorithm is still relatively low, so we are also training models using logistic regression and continuing to explore different defenders for the prevention model.
School Safety and Security Using RFID and Drones
Taiowa Donovan & Nick Schwartz, American Heritage School
The continuation of our school security project has grown to incorporate drones with thermal imagery along with RFID (Radio Frequency Identification) The new drone platform allows us to collect more detailed real time data, higher resolution real time images for the facial recognition software, double our flight time with extra battery capabilities, a camera with 56x zoom, a wide-angle camera for high-precision campus mapping, and a thermal camera for heat source inspection. This new drone has increased rapid response time to investigate security threats detected by our current patrol drones and security staff. Thermal cameras allow us to track students during code red events and have assisted in the training of security personnel.
Exploring the Capabilities of HPCC Systems in Facilitating Inter Fog Communication
Henrique Antonio Buzin Vargas, Federal University of Santa Catarina (UFSC)
The increasing proliferation of devices connected to the Internet of Things (IoT) has generated significant challenges in terms of integration and interoperability due to the diversity of communication protocols. Fog computing has emerged as an effective solution to reduce latency and improve system efficiency by bringing data processing closer to the source. However, communication between different fogs and clouds still faces considerable obstacles due to the absence of a protocol conversion layer. This session will explore how HPCC Systems can be used to facilitate efficient communication between different fogs. The presentation will address a modular architecture that supports protocol conversion and communication between fogs and clouds, providing a flexible and scalable solution for IoT environments. Key topics to be discussed include an overview of HPCC Systems and its large-scale data processing capabilities, the structure and functionalities of the modular layered architecture developed to support interoperability between different devices and systems, the methodology adopted for the implementation and testing of the architecture, including the evaluation of its efficiency, latency, security, and scalability, and preliminary results and performance analysis of the architecture in practical scenarios. Lastly, insights will be shared about the challenges faced during the development of the architecture and proposed guidelines for future implementations. This session is aimed at researchers, developers, and technology professionals seeking solutions to improve the integration and efficiency of fog computing systems. Discover how HPCC Systems can transform communication and data processing in distributed environments and drive the evolution of IoT.
Internship to Impact: Real Life Success in the HPCC Systems Community
George S Foreman, Christopher Connelly, Jack del Vecchio, Yash Mishra, Nathalia Ribas, & Fulvio Favilla Filho, LexisNexis Risk Solutions
Join us for a moderated panel style interview with former interns who have emerged from the HPCC Systems Summer Internship Program. In this session, the next generation of technologists will share their experiences and successes working with the HPCC Systems community and learn from their experiences during their internship and how they transitioned to full time RELX employees.

ECL Training Workshop

The HPCC Systems Training and Support team has been very busy this year visiting universities and trade show events to promote HPCC Systems and ECL through Code Days, Hackathons, and Workshops. Here is this year’s Community Summit workshop:

Sleep Well with ECL: Job Automation and Scheduling
Bob Foreman, Software Engineering Lead, LexisNexis Risk Solutions
This workshop showcased the latest work regarding ECL Scheduling and Process Automation. In addition, many ECL best practices are demonstrated along the way.
Topics included:
- Creating and using FUNCTIONMACROs
- Creating and using MACROs
- The ECL Template Language
- Automating ECL
- ECL Scheduling