Browse: Home, Abstracts, Winners and runners up, Awards Ceremony (Watch Recording), Posters by HPCC Systems Interns, Posters by Academic Partners, Poster Judges, Virtual Judging, Virtual Poster Booths

Open	Nathan Halliday joined the program as a high school student. Following his internship experience with HPCC Systems, Nathan has now moved on to study Mathematics at university.  2020 Best Poster - Platform Enhancement

Poster Abstract

The ECL language is centered around high performance. HPCC Systems focuses on parallelism to enable highly optimized dataset operations.

The parallel workflow engine increases the scope of parallel processing from within activity graphs to the entire workflow. The goal is to make workunits faster but maintain the existing behavior of the sequential engine.

During my project, I have gradually extended the parallel engine to increase support for different ECL language constructs. Regression tests for different workflow modes in combination, ensure that the engine can process diverse queries.

One major challenge of the parallel engine was to implement condition items, since only one sub-branch of dependencies are executed by the engine. It also has a complex task of mimicking the sequential engine if the workflow fails.

The parallel workflow algorithm is planned to become default in HPCC Systems version 7.12.0 It is beneficial for all ECL programmers and the speedup is achieved without altering the language functionality. For production systems, money will be saved, by providing the clusters with more work sooner. For cloud environments, additional resources can be added dynamically, to maximize the benefits of the faster processing.

Presentation

In this Video Recording, Nathan provides a tour and explanation of his poster content.

Poster Title: The Parallel Workflow Engine

Click on the poster for a larger image. 

Open