About Jefferson Mao
...
Jefferson Mao is a High School student studying at
...
Lambert High School, Georgia, USA. Jeff's project was one he suggested himself having learned about HPCC Systems from his involvement in CodeDay during the last year.
Poster Abstract
The ECL language is centred around high performance. HPCC Systems focuses on parallelism to enable highly optimised dataset operations.
The parallel workflow engine increases the scope of parallel processing from within activity graphs to the entire workflow. The goal is to make workunits faster but maintain the existing behaviour of the sequential engine.
During my project, I have gradually extended the parallel engine to increase support for different ECL language constructs. Regression tests for different workflow modes in combination, ensure that the engine can process diverse queries.
One major challenge of the parallel engine was to implement condition items, since only one sub-branch of dependencies are executed by the engine. It also has a complex task of mimicking the sequential engine if the workflow fails.
The parallel workflow algorithm is planned to become default in HPCC Systems version 7.12.0 It is beneficial for all ECL programmers and the speedup is achieved without altering the language functionality. For production systems, money will be saved, by providing the clusters with more work sooner. For cloud environments, additional resources can be added dynamically, to maximise the benefits of the faster processingGoogle Anthos is an Application Management Platform that manages multi-cloud and on-premise environments. It allows HPCC-Systems to be managed from separate cloud platforms through one centralized command center.
Google Anthos comes with a plethora of options that range from config management (configurations through code) to service mesh (microservice controller/manager).
The main benefit Anthos provides for HPCC-Systems is the ability to manage Kubernetes environments on any cloud. With Anthos, HPCC Systems has access to a common abstract layer that manages deployment, upgrades, configurations, networking, and scaling.
Presentation
In this Video Recording, Nathan provides a tour and explanation of his poster content.
Poster Title: The Parallel Workflow Engine
Click on the poster for a larger image. The original PDF version can be found here. (Available for download).
