Versions Compared

Version Old Version 49 New Version Current
Changes made by

Lorraine Chapman

Hugo Watanuki

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The proposal application period for 2021 internships is now closed. The proposal period for 2022 internships will open in the Fall.

This project is available as a student work experience opportunity with HPCC Systems. Curious about other projects we are offering? Take a look at our Ideas List. 

...

This project was completed by a student accepted on to the 2022 HPCC Systems Intern Program.

Project Description

The HPCC Systems platform currently supports embedding Python, Java, Javascript, R, MySQL and C++ and Cassandra code.The goal of this project is to support MongoDB by allowing the embedding of MongoDB database queries within ECL code running on HPCC Systems.

One of the challenges of this project, is to address how an external key-value store interacts with a distributed thor query so that the external datastore acts like a distributed file read by each node in the thor or where only a portion of a result is written. This is currently something the HPCC Systems developers are looking at and are actively discussing but have not resolved.

...

Completion of this project involves:

  • Investigating the API for calling MongoDB from C/C++.

  • Creating a simple wrapper for scalar values between the ECL embed API and the MongoDB API using one of the existing embed plugin implementations as an example.

  • Extending the simple wrapper to handle structured data.

  • In parallel with the above, developing test cases for the plugin that include coverage of all data types both passed in and returned, including multi-threaded access from the ECL side. This includes testing the performance and throughput of the system for some examples that approximate to real-world usage.

By the mid term review we would expect you to have:

...

  • Implemented a simple example that passes and returns scalar values (which are usually much simpler than passing/returning structures).

...

Mentor

Dan Camper
Contact Details

...

Skills needed

  • Ability to code in C++.

  • Ability to build and test the HPCC system (guidance will be provided).

  • Knowledge of MongoDB sufficient to write and run test cases.

  • Ability to write test code. Knowledge of ECL is not a requirement since it should be possible to re-use existing code with minimal changes for this purpose. Links are provided below to our ECL training documentation and online courses should you wish to become familiar with the ECL  language.

Deliverables

Midterm

  • A simple example that passes and returns scalar values.

End of project

  • A plugin that supports interfacing to MongoDB from ECL, that will implement the ECL embedded language API and make calls to MongoDB embedded via its C/C++ api (assuming it has one!).

  • Test cases demonstrating the correct behaviour and performance of the plugin.

  • Documentation of how datatypes and structures in ECL are mapped to MongoDB.

Other resources