Page Comparison

This project is available as a student work experience opportunity with HPCC Systems. Curious about other projects we are offering? Take a look at our Ideas List.

Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community. 

Find out about the HPCC Systems Summer Internship Program.

Project Description

ESP (Enterprise Services Platform) is an HPCC Systems component that hosts web services functions. It’s a container for running services. ESP is what provides you users and organizations with the ability to write and run web services in HPCC Systems, support revenue based products, and use can leverage the dynamic ESDL engine which gives you for flexibility and ease of use. A couple of examples of ESP services are WsECL, which allows you to interface to published queries on Roxie or Thor, and ECL Watch which is our the web-based query execution, monitoring and file management interface. ESP supports a number of protocols including, SOAP, XML, HTTP, REST and JSON.  

...

The objective of this project is to develop a command line tool with such capabilities. The tool would take the XML files generated from the SCM ECM files for the ESP services and use them to dynamically configure a tool for calling those services. The syntax supported by the command line tool would be something like "hpcc <service> <method> <target> :

hpcc <service> <method> <target> [options]

...

where the options parameter are all of the form "--<method-parameter>=<value>". For example:

hpcc WsDfu DFUSearch myEspUrl --showExample=yes

To accomplish this, the following process needs to be followed:

1. Make an educated guess of bucket limits (probably for each method) and add support to the SCM file
2. Run a variety of tests against the service that represents a mix of work
3. Review the results in Prometheus (or the file or log sink could be used)
4. Adjust the bucket limits to improve the distribution, repeat starting at 2 above

These steps need to be executed iteratively through each ESP service for which execution profiling needs to be enabled and would give us a valid starting point to collect ESP service metrics.requirements must be considered:

• It should be completely generalized with no special casing - configured purely by the XML files.
• It should be integrated with the Linux command completion and provide context sensitive completion.
• It should have built in help, also generated from the XML files.
• Extension: The output could be controlled to use JSON/XML and it could use XPATH to extract values from the result.
• It would initially only support scalar arguments. 

Completion of this project involves:

• Learning general HPCC Systems architecture.
• Learning ESP web services frameworkLearning monitoring tools such as Prometheus.
• Development of a functional command line tool as specified above.
• Creation of test cases for web ESP services and specific methods
• Definition of appropriate bucket limits for each method/service 
• .
• Creation of documentation including a github GitHub repository for the project.
• A blog, a recorded presentation, and a poster artifact about your project (see examples from previous years here).