Copy of Add execution profiling for ESP services

This project is available as a student work experience opportunity with HPCC Systems. Curious about other projects we are offering? Take a look at our Ideas List.

Student work experience opportunities also exist for students who want to suggest their own project idea. Project suggestions must be relevant to HPCC Systems and of benefit to our open source community. 

Find out about the HPCC Systems Summer Internship Program.

Project Description

ESP (Enterprise Services Platform) is an HPCC Systems component that hosts web services functions. It’s a container for running services. ESP is what provides you with the ability to write and run web services in HPCC Systems, support revenue based products and use the dynamic ESDL engine which gives you flexibility and ease of use. A couple of examples of ESP services are WsECL, which allows you to interface to published queries on Roxie or Thor, and ECL Watch which is our web-based query execution, monitoring and file management interface. ESP supports a number of protocols including, SOAP, XML, HTTP, REST and JSON.  

Such a powerful component of the HPCC Systems stack requires refined instrumentation techniques and the current goal is to add execution profiling for ESP services at the method level. Key to taking advantage of this feature is to define the execution timing buckets used to measure the execution time distribution for each method. Unfortunately, this is something that needs to be set manually as the histogram metric being used does not adjust the buckets based on measurement data (nor should it). This means that the bucket limits need to be tailored at a minimum for each service and probably individually for each method.

The objective of this project is to provide starting point values for the execution timing buckets used to measure the execution time distribution of each method. To accomplish this, the following process needs to be followed:

1. Make an educated guess of bucket limits (probably for each method) and add support to the SCM file
2. Run a variety of tests against the service that represents a mix of work
3. Review the results in Prometheus (or the file or log sink could be used)
4. Adjust the bucket limits to improve the distribution, repeat starting at 2 above

These steps need to be executed iteratively through each ESP service for which execution profiling needs to be enabled and would give us a valid starting point to collect ESP service metrics.

Completion of this project involves:

• Learning general HPCC Systems architecture
• Learning ESP web services framework
• Learning monitoring tools such as Prometheus
• Creation of test cases for web services and specific methods
• Definition of appropriate bucket limits for each method/service 
• Creation of documentation including a github repository for the project