Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Improve WUListQueries performance on clusters with several thousand roxie queries

Description

When viewing queries on clusters that have several thousand roxie queries (for example http://ramps_dr_esp:8010/?Widget=QuerySetQueryWidget#/stub/PublishedQueries) , the performance of WUListQueries slows to a crawl. Sometimes it times out.

 

This is occurring on a 6.4.x cluster, but looking at the code in both the 6.4 and master branches of the hpcc platform, there doesn't appear to be anything that jumps out as a perfromance enhancement, so the problem probably still exists in 7.x.

 

https://github.com/hpcc-systems/HPCC-Platform/blob/e7ead5c2ee3304c35113906a6774fc5878f6f5fa/esp/services/ws_workunits/ws_workunitsQuerySets.cpp#L1432  (master branch)

https://github.com/hpcc-systems/HPCC-Platform/blob/candidate-6.4.26/esp/services/ws_workunits/ws_workunitsQuerySets.cpp#L1546 (6.4.26 branch)

 

Which calls this :

https://github.com/hpcc-systems/HPCC-Platform/blob/0660e4a2fec76c9457770eda0e0077282be0e969/common/workunit/workunit.cpp#L5323  (master branch)

https://github.com/hpcc-systems/HPCC-Platform/blob/candidate-6.4.26/common/workunit/workunit.cpp#L2476 (6.4.26 branch)

Conclusion

None

Activity

Show:

Jacob Cobbett-Smith June 19, 2020 at 1:28 PM

yes should be.

Drea Leed June 19, 2020 at 1:14 PM

dang, that was fast! Will this be in 7.10?

Jacob Cobbett-Smith June 19, 2020 at 1:10 PM

The attached PR should speed things up significantly I think.
The bulk of the remaining time (when CheckAllNodes is set to false) was consumed in the Alias matching code.
In Ramps it took ~ 2.5 minutes
In local testing it took ~30 seconds, with this PR it took ~ 1.1 seconds.
So once this change is in Ramps, it is likely to reduce the remaining 2.5 seconds to a few seconds.

Jacob Cobbett-Smith June 19, 2020 at 10:48 AM

- one way to speed WUListQueries up, if you don't require it to validate all roxie nodes, is to pass CheckAllNodes=false to the request.
By default it is set to true.

By setting to false, it reduces the request time by approx. 50%.

The bulk of the rest of the time appears to be from the way the services scans query aliases. I plan on releasing an improved version soon.

Drea Leed June 18, 2020 at 4:03 PM

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Fix versions

Pull Request URL

https://github.com/hpcc-systems/HPCC-Platform/pull/13880

Created June 18, 2020 at 4:02 PM
Updated July 8, 2020 at 12:24 PM
Resolved June 22, 2020 at 1:12 PM