Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Add support for releasing rows in blocks on parallel threads

Description

If rows cover a large address space, and the rows are being released in a random order (e.g., following a sort), a significant proportion of the time can be spent while the cpu is stalled waiting for the pages to get into the level3 cache.

Disposing of the rows in blocks on multiple threads mitigates much of the problem.

A first experiment applying this to 280M rows, which occupied ~30Gb of memory cut the time to dispose from about 35s to 5-10s.

Conclusion

None

Activity

Show:

Richard Chapman November 27, 2015 at 10:23 AM

Gavin Halliday June 10, 2015 at 12:23 PM

Note to self - my branch issue13690 contains the proof of concept.

Better would be to

  • move the code into the heap manager? Almost certainly needed to allow good async release.

  • add a global function to release a block of rows (both synchronous and async)

  • Use a semaphore to wait for all the releases to complete (but careful about the count)

  • Think more about the cases that might commonly benefit + how to support them

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Fix versions

Pull Request URL

https://github.com/hpcc-systems/HPCC-Platform/pull/7895
https://github.com/hpcc-systems/HPCC-Platform/pull/7895

Created June 8, 2015 at 10:50 AM
Updated November 27, 2015 at 10:23 AM
Resolved November 25, 2015 at 2:53 PM