Balanced splitter doesn't adequately consider memory footprint of row usage where child datasets involved

Description

A balanced splitter uses a [configurable] small memory buffer to hold rows.
It blocks when this limit is read until other arms have caught up.
To know when it has read ahead enough, it uses a rowSize method, but unfortunately this was based on the size returned from the generated output meta, and not the actual potential footprint.
This is particularly significant when child datasets are involved, where the actual memory footprint is variable and quite likely much bigger than the nominal size of the parent record.

Elsewhere in similar situations a size serializer is used to get the actual size.
Balanced splitter should do the same.

Conclusion

None

Activity

Show:

Jacob Cobbett-Smith December 9, 2019 at 4:03 PM

The symptom of this issue, is to hit roxiemem out of memory errors.
If seen, a workaround is to add:

#option('splitterSpill', 1);

..to the query.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Point

Fix versions

Created November 18, 2019 at 1:13 PM
Updated December 9, 2019 at 4:03 PM
Resolved November 21, 2019 at 1:48 PM

Flag notifications