Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Filter activities are treated as 'fast through' but if sparse can cause downstream slowness due to lack of look aheads

Description

Filters are marked as fast through at the moment, with the expectation they output records without delay, but if the filter doesn't match often, the output will be slow.
FastThrough is used downstream to determine whether a lookahead is needed in various cases, e.g. by a CHOOSEN.
In CHOOSEN's case, if there's no lookahead and the input count is slow and small it will mean there's a big delay before it even looks at the next part, etc., so the operation becomes a serial process, making it a lot slower than it would be if a lookahead had been involved and allowed the filtered read to be parallelized.

Conclusion

None

Activity

Show:

Jacob Cobbett-Smith June 21, 2018 at 9:42 AM

- a potential workaround for this is to use CHOOSEN(CHOOSEN(expr, <n>, LOCAL), <n>);
i.e. force a local choosen 1st, which will ensure done in parallel.

Jacob Cobbett-Smith June 18, 2018 at 4:24 PM

Comment from PR:

This address the issue a serial activities (like choosen) whose input is a compound diskread/filter or standard filter activity, not using a read ahead reader because the inputs were deemed 'fast', but can not in general be assumed to be because a filter may filter out most rows and deliver output very slowly.

There is a wider issue that this was being caused by isFastThrough being called too early before the input started, which is beyond the scope of 6.4.x. A separate JIRA for that address those issues - see

Jacob Cobbett-Smith June 6, 2018 at 4:53 PM

Options:
1) Could mark all filters as not 'fast through' - as can't tell whether they will be or not.
2) Or could treat 'fast through' as a hint, but for activities that are serial in nature, always use at least a in-memory lookahead to counter any pathological slowness up stream per part.

- thoughts?

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Major

Fix versions

Pull Request URL

https://github.com/hpcc-systems/HPCC-Platform/pull/11327

Created June 6, 2018 at 4:35 PM
Updated June 29, 2018 at 8:18 AM
Resolved June 28, 2018 at 12:58 PM