Index Build: getNextSize Request for -65443 bytes

Description

When running an Index Build we are getting a strange error that appears to be trying to grab a negative record size based on the error message:

System error: -1: [Graph graph1[102], rollupgroup[110]|http://alpha_vault_thor_esp.risk.regn.net:8010/?Widget=WUQueryWidget&__filter=Owner%3Dsvc-kel#]: SLAVE #104 [10.194.90.104:24000]: getNextSize: Request for -65443 bytes oldMax = 0, - caused by (-1, getNextSize: Request for -65443 bytes oldMax = 0)

 

http://alpha_vault_thor_esp.risk.regn.net:8010/?Wuid=W20200612-171431&Widget=WUDetailsWidget

 

Conclusion

None

Activity

Show:

Gavin Halliday June 19, 2020 at 4:11 PM

Assigning to me to improve the error so that it reports the correct old size.

Jacob Cobbett-Smith June 19, 2020 at 4:11 PM

Assigning to to improve the error message.

Jacob Cobbett-Smith June 19, 2020 at 4:03 PM

The dataset that is trying to fit into the payload field of 1 row of the index it's building here is unworkably large.
It's over 21 million rows in size, and has exceeded 2^32 bytes in size at the point of failure (not sure what final total size would have been).
NB: based on it's estimated record size, it might be as ~20GB

Even if less and it managed to succeed to build this index,
the resulting index would have been practically unusable, with single rows of ~4GB,
and with the nature of indexes being used to lookup and serialize rows remotely, it would have performed extremely inefficiently.

The error should be improved a bit, so that's clearer that huge amounts of memory are involved.
There is also an existing JIRA to complain if an ECL query tries to create indexes with huge blob fields like this: https://hpccsystems.atlassian.net/browse/HPCC-24259

Richard Chapman June 16, 2020 at 3:48 PM

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Point

Fix versions

Affects versions

Created June 15, 2020 at 3:53 PM
Updated June 22, 2020 at 1:14 PM
Resolved June 22, 2020 at 1:14 PM

Flag notifications