EclWatch logical file copy to other clusters, loses blockCompression flag and adds other spurious meta flags.

Environment

Salt 3.3; ecl IDE community_5.4.2-1; server internal_5.4.8-1 BIPV2_POWID_DOWN_HS and BIPV2_POWID_HS

Description

I created BIPV2_POWID_HS and applied salt verson 3.3 to the _SPC to generate the ecl code. When I ran the specificities, I got error.
W20160213-135916 in dataland. Everything is checked in.
The error:
Error: System error: 3000: Graph[1], diskread[2]: SLAVE #1 [10.241.12.101:25000]: assert(required <= maxOffset) failed - file: /var/lib/jenkins/workspace/LN-Candidate-withplugins-5.4.8-1/LN/centos-6.4-x86_64/HPCC-Platform/common/thorhelper/thorcommon.ipp, line 821, assert(required <= maxOffset) failed - file: /var/lib/jenkins/workspace/LN-Candidate-withplugins-5.4.8-1/LN/centos-6.4-x86_64/HPCC-Platform/common/thorhelper/thorcommon.ipp, line 821 - handling file: /var/lib/HPCCSystems/hpcc-data/thor/thor_data400/bipv2_powid/init/20160112copy._1_of_400
Info: 4, MP link closed (10.241.12.130:25000) : Error aborting job, will cause thor restart

Please help. I cannot figure out the code. What is “thorcommon.ipp”?

Conclusion

None

Activity

Show:

Jacob Cobbett-Smith February 26, 2016 at 11:05 AM
Edited

But this is a flat file, not xml/csv, so serve no purpose afaik.
[ And attributes were not needed in the original file. ]

Attila Vamos February 26, 2016 at 9:36 AM

The footerLength="0" and headerLength="0" are not an erroneous addition. They are result of issue Should be able to despray XML without rowTag specification

Jacob Cobbett-Smith February 25, 2016 at 5:32 PM

- I'm guessing this has something to do with the preserve compression issue.

You can reproduce it, by creating a compressed file on 1 cluster, then copying it (leaving Preserve Compression ticket) to another cluster.

NB: There's a related issue I opened : https://hpccsystems.atlassian.net/browse/HPCC-15004#icft=HPCC-15004, which prevents you turning off Preserve Compression.
But for the purposes of this bug, that is not relevant.

Jacob Cobbett-Smith February 25, 2016 at 5:28 PM

Definitely related to DFU copy + source files that are compressed.

It seems to be losing the compression flag 'blockCompressed=1'.
Without it, it's reading the compressed parts as uncompressed and reading nonsensical noise as a result (and ultimately hitting the assert seen).

For the original file reported, the source file has been deleted/no longer exists, which is a pity.

For the second example, "~thor_data400::bipv2_empid_platform::init::20160112" and the DFU copy "~thor_data400::bipv2_empid_platform::init::20160112copy", both copies still exist and not only is the blockCompressed flag missing, there are other oddities in the meta data for the copy versus the original.

Original attributes:

<Attr accessed="2016-02-23T20:55:01" blockCompressed="1" checkSum="4294967295" compressedSize="23227423550" formatCrc="-1325046051" job="BIPv2_EmpID 20160112 Init " kind="flat" owner="hsun" recordCount="589236292" recordSize="979" size="576862329868" workunit="W20160223-132922">

Copy attributes:

<Attr accessed="2016-02-23T21:07:09" compressedSize="34038286219" footerLength="0" formatCrc="-1325046051" headerLength="0" job="thor_data400::bipv2_empid_platform::init::20160112copy" kind="flat" owner="hsun" recordCount="589236292" recordSize="979" size="576862329868" workunit="D20160223-145058">

Note the missing blockCompressed="1" in the copy + the addition of erroneous footerLength="0" and headerLength="0"
checkSum is also missing.

Hongzhi Sun February 23, 2016 at 9:31 PM

I forgot to provide the Rec record structure. Add it below:
Rec:=RECORD
unsigned6 rcid;
string2 source;
unsigned6 empid;
unsigned6 seleid;
unsigned6 orgid;
unsigned6 ultid;
string50 contact_job_title_derived;
string10 prim_range;
string28 prim_name;
string5 zip;
string4 zip4;
string20 fname;
string20 lname;
string10 contact_phone;
unsigned6 contact_did;
string9 contact_ssn;
string120 company_name;
string8 sec_range;
string25 v_city_name;
string2 st;
string2 company_inc_state;
string32 company_charter_number;
string9 active_duns_number;
string9 hist_duns_number;
string30 active_domestic_corp_key;
string30 hist_domestic_corp_key;
string30 foreign_corp_key;
string30 unk_corp_key;
string9 company_fein;
string10 cnp_btype;
string250 cnp_name;
string50 company_name_type_derived;
unsigned6 company_bdid;
unsigned3 nodes_total;
unsigned4 dt_first_seen;
unsigned4 dt_last_seen;
string1 iscorp;
string120 cname_devanitize;
string1 iscorpenhanced;
END;

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Minor

Fix versions

Created February 16, 2016 at 8:27 PM
Updated April 19, 2016 at 8:33 AM
Resolved April 19, 2016 at 8:33 AM

Flag notifications