EclWatch logical file copy to other clusters, loses blockCompression flag and adds other spurious meta flags.
Environment
Description
Conclusion
Activity
Jacob Cobbett-Smith February 26, 2016 at 11:05 AMEdited
But this is a flat file, not xml/csv, so serve no purpose afaik.
[ And attributes were not needed in the original file. ]
Attila Vamos February 26, 2016 at 9:36 AM
The footerLength="0" and headerLength="0" are not an erroneous addition. They are result of issue Should be able to despray XML without rowTag specification
Jacob Cobbett-Smith February 25, 2016 at 5:32 PM
@Attila Vamos - I'm guessing this has something to do with the preserve compression issue.
You can reproduce it, by creating a compressed file on 1 cluster, then copying it (leaving Preserve Compression ticket) to another cluster.
NB: There's a related issue I opened : https://hpccsystems.atlassian.net/browse/HPCC-15004#icft=HPCC-15004, which prevents you turning off Preserve Compression.
But for the purposes of this bug, that is not relevant.
Jacob Cobbett-Smith February 25, 2016 at 5:28 PM
Definitely related to DFU copy + source files that are compressed.
It seems to be losing the compression flag 'blockCompressed=1'.
Without it, it's reading the compressed parts as uncompressed and reading nonsensical noise as a result (and ultimately hitting the assert seen).
For the original file reported, the source file has been deleted/no longer exists, which is a pity.
For the second example, "~thor_data400::bipv2_empid_platform::init::20160112" and the DFU copy "~thor_data400::bipv2_empid_platform::init::20160112copy", both copies still exist and not only is the blockCompressed flag missing, there are other oddities in the meta data for the copy versus the original.
Original attributes:
<Attr accessed="2016-02-23T20:55:01"
blockCompressed="1"
checkSum="4294967295"
compressedSize="23227423550"
formatCrc="-1325046051"
job="BIPv2_EmpID 20160112 Init "
kind="flat"
owner="hsun"
recordCount="589236292"
recordSize="979"
size="576862329868"
workunit="W20160223-132922">
Copy attributes:
<Attr accessed="2016-02-23T21:07:09"
compressedSize="34038286219"
footerLength="0"
formatCrc="-1325046051"
headerLength="0"
job="thor_data400::bipv2_empid_platform::init::20160112copy"
kind="flat"
owner="hsun"
recordCount="589236292"
recordSize="979"
size="576862329868"
workunit="D20160223-145058">
Note the missing blockCompressed="1" in the copy + the addition of erroneous footerLength="0" and headerLength="0"
checkSum is also missing.
Hongzhi Sun February 23, 2016 at 9:31 PM
I forgot to provide the Rec record structure. Add it below:
Rec:=RECORD
unsigned6 rcid;
string2 source;
unsigned6 empid;
unsigned6 seleid;
unsigned6 orgid;
unsigned6 ultid;
string50 contact_job_title_derived;
string10 prim_range;
string28 prim_name;
string5 zip;
string4 zip4;
string20 fname;
string20 lname;
string10 contact_phone;
unsigned6 contact_did;
string9 contact_ssn;
string120 company_name;
string8 sec_range;
string25 v_city_name;
string2 st;
string2 company_inc_state;
string32 company_charter_number;
string9 active_duns_number;
string9 hist_duns_number;
string30 active_domestic_corp_key;
string30 hist_domestic_corp_key;
string30 foreign_corp_key;
string30 unk_corp_key;
string9 company_fein;
string10 cnp_btype;
string250 cnp_name;
string50 company_name_type_derived;
unsigned6 company_bdid;
unsigned3 nodes_total;
unsigned4 dt_first_seen;
unsigned4 dt_last_seen;
string1 iscorp;
string120 cname_devanitize;
string1 iscorpenhanced;
END;
I created BIPV2_POWID_HS and applied salt verson 3.3 to the _SPC to generate the ecl code. When I ran the specificities, I got error.
W20160213-135916 in dataland. Everything is checked in.
The error:
Error: System error: 3000: Graph[1], diskread[2]: SLAVE #1 [10.241.12.101:25000]: assert(required <= maxOffset) failed - file: /var/lib/jenkins/workspace/LN-Candidate-withplugins-5.4.8-1/LN/centos-6.4-x86_64/HPCC-Platform/common/thorhelper/thorcommon.ipp, line 821, assert(required <= maxOffset) failed - file: /var/lib/jenkins/workspace/LN-Candidate-withplugins-5.4.8-1/LN/centos-6.4-x86_64/HPCC-Platform/common/thorhelper/thorcommon.ipp, line 821 - handling file: /var/lib/HPCCSystems/hpcc-data/thor/thor_data400/bipv2_powid/init/20160112copy._1_of_400
Info: 4, MP link closed (10.241.12.130:25000) : Error aborting job, will cause thor restart
Please help. I cannot figure out the code. What is “thorcommon.ipp”?