Missing Data (blank row) in ECL IDE and ECL Watch for workunit results

Description

Hello:

I'm encountering a strange issue in Fido Prod. Here's my WUID W20220826-100137. With it, a txt file is generated. When opening result in ECL IDE or the txt file in ECL watch, there're many rows being blank. If I export it from here by clicking "CSV" button, the empty rows remain so that we miss data in reporting.  But if I despray the txt file to landing zone, all rows are there with nothing missing. Can anyone please advise what's wrong with ECL IED/Watch? 

 

Conclusion

None

Activity

Show:

Gavin Halliday September 8, 2022 at 1:10 PM

Note: 0x96 is not legal utf8, and the file is marked as utf8 - which is causing the problem.

Sometimes it would have been substituted with a substitution character, but I found at least one case where an invalid utf8 byte would lead to a blank string.  I will release a fix - it will make it clearer when there is an issue - but the files still need cleaning/encoding properly.

 

 

Gavin Halliday September 8, 2022 at 10:59 AM

You probably need to replace 0x96 with 0x2D (hyphen minus) as part of the cleaning.

 

 

Gavin Halliday September 8, 2022 at 10:57 AM

The first two blank lines (line 187 and 200) both correspond to lines in the file that contain a character hex 0x96. 

That corresponds to an EN dash in the windows-1252 code pages, but is not defined in the iso-8859-1 code page that we use for string representations.  I suspect the conversion from string to utf8 fails for some reason because of that character.  I'll debug locally.

 
 

Kanghua Wang August 27, 2022 at 12:00 AM

Not sure the following information would help for debugging the issue: I desprayed the logical file to a landing zone. I sprayed the file back and got another logical file with the same size as the original logical file. But, I can see the missing data rows in the second logical file.

Stuart Ort August 26, 2022 at 6:52 PM

 

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Point

Fix versions

Roadmap

Not applicable

Created August 26, 2022 at 6:42 PM
Updated September 9, 2022 at 8:57 AM
Resolved September 9, 2022 at 8:57 AM

Flag notifications