HPCC Systems 5.0.x Releases
- Lorraine Chapman
- Jim DeFabia
Welcome to HPCC Systems® 5.0. Please review the information included in both categories shown below.
General -Details of the impact of a change and information about any modifications you may need to make to your ECL code. Where relevant, the release which includes the fix is indicated, although be aware that this could be a future release. (Check the HPCC Systems® download page for the latest releases) Other entries contain useful information about new behaviour in 5.0, for your infomation.
Significant New Features - We recommend you consider using these features right away.
General
Updated for 6.0.0 - New 100% Javascript Graph Viewer
From HPCC Systems release 6.0.0, a new 100% Javascript Graph Viewer will be available to circumvent the problems caused by the phasing out of NPAPI support. (Chrome users will be unable to view graphs in ECL Watch which previously relied on NPAPI support.) There will be no separate installation process for the new Javascript Graph Viewer, it will simply run automatically within your browser. This also has security benefits since it will run without the need to activate or install any controls or plugins from potentially unknown sources. It will also be supported by most modern browsers removing the restrictions imposed by the withdrawal of NPAPI support.
Note: This affects users of Chrome 35 on Linux and Chrome 42 on Windows. However, support will also be phased out for Chrome on OSX and Windows at some point in the future. You can find out more information about this from the Chromium blog here: http://blog.chromium.org/2013/09/saying-goodbye-to-our-old-friend-npapi.html and information for developers is here: http://www.chromium.org/developers/npapi-deprecation.
In HPCC Systems 6.0.0 the "tree graph view" is now the default Graph Viewer. It also now defaults to using the JavaScript graph control. The old version of Graph Control is still available and can be invoked from the Graphs tab in ECL Watch. Select a graph and choose:
Open (native) option. Use the new graph page with the ActiveX graph control if its available.
Open (legacy) option. Use the old default graph page which uses the ActiveX control if available otherwise it will use the JS graph control.
Note: The Javascript Graph Viewer was introduced as a technical preview in HPCC Systems 5.2.0. You can find out more about it in the blog about new features in HPCC Systems® 5.2 which illustrates how it will enable us to improve the quality of information we can provide for you about your job via graphs.
https://track.hpccsystems.com/browse/HPCC-12805
Issue deleting logical filenames with leading digits on scopes or names (fixed in 5.4.0)
In some situations, Logical filenames created via spraying may create scopes/files with leading digits. Deleting logical filenames in this situation produces an error similar to the one below and the delete action fails:
Cannot delete .::2015050100.filename__filename: [ 25: SDS: IPropertyTree exception
SDS Reply Error : SDS: IPropertyTree exception
IPropertyTree: xpath parse error
XPath Exception: Qualifier expected e.g. [..]
in xpath = 2015050100_filename_95_filename^ in xpath '/_Locks/SuperOwnerLock/2015050100_filename_95_filename']
Logical filenames of this type can only be removed using Daliadmin .
This issue was introduced in HPCC Systems 5.0 as a result of this issue: https://track.hpccsystems.com/browse/HPCC-10770 and still exists in HPCC Systems 5.2.0. However it has been fixed in HPCC Systems 5.4.0.
https://track.hpccsystems.com/browse/HPCC-13981
Restart needed for ECL IDE after downloading GraphViewer in ECL Watch
When installing a clean version of the platform, you may find that graphs are not displaying in ECL IDE and you need to install the new Graph Viewer. To do this, go to ECL Watch and you will be prompted to confirm that you want to use the new Graph Viewer. After confirming in ECL Watch, restart the ECL IDE and your graphs will be displayed.
https://track.hpccsystems.com/browse/IDE-405
Blank row tag changes in HPCC Systems® 5.x
In versions prior to 5.0, specifying a blank row on an xml output using row('') would cause invalid xml to be generated containing blank row tags of the form <>.
In 5.0 this issue has been fixed and the empty row tag is suppressed. However, this has revealed an issues in the despray process whereby an xml file which has no row tag and has a header or footer specified cannot successfully be desprayed. A zero byte file is desprayed instead.
The simplest way to work rround this is by adding a dummy, non-blank row tag which can be removed by post processing after despraying
https://track.hpccsystems.com/browse/HPCC-13331
Some file service operations can leak a lock causing other superfile operations in the same process to deadlock (Fixed in 5.0.6)
If you use one of the Super File operations listed below with an irregular logical file name, (i.e. a filename with a wildcard in it, or a special qualifier like ~foreign:: ) then a lock to the subfile(s) of the super file involved will be leaked for the duration of the job. The effect of this is that if a subsequent superfile operation in the same job tries to manipulate the subfiles, (for example tried to add/remove/promote them in the superfile), the job will deadlock.
GetSuperFileSubCount
GetSuperFileSubName
FindSuperFileSubName
SuperFileContents
https://track.hpccsystems.com/browse/HPCC-12087
System error on deleteing logical files in 5.0.4-1 - (Fixed in 5.0.6)
In 5.0.4-1, deleting logical files either by overwriting them in ECL or by deleting them via EclWatch may cause a spurious timeout error:
SDS: Lock timeout Lock timeout trying to establish lock to Files/
There is no sensible workaround. This is fixed in 5.0.6 and we recommended that you upgrade at least to this release but preferably the latest release.
https://track.hpccsystems.com/browse/HPCC-12901
Using New ECL Watch does not require compatibility mode to be switch on in IE
Compatibility mode needed to be switched on when viewing legacy ECL Watch in order for graphs to display correctly. This is no longer required from release 5.x when using the new ECL Watch interface (now displayed by default). In fact, users have reported a significant performance slowdown when using new ECL Watch if compatibility mode is switched on. Since IE will remember your settings, you will need to go and switch this mode off using the icon supplied in the IE address bar:
Logical filenames are now validated consistently
In HPCC Systems® 5.x and later, checks for valid filenames are applied consistently for all places where filenames are used. In 4.2 and earlier, different parts of the code applied slightly different rules for what was considered a valid logical filename. The following characters are invalid in logical filenames:
*\"/<>?\|
Anything below ascii 32
Anything above ascii 127
The following rules are also enforced:
A : (colon) is permitted only as part of the :: scope separator
Only a single ~ (tilde) is allowed, and only at the start of the filename
Spaces are allowed (and ignored) either side of the :: but not elsewhere
There are additional rules concerning the use of ^ (for escaping character case to match physical filenames) and { } (for multi file expansion), see the Language Reference Manual for details.
Note: A ~ (tilde) other than at the start would have been accepted in some circumstances in 4.2.x, but will not be permitted in 5.0.x or later.
DeleteLogicalFiles in a PROJECT or APPLY requires NOTHOR
Thorslaves are prevented from having direct access to DALI which means that most FileServices calls cannot be used in a PROJECT/APPLY if they are executed on slaves. To avoid getting an error, wrap the PROJECT/APPLY with a NOTHOR to force the expressions/action to be executed in the ECL Agent context. For example instead of using the following code:
APPLY(ddd,STD.File.DeleteLogicalFile(ddd.name)); //this does not work
Use:
NOTHOR(APPLY(ddd,STD.File.DeleteLogicalFile(ddd.name))); // NOTHOR forces the expression to be evaluated outside (using ECL Agent)
This new approach applies to all releases since 3.10.8.
https://track.hpccsystems.com/browse/HPCC-12514
varstrings should not be used as key fields (Fixed in 5.2)
Varstrings are terminated with a character with value \0, and can cause performance issues if they are used as keyed fields in an index.
From 5.2 onward, an error is reported if they are used in this way. The suggested course of action is to create new indexes using fixed length fields.
https://track.hpccsystems.com/browse/HPCC-12528
File Monitor(File) problem causing events not to be sent (Fixed in 5.0.4)
Calls to File.MonitorFile() were ignoring the parameter that supplied the name of the event to send, and were typically using a blank event instead, causing the events to not be sent. This issue is a regression in 5.0 caused by a parameter name (eventName) clashing with a reserved identifier.
https://track.hpccsystems.com/browse/HPCC-12401
DFU file query in not completely case insensitive (Fixed in 5.0.4)
DFU Query fails if the cluster prefix portion of the filename does not match exactly including case. This can be worked around by adding a wildcard at the start of the string. This is a regression in 5.0 which is fixed in 5.0.4.
https://track.hpccsystems.com/browse/HPCC-12339
Warnings when using the VM with VM Player/Virtual Box
When using the VM with either VM Player or Virtual Box you may get the following message boxes:
In this case, click Retry to relax the specification and start the import. More information about this problem can be found in the following VMware article:
If you see the following warning, open the *.vmx shown in the message and modify the 'virtualhw.version' field based on the following information in this article by VMware:
https://track.hpccsystems.com/browse/HPCC-12179
ECL Plugin (Eclipse) users on Linux may get 'Warning failed to create web browser' error
On using ECL Watch from within Eclipse, the ECL Watch tab for individual workunit may not display producing a warning. You may also see the following reported in the error log:
Internal browser is not available: No more handles [Unknown Mozilla path (MOZILLA_FIVE_HOME not set)]
Installing libwebkitgtk and restarting Eclipse solves this issue.To install this dependency use this command:
sudo apt-get install libwebkitgtk-1.0-0
https://track.hpccsystems.com/browse/EPE-117
PDFs displaying poorly inside Chrome under Fedora 16
If you encounter this issue, please use the free Acrobat Reader or use Firefox to view our PDFs.
Graph Control may stop working after a Firefox upgrade on Linux
We have received reports that Graph Control may stop working with Firefox under Linux after an automatic Firefox upgrade. This problem can be resolved by either rebooting or manually removing and reinstalling the Graph Control.
TCP network settings that can help with slow or dropped connections between nodes on CentOS 5.x and 6.x
We have noticed that when a long running thor job goes into a largely local operation for a period of greater than 2h 11m, the job may fail at some point after that with “failed to connect to thorslave” or “jsocket thorslave:xxx". This is caused by how Thor and CentOS interact with keepalive packets. The workaround for this issue is to set the keepalive interval to a longer period of time, for example, 86400 seconds.
A second issue has also been identifed on Roxie. When Roxie returns particularly large datasets it starts throwing UDP send timeouts, retrying packet messages in the logs and query times run longer than expected. The workaround for this is to set the tcp_syn_cookies settings in proc to a value of 0. Other useful Roxie tuneables are as follows:
net.ipv4.tcp_syncookies = 0
net.core.rmem_default = 33554432
net.core.wmem_default = 33554432
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.udp_mem=2314176 3085568 4628352
net.ipv4.udp_rmem_min=33554432
net.ipv4.udp_wmem_min=33554432
net.ipv4.udp_mem=2314176 3085568 4628352
Windows PATH for Client Tools does not automatically update from 4.2
Since it is common for multiple versions of Client Tools to be installed we have intentionally stopped updating the path in Windows. As a result, you will now need to update the path manually to ensure that you are using the correct version. Alternatively, if you don't need to keep multiple versions of Client Tools installed, simply uninstall the old version to workaround this issue.
https://track.hpccsystems.com/browse/HPCC-11503
Changes made to MAXLENGTH may cause problems when indexes from HPCC Systems® 4.x and 5.x are combined in a superkey
The maximum record size for variable length keys now defaults to 32767. However:
If the record is of a fixed size, then the maximum will be the fixed size.
If ,MAXLENGTH(value) is specified on the INDEX or on the BUILD then <value> is the maximum size.
If MAXLENGTH - no argument - is specified on an INDEX or BUILD then the maximum size calculated from the record is used.
To make indexes created on an earlier version of HPCC Systems® compatible with HPCC Systems® 5.0 simply add ,MAXLENGTH onto the BUILD statement.
https://track.hpccsystems.com/browse/HPCC-10507
Side effects that precede a module definition tend to get lost
There is a long standing issue in the ECL code generator, where side-effects could be lost if they preceded a definition of a symbol that was a module. The original reason this was reported and found was that there were some #CONSTANTs that weren’t being processed, but it can also apply to OUTPUTs, #workunit directives and other actions.
This issue has now been fixed in 5.0. Instead of silently ignoring the side-effects it now generates an error, complaining that the side-effects will be lost. If you hit this when compiling your code on a 5.0 system, this error is produced:
warning C2395: Cannot associate a side effect with a module - action will be lost
You can convert the error to a warning by adding the following code into the query, but it is recommended that you implement a proper fix:
#ONWARNING(2395, warning);
https://track.hpccsystems.com/browse/HPCC-10596
In 5.0 we have provided a new feature allowing you to customize warning/error settings using the eclcc command line warning mapping options:
https://track.hpccsystems.com/browse/HPCC-569
Superfiles/superkeys are no longer locked in HPCC Systems® 5.0
In previous versions, if a query referenced a superfile/superkey in Dali, Roxie locked the superfile/superkey definition in Dali preventing users from modifying the contents. From 5.0 onwards, instead of locking superfiles and superkeys we subscribe to them so that if they change, any queries using them are automatically reloaded.
This issue did not affect superfiles and superkeys defined using package files.
https://track.hpccsystems.com/browse/HPCC-11253
Persists may fail to rebuild if files are updated mid-job
This issue is best described using the following example:
A job has several persists; persist 'p1' which is based on superfile X, a persist 'p2' which is based on other files, and a persist 'p3' which is based on p1 and p2.
Consider the flow of events as follows:
a) Day 1: workunit runs
p1 builds
p2 builds
p3 builds.
b) Day 2: workunit runs
p1 is up to date
p2 rebuilds - something has changed. While it is rebuilding the superfile X is updated!
p3 rebuilds because file X has changed
c) Day 6 workunit runs
p1 rebuilds because file X has changed
p2 is up to date
p3 is considered up to date - because file X hasn't changed since last rebuild.
Because the persist rebuild dependencies are now based on the raw input files, a persist rebuild won't be triggered by another persist rebuilding any more.
https://track.hpccsystems.com/browse/HPCC-11756
The default quote character for quoting fields within CSV files has changed from ‘ to “
This is more in line with industry standards and removes problems with names which include an apostrophe.
https://track.hpccsystems.com/browse/HPCC-10100
Stored strings with no explicit type now default to a variable length string
Previously, the length of the string was taken from the length of the default for the stored as in this example where myname was defined as a string5:
Myname :=’Gavin’ : stored(‘name’)
There are no compatibility concerns with this change.
https://track.hpccsystems.com/browse/HPCC-11464
thorlib.node() now returns consistent results across all cluster types
Previously, Roxie returned as result of 1, hthor returned 0 and thor returned 0-based values. Roxie now returns 0 which is consistent with hthor and a 1-way thor.
https://track.hpccsystems.com/browse/HPCC-11070
Significant New Features
New FileScopeAccess resource requires manual addition of a resource tag in ECL Watch on existing systems
This new resource has been added so that access to filescopes can be given to users who are not system administrators but need to carry out housekeeping tasks in this area. For new systems, the FileScopeAccess resource tag is created automatically. However, for existing systems it must be added manually. In the new ECL Watch users, groups and permissions are controlled from within the Operations area under the Security menu and you must be an administrator to enter this area. Go to the Permissions tab and select the Add menu item. Select ESP Features for SMC as the Type, enter the Name as FileScopesAccess, type a Description, then press ADD. The new resource is added to the ESP Features for SMC list (a refresh may be required). You can then edit the permissions for this resource for any user or group in the usual way.
This feature is only available on security enabled systems.
https://track.hpccsystems.com/browse/HPCC-11534
All pages in this wiki are subject to our site usage guidelines.