HPCC Systems 6.4.x Releases

Welcome to the Red Book for HPCC Systems^® 6.4.x. There are several sections in this Red Book as follows:

General HPCC Systems Core Platform
Significant New Features Core Platform
ECL IDE
MySQL

This is a minor release including some new features and performance enhancements.

Users upgrading from releases prior to HPCC Systems 6.x.x may benefit from glancing at the Red Book entries from previous releases.

Here's how to contact us if you find an issue or want to add something to the Red Book:

To raise an issue, use our Community Issue Tracker. Please create yourself an account if you don't already have one, to get automatic updates as your issue progresses through the workflow.
To ask a developer a technical question about something you are doing or have encountered, post in the Developer Forum.

General HPCC Systems Core Platform

NEW - Antivirus software may report false positives for NSIS created install packages

We use the NSIS windows installation framework for installing and uninstalling HPCC Systems Client Tools and Graph Control. Users have reported that these executables are being flagged as a security risk by their antivirus software. This is a known issue and more information is available on the NSIS sourceforge wiki: http://nsis.sourceforge.net/NSIS_False_Positives. Using the links provided on the NSIS wiki, you can report the false positive and get it white listed for the antivirus software you are using.

NEW - Chrome 67.x loading issue affects the display of ECL Watch

This is true for all versions of the HPCC Systems Platform prior to 6.4.20 and we recommend you upgrade to avoid the issue. The Chrome upgrade revealed an issue in one of the third party libraries that ECL Watch uses. The fix includes a newer version of that library to resolve the problem.

https://track.hpccsystems.com/browse/HPCC-19818

6.4.6 requires the updating of all plugins

When updating the HPCC platform to 6.4.6, you must also update any plugins (such as WsSQL, Ganglia, or Nagios).

This is as a result of the following change made in the platform:

https://track.hpccsystems.com/browse/HPCC-18504

Python2 and Python3 plugins cannot be enabled at the same time

The selection about which of these plugins is enabled is made in the environment.conf file. If you experience a link error of the form "ld: cannot find –lpy?embed” then the probable cause is either that you have not enabled the appropriate python plugin in the environment.conf file, or you are attempting to use the wrong Python plugin.

Please note the following:

New installations default to python2 support.
A cluster that is upgraded from an older version may not have the required lines (see 3 and 4 below) in the environment.conf file to support either version.
If you attempt to use the python2 plugin (IMPORT Python) on a system that does not contain the line "additionalPlugins=python2” in the environment.conf file, you will get an error "ld: cannot find –lpy2embed”
If you attempt to use the python3 plugin (IMPORT Python3) on a system that does not contain the line "additionalPlugins=python3” in the environment.conf file, you will get an error "ld: cannot find –lpy3embed”

https://track.hpccsystems.com/browse/HPCC-17199

Changes in Dropzone configuration in HPCC Systems 6.4.0

Prior to 6.4.0, dropzone configurations contained an attribute called “Computer”, however, in 6.4.x configmngr does not display this attribute. While you are not required to update a legacy dropzone configuration, (the system is backwards compatible), you may manually update a dropzone using configmanager by viewing the dropzone entry in “XML edit view” and deleting the “Computer” attribute. You may then navigate to the dropzone in the tree view to add the appropriate IP address/hostname in the Server list tab.

https://track.hpccsystems.com/browse/HPCC-17738

You can now supply a list of preferred dropzone destinations but keep some destinations hidden from users using the ECLWatchVisible attribute in the HPCC Systems Configuration Manager. When set to true, a specified landing zone will be shown in the dropdown list of available dropzones and ECL Watch. When set to false, the drop zone is hidden. This may be useful in situations where you want to maintain access to a number of dropzones for your users but protect some locations from being used as general drop zone locations. You can also add notes for other administrators to keep a record of why your dropzones are configured the way they are. More information about dropzone configuration settings can be found in the HPCC Systems Configuration Guide.

https://track.hpccsystems.com/browse/HPCC-17084

Inconsistent #stored/#constants are now caught and reported as errors

A warning (defaulting to error severity) has been added in 6.4.0 to catch these inconsistencies, which means that some queries will need fixing when migrated to 6.4.0. The correct fix is to only have a single #STORED or #CONSTANT for a single id.

A short term work around is to add:

^{#onwarning(4207, warning);}

to the query. However, while your query may still work as before, there is no guarantee which of the inconsistent values it will be using so it is recommended that you implement the correct fix as soon as possible.

https://track.hpccsystems.com/browse/HPCC-16684

New supported/retired distributions in HPCC Systems 6.4.0

We now support Ubuntu 17.04. The following distributions are no longer supported:

Precise 12/04
CENTOS 5

For a full list of supported distributions and how to build the HPCC Systems platform from the sources for these distributions, visit the HPCC Systems Developer Wiki on GitHub.

https://track.hpccsystems.com/browse/HPCC-17196

Using a single colon in logical files names generates incorrect filenames which can't be renamed in ECL Watch/ECLIDE

The recommended way to reference the path of a logical file is to use a double colon following example:

^{thor_cluster::parent_folder::child_folder::logical_filename}

A case was reported to us recently where a user (either by design or by accident) had included a single colon in the path as shown in this example:

^{thor_cluster::parent_folder::child_folder:logical_filename}

On writing to the file system, this was interpreted as:

^{/var/lib/HPCCSystems/thor/thor_cluster/parent_folder/child_folder%3Alogical_filename_n_of_n}

Once this has happened, it is not possible to rename the file correctly using ECL Watch or ECL IDE. Users who find themselves in this situation may use the following workaround to manually rename the file(s) on the file system to %3A using this bash command:

^{for BADNAME in /var/lib/HPCCSystems/thor/thor_cluster/parent_folder/child_folder%3Alogical_filename* ; do mv $BADNAME ${BADNAME/\%3A/\%3a}; done}

This makes it possible to rename the file(s) in ECL Watch or ECL IDE in the usual way.

https://track.hpccsystems.com/browse/HPCC-17511

Simplified installation process for the Rembed plugin package

The Rembed plugin package now includes RInside to prevent installation failure and obscure error messages on systems previously using an incompatible version of RInside. More information about installing the Rembed plugin can be found in the Embedded Languages section of the Installing and Running the HPCC Systems guide.

https://track.hpccsystems.com/browse/HPCC-17340

The default behavior for the R plugin has changed

Previously, embedded R code used persistent, shared environment for R execution. From 6.4.0, each embedded R function will be run in a local environment with options to persist/share this environment between embed calls in the same way as for Python. This improvement resolves issues experienced when using embedded R on Roxie/Thor where the R environment from other queries would still be loaded. More information about the options available can be found in the Embedded Languages section of the Installing and Running the HPCC Systems guide.

https://track.hpccsystems.com/browse/HPCC-17280

Changes to denormalize to correctly preserve values from the left record if there are no matches

In versions prior to HPCC Systems 6.4.0, any constants assignments to fields inside the transform for the denormalize would be propagated into any following operations. The was incorrect behaviour. If there are no matching records then the left row is passed as-is to the output.

From HPCC Systems 6.4.0, values are correctly preserved from the left record if there are no matches.

https://track.hpccsystems.com/browse/HPCC-17505

TBB (Thread Building Blocks) libraries are now included as a GitHub sub module in our source code

Previously, we used a Linux vendor distribution version of the TBB libraries which did not always provide the most current version. Since older versions of the TBB libraries can core with more recent versions of HPCC System, we have decided to include the libraries as a sub-module in our source code, to ensure that the most current version is always used.

https://track.hpccsystems.com/browse/HPCC-17228

Couchbase plugin xpath parse error when processing dataset (fixed in HPCC Systems 6.4.2)

Parsing the 'schedule' child dataset when there isn't an explicit XPATH defined will result in the following error:

^{Error: IPropertyTree: xpath parse error}
^{XPath Exception: Qualifier expected e.g. [..]}
^{in xpath = ?Row}
^{^ (0, 0), 1,}

https://track.hpccsystems.com/browse/HPCC-18006

64-bit Client Tools installation location

HPCC Systems users who are accustomed to locating Client Tools in the C:\Program Files\HPCCSystems\version_no\clienttools folder, should note that while this is the correct location for the 32-bit version, the 64-bit version will be located in C:\Program Files (x86)\HPCCSystems\version_no\clienttools folder.

https://track.hpccsystems.com/browse/HPCC-17659

Incorrect code generation on some aggregation operations on inline datasets

Previously, when an operation on an inline dataset (e.g. a filter) included a min/max (or other aggregate operation) of a filter/limit of the same dataset, the resulting values from the inline dataset were being incorrectly substituted into the min/max operation.

E.g. catch3 which has the following structure:

^{ds := inline-dataset
child := limit(ds, 4); (a filter might also trigger it)
m := max(child, some-field);
output(ds(some-other-field != m));}

This was incorrect behaviour which has now been corrected. Be aware that the behaviour of some existing queries that were relying on this incorrect behaviour, may now change.

https://track.hpccsystems.com/browse/HPCC-17652

(string)(string100)string50value no longer converted to string50value

Previously (string)(string100)string50field would have removed both casts. If the value was then being compared against another string, it would not cause any problems because trailing spaces are ignored. However if for some reason the untrimmed length was being used, that length will now change.

Previously,

^{LENGTH((string)(string100)string50field)}

would return 50, now it will correctly return 100.

https://track.hpccsystems.com/browse/HPCC-17263

LOOKUP attribute available in HPCC Systems 6.4.0

While the layout of a file is specified in the ECL Code, if the file on disk is using a different layout, version control problems can be caused. This feature allows file resolution to be done at compile time. Two language constructs have been added which are associated with this new feature.

A DATASET declaration can be given a lookup attribute to indicate that the filename should be looked up at compile time.
LOOKUP can also be used in the RECORDOF function taking a filename rather than a dataset. The result will be expanded at compile time to the record layout stored in the named file's metadata.

More details are available about this new feature and how to use it in our blog post File layout resolution at compile time.

https://track.hpccsystems.com/browse/HPCC-17133