HPCC Systems 4.0.x Releases

Owned by Lorraine Chapman
Last updated: Feb 07, 2023 by Jim DeFabia
6 min read

Welcome to HPCC Systems release 4.0.0. Please take note of the following notes that may help you to avoid known problems and become familiar with useful work arounds when useing this release.  

DeleteLogicalFiles in a PROJECT or APPLY requires NOTHOR

Thor Worker nodes are prevented from having direct access to DALI which means that most FileServices calls cannot be used in a PROJECT/APPLY if they are executed on a worker. To avoid getting an error, wrap the PROJECT/APPLY with a NOTHOR to force the expressions/action to be executed in the ECL Agent context. For example instead of using the following code:

APPLY(ddd,STD.File.DeleteLogicalFile(ddd.name)); //this does not work

Use:

NOTHOR(APPLY(ddd,STD.File.DeleteLogicalFile(ddd.name))); // NOTHOR forces the expression to be evaluated outside (using ECL Agent)

This new approach applies to all releases since 3.10.8.

https://track.hpccsystems.com/browse/HPCC-12514

R plugin not reporting exceptions from R code effectively (fixed in 4.2)

Some errors would be caught (but without details of what the error was), while other errors would result in the process being terminated.

https://track.hpccsystems.com/browse/HPCC-10307

Embedded R fails if ^M characters are included in the source (fixed in 4.2)

When submitting queries from ECL IDE, errors may be reported from the R plugin and (on some, but not all distros) the Python plugin. This is caused by embedded carriage returns in the code being rejected by the R / Python interpreters. A fix was put into release 4.2.0 to address this. However, in earlier versions you can work around the issue by saving your source in an editor that does not put carriage returns on the end of each line prior to submitting, or by using the ECL playground for testing code snippets.

https://track.hpccsystems.com/browse/HPCC-10359

Incompatibility issues between HPCC Systems® VM and VMPLayer 4.0.6

The HPCC Systems® VM on VMPlayer 4.0.6 fails to start displaying an error message indicating incompatibility problems. To work-around, this problem, change the value "virtualhw.version" to 8.

The same HPCC Systems® VM image starts normally on VMPlayer 5.0.2.

However, please note that VMware is no longer supporting VMPlayer 4.x and as a result of this, we will no longer be supporting this version either.

For more information about VMPlayer support status look here: http://en.wikipedia.org/wiki/VMware_Player

For more information about the virtual hardware version look here: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003746

DeleteSuperFile possible crash (fixed in 4.2)

This was a regression introduced after 3.10.8. When DeleteSuperFile was added to the transaction system it caused DeletesuperFile with delete-sub-files=true to crash sometimes.

The nested removeSubFile call was causing the parent action (DeleteSuperFile) to be executed freeing up the superfile leaving the super object in removeSubFile in a bad state.

https://track.hpccsystems.com/browse/HPCC-10372

Issue using ClearSuperfile set to TRUE

In versions prior to 4.0, a bug in the implementation of ClearSuperfile (when passed TRUE to indicate that the subfiles should be deleted) meant that it would silently ignore any failure to delete a subfile. In version 4.0, if any subfiles cannot be deleted for this (or any other) reason, the operation will fail and an error will be reported.

Any code that was relying on the old bug to delete only files that were part of this subfile but not any other subfile, will now fail. We are looking into providing a new function that allows the delete option to ignore subfiles that are owned by multiple superfiles in 4.0.4

https://track.hpccsystems.com/browse/HPCC-10288

Logical file locking inconsistencies (fixed in 4.2)

Not all of the file reader activities in Thor keep the files locked for the duration that they are being read/used.

Consequently, it is be possible for someone to delete one of these files via EclWatch or another query, whilst the query is still in flight, potentially causing the query to fail with missing file part errors.

Javascript currently unsupported under Centos (fixed in 4.2)

The  RPM-based installation packages do not have the supporting plug-in for JavaScript support. This will be resolved in the next release (4.2). JavaScript support is present in the installation packages for other distros and in the VM.

https://track.hpccsystems.com/browse/HPCC-10198

Virtual Box network adaptors issue

When importing the 32-bit or 64-bit HPCC Systems® VM Image into Virtual Box on a Windows machine, you must open up the network settings before you start the VM due to an issue with network adapters (See their issue on thie here https://www.virtualbox.org/ticket/7067.)

After importing the Image, open the Settings for the Image.  Navigate to Network settings and select the tab for Adapter 2. Verify the following settings:

  •  Enable Network Adapter is checked

  •  Attached to: Host-only Adapter

  •  Name: VirtualBox Host-Only Ethernet Adapter

Make sure that the Name for Adapter 2 changes from 'vboxnet0' to 'VirtualBox Host-Only Ethernet Adapter'. Once confirmed, press the OK button and start the VM.

Upgrade issues under Centos

On rpm-based systems, if you have any version of HPCC Systems® Platform prior to 4.0.0-rc10 installed, you must first manually uninstall (using rpm -e) before attempting to upgrade.

https://track.hpccsystems.com/browse/HPCC-10081

Roxie requires all packages to have unique names (fixed in 4.0.2)

For environments with more than one roxie target defined there were issues when publishing the same packagemap to more than one target.  The issues could only be avoided by manually using unique names for the Packagemap for each target the packagemap was published to. 

Otherwise there would be issues with activating the packagmap and any changes to the packagemap that were intended for the current target would affect all targets immediately.

From 4.0.2 by default when a packagemap is added to a particular target it is considered a unique entry with a scope limited to that target.

Users who understand the implications can still force the same instance of a packagemap to be shared across targets using the --global-scope option on the command line.

https://track.hpccsystems.com/browse/HPCC-9869

Java embed not working on Centos 5 (fixed in 4.0.0-9)

On some distros (notably Centos5 but others may be affected), ECL code that tries to use the new embedded java feature may encounter an error at runtime similar to the following:

Eclagent 0: System error: 0: Error loading /mnt/disk1/var/lib/HPCCSystems/myeclccserver/libW20130724-082534.so:
libmawt.so: cannot open shared object file: No such file or directory

This was due to an additional unnecessary dependency in the javaembed plugin, that has now been removed.

https://track.hpccsystems.com/browse/HPCC-9769

Superfile cannot be deleted if a subfile associated with it has already been deleted  (fixed in 4.0.0-7)

Although this was found in 4.0.0-6, this issue was a regression from 3.10.8. The consequence was that a user could delete a subfile without error that belonged to a superfile, causing the superfile to become unreadable. Operations can fix the superfile by using:

daliadmin <daliip> checksuperfile <superfilename> fix=true

Checks have been added in 4.0.0-7 to make sure that a file cannot be deleted if is associated with a superfile. An error message is displayed if a user attempts to do this.

https://track.hpccsystems.com/browse/HPCC-9802

Package watcher notification/reload deadlock  (fixed in 4.0.0-7)

Multiple CDaliPackageWatcher's could be notified in parallel on different threads. Each invoked a reload of all packages in the manager. This could result in deadlock if two threads both tried to unregister a watcher that was already in its notify method.

https://track.hpccsystems.com/browse/HPCC-9828

ConfigMgr crash when trying to delete a single node ‘null’ roxie server  (fixed in 4.0.0-6)

When adding a new component (roxie) to an environment that did not previously contain a Roxie, the Roxie Server page showed the label as ‘null’ rather than ‘farm1’. The crash happened on trying the delete this ‘null’ roxie using the right click options.

https://track.hpccsystems.com/browse/HPCC-9771  

Roxie needs to figure out numChannels (fixed in 4.0.0-6)

Some hidden attributes that were normally calculated by ConfigMgr were not in the xsd or were marked as readonly. This meant that ConfigMgr was not able to calculate them making it very hard to set up a new Roxie cluster.

https://track.hpccsystems.com/browse/HPCC-9795

Workunit output limits left unchecked (fixed in 4.0.0-5)

This regression prevented the limits on workunit results being checked. Normally an excessively large (default 10MB) workunit result, produces an error and large results should be committed to disk.

The consequence can be quite serious. If a very large workunit result is created, it has the potential to overwhelm Dali and cause it to run out of memory.

***Workunits created with this regression should be deleted.***

https://track.hpccsystems.com/browse/HPCC-9784

struct ICodeContext has no member named GetDaliServers (fixed in 4.0.0-4)

When trying to compile/run a query, users would get a compile error complaining that the function GetDaliServers was not defined. GetDaliServers() has been added back into the ICodeContext.

https://track.hpccsystems.com/browse/HPCC-9745

Failure to check crcResources for files could be fatal (fixed in 4.0.0-4)

This issue caused all physical files referenced by a Roxie query for the first time, to be read in their entirety in order to calculate and check against the published crc values. The check was restored and changed to make sure that the check is always enforced.  

https://track.hpccsystems.com/browse/HPCC-9839

ECL Watch Technical Preview not working with Internet Explorer (fixed in 4.0.0-2)

This was due to a missing template file. It also affected users accessing ECL Watch via the ECL Plugin for Eclipse.

https://track.hpccsystems.com/browse/HPCC-9729



The following note relates to HPCC Systems 4.0.2 only:

Potential crash due to race condition in multicore join helper (fixed in 4.0.2-2)

This issue effects 4.0.2-1 only.

An ECL JOIN clause using, UNORDERED or, HINT(parallel_match) could cause Thor to crash as the JOIN starts.

https://track.hpccsystems.com/browse/HPCC-10010





All pages in this wiki are subject to our site usage guidelines.