DropZone file checks should be moved, so that not hit by remote logical file copy.

Activity

Show:

Jacob Cobbett-Smith May 4, 2017 at 11:37 AM

Currently, the checks for DZ is in a generic routine inside the dfu engine (called CDistributedFileSystem::import), which is called during a remote copy and when spraying.
That routine checks every path in a file descriptor, meaning it currently checks to see if each source ip:path in a remote foreign copy is a DZ.
And would currently cause failure if DZ restrictions were turned on.

These checks should not be called for a remote copy, so the code involved should be moved so that it called where appropriate (e.g. during a spray).

It was noticed that the DFU engine already calls a routine to check if there are any LDAP restrictions for each specific ip:path for a spray or despray.
So I think the DZ check should similarly be done at that time.

In the remote copy case, the DFU engine looks up a file using DFS, passing in a IUserDescriptor.
So remote foreign files are in fact already validated (*needs to be verified by testing*)

In conclusion, it looks like the thor-thor (remote copy) case was already handled.

The existing DZ checks need moving into the engine, so that the copy cases does not hit them.

Richard Chapman April 26, 2017 at 8:55 AM

Need to test these copies to make sure that these still work with dropzone restrictions in place (and that LDAP rules are applied).

Jacob Cobbett-Smith February 17, 2017 at 2:52 PM

Examining the code, all foreign files perform a scope check authentication locally and then remotely on the foreign Dali.

e.g. ~foreign::10.50.20.1::s1::s2::blah

Will :
1) LDAP validate scope "foreign::10.50.20.1::s1::s2" locally (if LDAP enabled locally).
2) send request for file lookup of "s1::s2::blah" to Dali on 10.50.20.1
3) Dali @ 10.50.20.1, will validate "s1::s2" scope on LDAP (if LDAP enabled)

Meaning, that a local environment can lock down access to prevent it, for example, having any access to foreign files etc.
And any foreign lookup will impose it's restrictions on the lookup.

Jacob Cobbett-Smith January 24, 2017 at 6:59 PM

added a comment - an hour ago
I feel I should clarify and/or separate two things:
1. Drop zone restriction applies to spray/ (remote) copy (import) and despray (export) operations only where the source or target can't be out-of-system (out of drop zone) path.
2. Access to external logical files like:'~file::<ip_address>::<filepath>' doesn't use drop zone (source/target) path restriction.
So, if we have a problem and it matches to 1 and the suggested server white list solution can make our/dev/ops life easier without compromise the security then I think it will be fine. However it assumes all white listed servers has same drop zone (directory) structure as the local.
But if our problem fall into 2 then I think the problem belongs to another area like user permission etc.
, what do you think?

I think it's reasonable for '~file::<ip_address>::<filepath>' to be treated in the same way the DropZone's are, because it can point anywhere and should only be used to point to external sources (external to HPCC logical file area).

However, the case we rolled back strict DropZone checking for this time, was a DFU file copy of a logical file in a foreign environment.
Which is akin to reading a foreign file in ECL like this:

ds := DATASET('~foreign::<foreign_dali>::foreignscope1::foreignscope2::foreignfilename', rec); OUTPUT(ds);

I am not sure how the security of this is treated - does it perform a LDAP authentication to the foreign Dali?
But it seems to me that when DFU is copying a file and has been given the foreign Dali IP and the logical filename, it is exactly the same as (from a security aspect) as the piece of ECL above. It is not dealing with spraying or despraying from or too a DZ, but it is reading a foreign logical file and should apply the same security mechanism as an ECL query would when reading a foreign remote file

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Fix versions

Labels

Affects versions

Created January 19, 2017 at 5:35 PM
Updated December 18, 2017 at 7:23 PM
Resolved May 5, 2017 at 3:14 PM

Flag notifications