Assignment of thor slaves based on environment.xml
Description
The current system for starting a thor relies on ssh and rsync. We would like to eventually be able to move towards just launching a thorslave on an individual node without needing an ssh connection of any type. To do this though we will need to know the individual slavenumber and port for each slave that is assigned to a cluster. This can be done at configuration time and embedded in the environment xml, and then accessed by a configgen -listall2 call.
Currently the output from a configgen call with "-listall2 -ip localhost -t thor" will yeild: ThorMasterProcess,mythor,10.20.8.37,20000,/var/lib/HPCCSystems/mythor, ThorSlaveProcess,mythor,10.20.8.37,,/var/lib/HPCCSystems/mythor, If the element after the IP could return the slaveport, and if we could possibly insert a column into the record to output the slavenum, that would be extremly helpful.
I could then parse that from configgen directly instead of doing the math for port and slavenumber allocation in our thorslave scripts.
"20100" is the slave port. And "1" is slaves per node.
Ken Rowland July 24, 2018 at 1:20 PM
As discussed
Jacob Cobbett-Smith November 30, 2017 at 5:12 PM
There are some routines in DFS that get a group for a named cluster e.g. getClusterNodeGroup(). I'm not sure if there's a method that returns a group with ports, but one could be added and used by configgen.
Lorraine Chapman November 30, 2017 at 4:54 PM
Edited
- What needs to be done to effect this change?
Michael Gardner December 5, 2016 at 4:59 PM
Well when I first proposed this, I was expecting Gleb to handle it. It was before we knew he was leaving. I can definitely look into it though and try to make the changes. Would be useful getting my hands on the configgen code. Always good to have another person familiar with that part of the code base just in case something goes wrong down the road.
Fixed
Pinned fields
Click on the next to a field label to start pinning.
The current system for starting a thor relies on ssh and rsync. We would like to eventually be able to move towards just launching a thorslave on an individual node without needing an ssh connection of any type. To do this though we will need to know the individual slavenumber and port for each slave that is assigned to a cluster. This can be done at configuration time and embedded in the environment xml, and then accessed by a configgen -listall2 call.
Currently the output from a configgen call with "-listall2 -ip localhost -t thor" will yeild:
ThorMasterProcess,mythor,10.20.8.37,20000,/var/lib/HPCCSystems/mythor,
ThorSlaveProcess,mythor,10.20.8.37,,/var/lib/HPCCSystems/mythor,
If the element after the IP could return the slaveport, and if we could possibly insert a column into the record to output the slavenum, that would be extremly helpful.
I could then parse that from configgen directly instead of doing the math for port and slavenumber allocation in our thorslave scripts.