DAFILESRV : port in use
Environment
Description
Conclusion
Activity
Michael Gardner August 25, 2016 at 2:20 PM
So I think part of the issue might be related to the fact that we don't have a status check for starting dafilesrv. So calls to start dafilesrv can lead to multiple init_dafilesrv instances. With each of these attempting to start dafilesrv, we're going to see port conflicts.
Mark Kelly July 19, 2016 at 5:46 PM
ok, perhaps it should try 3 times and then give up and when trying it can use netstat to see if any process has the port in use to know what to kill.
Adding in for any additional comments.
Eduardo Orochena July 18, 2016 at 1:22 PM
Hard to tell, going by the logs it looks like the application try to issue a restart, when it gets in this state the only way to restart the application is to kill the init_dafilesrv processes
00000000 2016-07-15 01:35:36.934 112608 112608 "Dafilesrv starting - Build internal_6.0.2-rc4"
00000001 2016-07-15 01:35:36.934 112608 112608 "Parallel request limit = 50, throttleDelayMs = 2000, throttleCPULimit = 90"
00000002 2016-07-15 01:35:36.935 112608 112608 "Opening Dali File Server on 7100"
00000003 2016-07-15 01:35:36.935 112608 112608 "Version: DS V1.9Linux "
00000004 2016-07-15 01:35:36.935 112608 112608 "Authentication: not required"
00000005 2016-07-15 01:35:36.935 112608 112608 "Throttler(stdCmdThrotlter): Increasing limit from 0 to 80"
00000006 2016-07-15 01:35:36.935 112608 112608 "Throttler(slowCmdThrotlter): Increasing limit from 0 to 20"
00000007 2016-07-15 01:35:36.935 112608 112608 "Throttler(slowCmdThrotlter): New delayMs=5000, previous: 1000"
00000008 2016-07-15 01:35:36.935 112608 112608 "Throttler(slowCmdThrotlter): New cpuThreshold=75, previous: 85"
00000009 2016-07-15 01:35:36.935 112608 112608 "CRemoteFileServer: maxThreads = 100, maxThreadsDelayMs = 60000, maxAsyncCopy = 10"
0000000A 2016-07-15 01:35:36.935 112608 112608 "Throttler(stdCmdThrotlter): Reducing limit from 80 to 50"
0000000B 2016-07-15 01:35:36.935 112608 112608 "Throttler(stdCmdThrotlter): New delayMs=2000, previous: 1000"
0000000C 2016-07-15 01:35:36.935 112608 112608 "Throttler(stdCmdThrotlter): New cpuThreshold=90, previous: 85"
0000000D 2016-07-15 01:35:36.935 112608 112608 "Creating sentinel file dafilesrv.sentinel for rerun from script"
0000000E 2016-07-15 01:35:36.936 112608 112608 "ERROR: -7: /var/lib/jenkins2/workspace/LN-Candidate-with-Plugins-6.0.2-rc4/LN/centos-6.0-x86_64/HPCC-Platform/dali/dafilesrv/dafilesrv.cpp(700) : DAFILESRV : port in use
Target: S>10.241.12.172, port = 7100, Raised in: /var/lib/jenkins2/workspace/LN-Candidate-with-Plugins-6.0.2-rc4/LN/centos-6.0-x86_64/HPCC-Platform/system/jlib/jsocket.cpp, line 929"
0000000F 2016-07-15 01:35:36.936 112608 112608 "CRemoteFileServer::stop"
Mark Kelly July 15, 2016 at 7:08 PM
What is the sequence of events that led to this issue ?
The only way to restarted again is to kill the init_dafilesrv processes
Target: S>10.241.12.172, port = 7100, Raised in: /var/lib/jenkins2/workspace/LN-Candidate-with-Plugins-6.0.2-rc4/LN/centos-6.0-x86_64/HPCC-Platform/system/jlib/jsocket.cpp, line 929"
00000010 2016-07-15 09:36:04.062 113982 113982 "CRemoteFileServer::stop"
00000011 2016-07-15 09:36:04.062 113982 113982 "Stopped Dali File Server"
00000001 2016-07-15 09:36:05.399 113988 113988 "Dafilesrv starting - Build internal_6.0.2-rc4"
00000002 2016-07-15 09:36:05.399 113988 113988 "Parallel request limit = 50, throttleDelayMs = 2000, throttleCPULimit = 90"
00000003 2016-07-15 09:36:05.399 113988 113988 "Opening Dali File Server on 7100"
00000004 2016-07-15 09:36:05.399 113988 113988 "Version: DS V1.9Linux "
00000005 2016-07-15 09:36:05.399 113988 113988 "Authentication: not required"
00000006 2016-07-15 09:36:05.399 113988 113988 "Throttler(stdCmdThrotlter): Increasing limit from 0 to 80"
00000007 2016-07-15 09:36:05.399 113988 113988 "Throttler(slowCmdThrotlter): Increasing limit from 0 to 20"
00000008 2016-07-15 09:36:05.399 113988 113988 "Throttler(slowCmdThrotlter): New delayMs=5000, previous: 1000"
00000009 2016-07-15 09:36:05.399 113988 113988 "Throttler(slowCmdThrotlter): New cpuThreshold=75, previous: 85"
0000000A 2016-07-15 09:36:05.399 113988 113988 "CRemoteFileServer: maxThreads = 100, maxThreadsDelayMs = 60000, maxAsyncCopy = 10"
0000000B 2016-07-15 09:36:05.399 113988 113988 "Throttler(stdCmdThrotlter): Reducing limit from 80 to 50"
0000000C 2016-07-15 09:36:05.399 113988 113988 "Throttler(stdCmdThrotlter): New delayMs=2000, previous: 1000"
0000000D 2016-07-15 09:36:05.399 113988 113988 "Throttler(stdCmdThrotlter): New cpuThreshold=90, previous: 85"
0000000E 2016-07-15 09:36:05.399 113988 113988 "Creating sentinel file dafilesrv.sentinel for rerun from script"
0000000F 2016-07-15 09:36:05.400 113988 113988 "ERROR: -7: /var/lib/jenkins2/workspace/LN-Candidate-with-Plugins-6.0.2-rc4/LN/centos-6.0-x86_64/HPCC-Platform/dali/dafilesrv/dafilesrv.cpp(700) : DAFILESRV : port in use
Target: S>10.241.12.172, port = 7100, Raised in: /var/lib/jenkins2/workspace/LN-Candidate-with-Plugins-6.0.2-rc4/LN/centos-6.0-x86_64/HPCC-Platform/system/jlib/jsocket.cpp, line 929"
[root@boca_dataland_dev:node010241012172 mydafilesrv]$ ps ax|grep daf
59318 ? S 0:02 /bin/bash /opt/HPCCSystems/bin/init_dafilesrv
65641 ? S 0:00 /bin/bash /opt/HPCCSystems/bin/init_dafilesrv
65702 ? Sl 12:53 dafilesrv -L /var/log/HPCCSystems -I mydafilesrv
75544 ? S 0:00 /bin/bash /opt/HPCCSystems/bin/init_dafilesrv
112792 ? S 0:08 /bin/bash /opt/HPCCSystems/bin/init_dafilesrv
116449 pts/0 S+ 0:00 grep daf