Potential deadlock if roxiemem callback is active when shared stream requested
Description
Conclusion
None
Activity
Show:
Jacob Cobbett-Smith November 7, 2018 at 1:40 PM
Yes, that's because a thread can be stuck due to this issue, whilst collecting stats in preparation to send a watchdog packet.
So the watchdog packets never sent and the master thinks the slave(s) are down after a while.
Mark Kelly November 7, 2018 at 1:08 PMEdited
I will add that when deadlock occurs the Thor error msg is often:
Watchdog has lost contact with Thor slave: <IP:port> (Process terminated or node down?)And the job is aborted and Thor restarts
Mark Kelly August 20, 2018 at 7:16 PM
Some more detailed info:
One thread is in:
BackgroundReleaseBufferThread::run->runReleaseBufferThread->releaseBuffersNow->doReleaseBuffers->callbackReleasesRows->CThorRowCollectorBase::freeBufferedRows
holding the CriticalBlock block(callbackCrit) (from releaseBuffersNow)
and is blocked trying to get CThorArrayLockBlock block(spillableRows)
Another thread is in:
getStream->CSharedSpillableRowSet->activateSpillingCallback->ensureSpillingCallbackInstalled->addRowBuffer->addRowBuffer
holding CThorArrayLockBlock block(spillableRows) (from getStream)
and is blocked trying to get CriticalBlock block(callbackCrit)
Jacob Cobbett-Smith August 20, 2018 at 5:08 PM
Some stacks from an instance of this where deadlock seen:
Thread 53 (Thread 0x7f47314b7700 (LWP 1911350)):
#2 0x00007f47364eed2f in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#3 0x00007f473c9f2aa7 in CThorArrayLockBlock (_alock=..., this=<synthetic pointer>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/thorutil/thmem.hpp:244
#4 CThorRowCollectorBase::
freeBufferedRows (this=0x7f452c0d0790, critical=<optimized out>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/thorutil/thmem.cpp:1937
#5 0x00007f4737beb032 in callbackReleasesRows (critical=false, callback=0x7f452c0d0790, this=0xe0c7b0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3640
#6 releaseRows (critical=false, minSuccess=1, whichSlave=0, manager=..., this=0x7f452c00a4b0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3513
#7 roxiemem::BufferedRowCallbackManager::doReleaseBuffers (this=this@entry=0xe0c7b0, slaveId=slaveId@entry=0, maxSpillCost=maxSpillCost@entry=4294967295, critical=critical@entry=false, minSuccess=minSuccess@entry=1) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3758
#8 0x00007f4737beb260 in releaseBuffersNow (slaveId=0, critical=false, checkSequence=false, prevReleaseSeq=0, maxSpillCost=4294967295, this=0xe0c7b0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3778
#9 runReleaseBufferThread (this=0xe0c7b0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3674
#10 roxiemem::BufferedRowCallbackManager::BackgroundReleaseBufferThread::run (this=<optimized out>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3391Thread 36 (Thread 0x7f451ffff700 (LWP 1911501)):
#2 0x00007f47364eed2f in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#3 0x00007f4737bebb35 in enter (this=0xe0c7b8) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/./../../system/jlib/jmutex.hpp:300
#4 CriticalBlock (c=..., this=<synthetic pointer>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/./../../system/jlib/jmutex.hpp:341
#5 addRowBuffer (callback=<optimized out>, slaveId=0, this=0xe0c7b0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:3579
#6 roxiemem::CCallbackRowManager::addRowBuffer (this=0xe0c650, callback=0x7f4530043ba0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/roxie/roxiemem/roxiemem.cpp:4875
#7 0x00007f473c9f456e in ensureSpillingCallbackInstalled (this=0x7f4530043ba0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/thorutil/thmem.cpp:211
#8 activateSpillingCallback (this=0x7f4530043ba0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/thorutil/thmem.cpp:192
#9 CSharedSpillableRowSet (_spillPriority=<optimized out>, _emptyRowSemantics=<optimized out>, _rowIf=<optimized out>, inRows=..., _activity=..., this=0x7f4530043ba0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/thorutil/thmem.cpp:392
#10 CThorRowCollectorBase::getStream (this=0x7f4530003890, allMemRows=<optimized out>, memUsage=<optimized out>, shared=<optimized out>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/thorutil/thmem.cpp:1737
#11 0x00007f473ca09f29 in CThorGraphResult::getLinkedRowResult (this=0x7f4530003320) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/graph/thgraph.cpp:197
#12 0x00007f473d08d66d in CThorGraphResults::getLinkedRowResult (this=<optimized out>, id=<optimized out>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/graph/./../graph/thgraph.hpp:1241
#13 0x00007f455522ab59 in cAc97::transform(ARowBuilder&, void const*, unsigned int) () from /var/lib/HPCCSystems/queries/thor400_sta01_20100/V1414386810_libW20180813-151107.so
#14 0x00007f473d407730 in nextRowNoCatch (this=0xe5d910) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/activities/normalize/thnormalizeslave.cpp:80
#15 NormalizeSlaveActivity::nextRow (this=0xe5d910) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/activities/normalize/thnormalizeslave.cpp:58
#16 0x00007f473d374f95 in ungroupedNextRow (this=0xe5da58) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/activities/./../../system/jlib/jio.hpp:158
#17 CParallelFunnel::CInputHandler::main (this=0x7f4528002ad0) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/thorlcr/activities/funnel/thfunnelslave.cpp:84
#18 0x00007f4737863d3d in CThreadedPersistent::main (this=0x7f4528002ae8) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jthread.cpp:570
#19 0x00007f4737922590 in CThreadedPersistent::CAThread::run (this=<optimized out>) at /mnt/disk1/jenkins/workspace/LN-Candidate-with-Plugins-6.4.24-1relwithdeb/LN/centos-7.0-x86_64/HPCC-Platform/system/jlib/jthread.hpp:179Fixed
Pinned fields
Click on the next to a field label to start pinning.
Details
Components
Assignee
Jacob Cobbett-SmithJacob Cobbett-SmithReporter
Jacob Cobbett-SmithJacob Cobbett-SmithPriority
MajorFix versions
Details
Details
Components
Assignee
Jacob Cobbett-Smith
Jacob Cobbett-SmithReporter
Jacob Cobbett-Smith
Jacob Cobbett-SmithPriority
MajorFix versions
Created August 20, 2018 at 4:57 PM
Updated November 7, 2018 at 1:45 PM
Resolved August 24, 2018 at 2:51 PM
If the roxiemem background OOM callback is active, calling a spillable row collector, and at the same time that row collector is being called to get its output stream and used by multiple outputs (e.g. because row collector is being used by a graph result), then deadlock can occur.
In this situation, the CThorRowCollectorBase::getStream locks the spillableRows mutex, and on the 1st call setups a shared spillable row set container for any remaining rows in memory. In doing to it calls back into roxiemem to setup a new callback.
If roxiemem is already blocked on the spillableRows mutex via it's background thread, then deadlock ensues.
The fix is likely to be to ensure that the new shared spillable row set object, only installs the new callback once it's released the row collector's spillableRows mutex.