Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

esp cored during packagemap list

Description

I'm not sure of the exact circumstances of recreating this error, but here's the excerpt from the esp logs on the 160:

 

000001C1 PRG 2020-05-26 16:56:21.005 28392 29397 "TxSummary[activeReqs=1;auth=Ok;contLen=102;rcv=0ms;handleHttp=4ms;user=mvazquez@10.145.0.44;req=POST wsstore.FETCH v1.0;total=4ms;]"
000001C2 PRG 2020-05-26 16:56:21.090 28392 29402 "HTTP First Line: POST /WsPackageProcess/GetPackageMapSelectOptions.json HTTP/1.1"
000001C3 PRG 2020-05-26 16:56:21.090 28392 29402 "POST /WsPackageProcess/GetPackageMapSelectOptions.json, from 10.145.0.44"
000001C4 PRG 2020-05-26 16:56:21.090 28392 29400 "HTTP First Line: POST /WsPackageProcess/GetPackageMapSelectOptions.json HTTP/1.1"
000001C5 PRG 2020-05-26 16:56:21.090 28392 29400 "POST /WsPackageProcess/GetPackageMapSelectOptions.json, from 10.145.0.44"
000001C6 PRG 2020-05-26 16:56:21.091 28392 29400 "Updated @timeoutAt for (/WSPACKAGEPROCESS/GETPACKAGEMAPSELECTOPTIONS) : 1590540981"
000001C7 PRG 2020-05-26 16:56:21.091 28392 29402 "Updated @timeoutAt for (/WSPACKAGEPROCESS/GETPACKAGEMAPSELECTOPTIONS) : 1590540981"
000001C8 PRG 2020-05-26 16:56:21.093 28392 29401 "HTTP First Line: POST /WsPackageProcess/GetPackageMapSelectOptions.json HTTP/1.1"
000001C9 PRG 2020-05-26 16:56:21.093 28392 29400 "TxSummary[activeReqs=1;auth=Ok;contLen=39;rcv=5ms;handleHttp=7ms;user=mvazquez@10.145.0.44;req=POST wspackageprocess.GETPACKAGEMAPSELECTOPTIONS v1.03;total=7ms;]"
000001CA PRG 2020-05-26 16:56:21.093 28392 29401 "POST /WsPackageProcess/GetPackageMapSelectOptions.json, from 10.145.0.44"
000001CB PRG 2020-05-26 16:56:21.093 28392 29403 "HTTP First Line: POST /WsPackageProcess/GetPackageMapSelectOptions.json HTTP/1.1"
000001CC PRG 2020-05-26 16:56:21.093 28392 29403 "POST /WsPackageProcess/GetPackageMapSelectOptions.json, from 10.145.0.44"
000001CD PRG 2020-05-26 16:56:21.094 28392 29401 "Updated @timeoutAt for (/WSPACKAGEPROCESS/GETPACKAGEMAPSELECTOPTIONS) : 1590540981"
000001CE PRG 2020-05-26 16:56:21.096 28392 29401 "TxSummary[activeReqs=2;auth=Ok;contLen=32;rcv=5ms;handleHttp=7ms;user=mvazquez@10.145.0.44;req=POST wspackageprocess.GETPACKAGEMAPSELECTOPTIONS v1.03;total=7ms;]"
000001CF PRG 2020-05-26 16:56:21.096 28392 29403 "Updated @timeoutAt for (/WSPACKAGEPROCESS/GETPACKAGEMAPSELECTOPTIONS) : 1590540981"
000001D0 PRG 2020-05-26 16:56:21.096 28392 29402 "TxSummary[activeReqs=3;auth=Ok;contLen=32;rcv=1ms;handleHttp=7ms;user=mvazquez@10.145.0.44;req=POST wspackageprocess.GETPACKAGEMAPSELECTOPTIONS v1.03;total=7ms;]"
000001D1 PRG 2020-05-26 16:56:21.097 28392 29403 "TxSummary[activeReqs=3;auth=Ok;contLen=34;rcv=0ms;handleHttp=4ms;user=mvazquez@10.145.0.44;req=POST wspackageprocess.GETPACKAGEMAPSELECTOPTIONS v1.03;total=4ms;]"
000001D2 PRG 2020-05-26 16:56:21.125 28392 29406 "HTTP First Line: POST /WsPackageProcess/ListPackages.json HTTP/1.1"
000001D3 PRG 2020-05-26 16:56:21.125 28392 29406 "POST /WsPackageProcess/ListPackages.json, from 10.145.0.44"
000001D4 PRG 2020-05-26 16:56:21.125 28392 29407 "HTTP First Line: POST /WsPackageProcess/GetPackageMapSelectOptions.json HTTP/1.1"
000001D5 PRG 2020-05-26 16:56:21.125 28392 29407 "POST /WsPackageProcess/GetPackageMapSelectOptions.json, from 10.145.0.44"
000001D6 PRG 2020-05-26 16:56:21.126 28392 29406 "Updated @timeoutAt for (/WSPACKAGEPROCESS/LISTPACKAGES) : 1590540981"
000001D7 PRG 2020-05-26 16:56:21.126 28392 29407 "Updated @timeoutAt for (/WSPACKAGEPROCESS/GETPACKAGEMAPSELECTOPTIONS) : 1590540981"
000001D8 PRG 2020-05-26 16:56:21.127 28392 29406 "TxSummary[activeReqs=1;auth=Ok;contLen=75;rcv=0ms;handleHttp=2ms;user=mvazquez@10.145.0.44;req=POST wspackageprocess.LISTPACKAGES v1.03;total=2ms;]"
000001D9 PRG 2020-05-26 16:56:21.129 28392 29407 "TxSummary[activeReqs=2;auth=Ok;contLen=34;rcv=0ms;handleHttp=4ms;user=mvazquez@10.145.0.44;req=POST wspackageprocess.GETPACKAGEMAPSELECTOPTIONS v1.03;total=4ms;]"
000001DA PRG 2020-05-26 16:56:21.223 28392 29410 "HTTP First Line: POST /WsPackageProcess/ListPackages.json HTTP/1.1"
000001DB PRG 2020-05-26 16:56:21.223 28392 29410 "POST /WsPackageProcess/ListPackages.json, from 10.145.0.44"
000001DC PRG 2020-05-26 16:56:21.225 28392 29410 "Updated @timeoutAt for (/WSPACKAGEPROCESS/LISTPACKAGES) : 1590540981"
000001DD USR 2020-05-26 16:56:21.235 28392 29410 "================================================"
000001DE USR 2020-05-26 16:56:21.235 28392 29410 "Program: 10.173.160.101:/mnt/disk1/HPCCSystems/bin/esp"
000001DF USR 2020-05-26 16:56:21.235 28392 29410 "Signal: 11 Segmentation fault"
000001E0 USR 2020-05-26 16:56:21.235 28392 29410 "Fault IP: 00007FF7710DB641"
000001E1 USR 2020-05-26 16:56:21.235 28392 29410 "Accessing: 0000000000000000"
000001E2 PRG 2020-05-26 16:56:21.235 28392 29410 "Backtrace:"
000001E3 PRG 2020-05-26 16:56:21.237 28392 29410 " /lib64/libc.so.6(+0x16f641) [0x7ff7710db641]"
000001E4 PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libjlib.so(_Z9WildMatchPKcS0_b+0x1b) [0x7ff7730236cb]"
000001E5 PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libws_packageprocess.so(_ZN19CWsPackageProcessEx14onListPackagesER11IEspContextR23IEspListPackagesRequestR24IEspListPackagesResponse+0x1cf) [0x7ff7513d27ff]"
000001E6 PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libws_packageprocess.so(ZN17ws_packageprocess28CWsPackageProcessSoapBinding17onGetInstantQueryER11IEspContextP12CHttpRequestP13CHttpResponsePKcS8+0x2e66) [0x7ff7513b0d68]"
000001E7 PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14EspHttpBinding5onGetEP12CHttpRequestP13CHttpResponse+0x5ab) [0x7ff7792a856b]"
000001E8 PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14EspHttpBinding14handleHttpPostEP12CHttpRequestP13CHttpResponse+0x17e) [0x7ff7792a3b9e]"
000001E9 PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer14processRequestEv+0x997) [0x7ff7792b6e67]"
000001EA PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libesphttp.so(_ZN11CHttpThread9onRequestEv+0xcb) [0x7ff7792aa60b]"
000001EB PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libesphttp.so(_ZN18CEspProtocolThread3runEv+0x95) [0x7ff7792e8b65]"
000001EC PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread5beginEv+0x28) [0x7ff77305e688]"
000001ED PRG 2020-05-26 16:56:21.237 28392 29410 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread11_threadmainEPv+0x1d) [0x7ff77305dd0d]"
000001EE PRG 2020-05-26 16:56:21.237 28392 29410 " /lib64/libpthread.so.0(+0x7e65) [0x7ff771341e65]"
000001EF PRG 2020-05-26 16:56:21.237 28392 29410 " /lib64/libc.so.6(clone+0x6d) [0x7ff77106a88d]"
000001F0 USR 2020-05-26 16:56:21.237 28392 29410 "Registers:"
000001F1 USR 2020-05-26 16:56:21.237 28392 29410 "EAX:0000000000000000 EBX:00007FF733FF5820 ECX:0000000000000000 EDX:0000000000000001 ESI:0000000000000000 EDI:0000000000000000"
000001F2 USR 2020-05-26 16:56:21.237 28392 29410 "R8 :00007FF76801D180 R9 :00007FF76801DAF0 R10:00000000FFFFFFC0 R11:00007FF7710F9C10"
000001F3 USR 2020-05-26 16:56:21.237 28392 29410 "R12:00007FF733FF5820 R13:0000000000000001 R14:00007FF75B0E7060 R15:0000000000000000"
000001F4 USR 2020-05-26 16:56:21.237 28392 29410 "CS:EIP:0033:00007FF7710DB641"
000001F5 USR 2020-05-26 16:56:21.237 28392 29410 " ESP:00007FF733FF5798 EBP:0000000000000000"
000001F6 USR 2020-05-26 16:56:21.237 28392 29410 "Stack[00007FF733FF5798]: 00007FF7730236CB 33FF582000007FF7 00007FF733FF5820 5800C7B000007FF7 00007FF75800C7B0 6801DB8000007FF7 00007FF76801DB80 33FF582000007FF7"
000001F7 USR 2020-05-26 16:56:21.237 28392 29410 "Stack[00007FF733FF57B8]: 00007FF733FF5820 513EEDDD00007FF7 00007FF7513EEDDD 513D27FF00007FF7 00007FF7513D27FF 33FF581000007FF7 00007FF733FF5810 47AE147B00007FF7"
000001F8 USR 2020-05-26 16:56:21.237 28392 29410 "Stack[00007FF733FF57D8]: 3FF07AE147AE147B 000000003FF07AE1 0000000000000000 58004A9000000000 00007FF758004A90 6801EA4000007FF7 00007FF76801EA40 6801ABC800007FF7"
000001F9 USR 2020-05-26 16:56:21.237 28392 29410 "Stack[00007FF733FF57F8]: 00007FF76801ABC8 33FF604000007FF7 00007FF733FF6040 7929C98B00007FF7 00007FF77929C98B 6801DB3000007FF7 00007FF76801DB30 0000000300007FF7"
000001FA USR 2020-05-26 16:56:21.238 28392 29410 "Stack[00007FF733FF5818]: 0000000800000003 69786F7200000008 36315F6569786F72 0000003036315F65 0000000000000030 33FF582000000000 00007FF733FF5820 0000000900007FF7"
000001FB USR 2020-05-26 16:56:21.238 28392 29410 "Stack[00007FF733FF5838]: 0000000000000009 0000001000000000 0000000000000010 6801EE0800000000 00007FF76801EE08 6801EDE000007FF7 00007FF76801EDE0 6801942000007FF7"
000001FC USR 2020-05-26 16:56:21.238 28392 29410 "Stack[00007FF733FF5858]: 00007FF768019420 33FF5F9000007FF7 00007FF733FF5F90 01CBA06000007FF7 0000000001CBA060 513D2C6000000000 00007FF7513D2C60 6801EA4000007FF7"
000001FD USR 2020-05-26 16:56:21.238 28392 29410 "Stack[00007FF733FF5878]: 00007FF76801EA40 33FF604000007FF7 00007FF733FF6040 513B0D6800007FF7 00007FF7513B0D68 0000004000007FF7 0000000000000040 47AE147B00000000"
000001FE USR 2020-05-26 16:56:21.238 28392 29410 "ThreadList:
7FF76E055700 140700679493376 28393: CMPNotifyClosedThread
7FF76D854700 140700671100672 28394: MP Connection Thread
7FF76C852700 140700654315264 28396: CSocketBaseThread
7FF76D053700 140700662707968 28397: LogMsgParentReceiver
7FF767FFF700 140700578477824 28398: LogMsgFilterReceiver
7FF7677FE700 140700570085120 28402: CMemoryUsageReporter
7FF766FFD700 140700561692416 28403: CSessionCleaner
7FF7667FC700 140700553299712 28404: unknown
7FF7651D8700 140700530083584 28405: CDaliPublisherClient
7FF7649D7700 140700521690880 28406: unknown
7FF752108700 140700210464512 28407: unknown
7FF74AD6F700 140700089251584 28408: Activity Reader
7FF74A00D700 140700075218688 28409: unknown
7FF743FFF700 140699974498048 28410: unknown
7FF7437FE700 140699966105344 28411: unknown
7FF742FFD700 140699957712640 28412: Usage Reader
7FF764076700 140700511856384 28413: WsMachine Thread Pool
7FF764045700 140700511655680 28414: WsMachine Thread Pool
7FF764024700 140700511520512 28415: WsMachine Thread Pool
7FF7480AC700 140700042315520 28416: WsMachine Thread Pool
7FF74809B700 140700042245888 28418: WsMachine Thread Pool
7FF74808A700 140700042176256 28419: WsMachine Thread Pool
7FF748069700 140700042041088 28421: WsMachine Thread Pool
7FF748048700 140700041905920 28422: WsMachine Thread Pool
7FF748027700 140700041770752 28423: WsMachine Thread Pool
7FF740106700 140699908466432 28424: WsMachine Thread Pool

the core file is on 10.173.160.101:/var/lib/HPCCSystems/myesp/core_esp.35018

 

Conclusion

None

Activity

Show:

Kanghua Wang May 27, 2020 at 4:01 PM

The empty Process in the second round causes ESP cores. I am thinking about a fix now.

Miguel Vazquez May 27, 2020 at 2:39 PM
Edited

During investigating with Kevin we see any following calls to http://localhost:8080/WsPackageProcess/ListPackages.json seem to core.

 

First round was with params:

Target: *, Process: *, ProcessFilter: *, PageStartFrom: 0, PageSize: 50

Second round with params with active package map target

Target: roxie_160

Christopher Lo May 27, 2020 at 1:55 PM

do you know off the top of your head what the equivalent ecl command line is?

Miguel Vazquez May 27, 2020 at 1:44 PM
Edited

 

WsPackageMaps.GetPackageMapSelectOptions({    IncludeTargets: 1, IncludeProcesses: 1, IncludeProcessFilters: 1)

 

 

Kanghua Wang May 26, 2020 at 10:00 PM
Edited

Yes. ESP should not core. I just checked the esp log. Only cores if the request is from Miguel's computer. Not for me and Chris. could you please check any parameter used in your requests?

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Point

Fix versions

Pull Request URL

https://github.com/hpcc-systems/HPCC-Platform/pull/13788

Affects versions

Created May 26, 2020 at 9:22 PM
Updated May 28, 2020 at 9:08 AM
Resolved May 28, 2020 at 9:08 AM

Flag notifications