mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gilbert Song (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-8346) Resubscription of a resource provider will crash the agent if its HTTP connection isn't closed
Date Wed, 20 Dec 2017 08:55:00 GMT

    [ https://issues.apache.org/jira/browse/MESOS-8346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298109#comment-16298109
] 

Gilbert Song commented on MESOS-8346:
-------------------------------------

[~nfnt], seems like this is targeted for 1.5.0. Do we have an estimate when it will land?

> Resubscription of a resource provider will crash the agent if its HTTP connection isn't
closed
> ----------------------------------------------------------------------------------------------
>
>                 Key: MESOS-8346
>                 URL: https://issues.apache.org/jira/browse/MESOS-8346
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 1.5.0
>            Reporter: Jan Schlicht
>            Assignee: Jan Schlicht
>            Priority: Blocker
>              Labels: mesosphere
>
> A resource provider might resubscribe while its old HTTP connection wasn't properly closed.
In that case an agent will crashm with, e.g., the following log:
> {noformat}
> I1219 13:33:51.937295 128610304 manager.cpp:570] Subscribing resource provider {"id":{"value":"8e71beef-796e-4bde-9257-952ed0f230a5"},"name":"test","type":"org.apache.mesos.rp.test"}
> I1219 13:33:51.937443 128610304 manager.cpp:134] Terminating resource provider 8e71beef-796e-4bde-9257-952ed0f230a5
> I1219 13:33:51.937760 128610304 manager.cpp:134] Terminating resource provider 8e71beef-796e-4bde-9257-952ed0f230a5
> E1219 13:33:51.937851 129683456 http_connection.hpp:445] End-Of-File received
> I1219 13:33:51.937865 131293184 slave.cpp:7105] Handling resource provider message 'DISCONNECT:
resource provider 8e71beef-796e-4bde-9257-952ed0f230a5'
> I1219 13:33:51.937968 131293184 slave.cpp:7347] Forwarding new total resources cpus:2;
mem:1024; disk:1024; ports:[31000-32000]
> F1219 13:33:51.938052 132366336 manager.cpp:606] Check failed: resourceProviders.subscribed.contains(resourceProviderId)

> *** Check failure stack trace: ***
> E1219 13:33:51.938583 130756608 http_connection.hpp:445] End-Of-File received
> I1219 13:33:51.938987 129683456 hierarchical.cpp:669] Agent 0019c3fa-28c5-43a9-88d0-709eee271c62-S0
(172.18.8.13) updated with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000]
>     @        0x1125380ef  google::LogMessageFatal::~LogMessageFatal()
>     @        0x112534ae9  google::LogMessageFatal::~LogMessageFatal()
> I1219 13:33:51.939131 129683456 hierarchical.cpp:1517] Performed allocation for 1 agents
in 61830ns
> I1219 13:33:51.945793 2646795072 slave.cpp:927] Agent terminating
> I1219 13:33:51.945955 129146880 master.cpp:1305] Agent 0019c3fa-28c5-43a9-88d0-709eee271c62-S0
at slave(1)@172.18.8.13:64430 (172.18.8.13) disconnected
> I1219 13:33:51.945979 129146880 master.cpp:3364] Disconnecting agent 0019c3fa-28c5-43a9-88d0-709eee271c62-S0
at slave(1)@172.18.8.13:64430 (172.18.8.13)
> I1219 13:33:51.946022 129146880 master.cpp:3383] Deactivating agent 0019c3fa-28c5-43a9-88d0-709eee271c62-S0
at slave(1)@172.18.8.13:64430 (172.18.8.13)
> I1219 13:33:51.946081 131293184 hierarchical.cpp:766] Agent 0019c3fa-28c5-43a9-88d0-709eee271c62-S0
deactivated
>     @        0x115f2761d  mesos::internal::ResourceProviderManagerProcess::subscribe()::$_2::operator()()
>     @        0x115f2977d  _ZN5cpp176invokeIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS2_14HttpConnectionERKNS1_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEDTclclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSG_DpOSH_
>     @        0x115f29740  _ZN6lambda8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS3_14HttpConnectionERKNS2_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEE13invoke_expandISC_NSt3__15tupleIJSG_EEENSK_IJEEEJLm0EEEEDTclsr5cpp17E6invokeclsr3stdE7forwardIT_Efp_Espcl6expandclsr3stdE3getIXT2_EEclsr3stdE7forwardIT0_Efp0_EEclsr3stdE7forwardIT1_Efp2_EEEEOSN_OSO_N5cpp1416integer_sequenceImJXspT2_EEEEOSP_
>     @        0x115f296bb  _ZNO6lambda8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS3_14HttpConnectionERKNS2_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEclIJEEEDTcl13invoke_expandclL_ZNSt3__14moveIRSC_EEONSJ_16remove_referenceIT_E4typeEOSN_EdtdefpT1fEclL_ZNSK_IRNSJ_5tupleIJSG_EEEEESQ_SR_EdtdefpT10bound_argsEcvN5cpp1416integer_sequenceImJLm0EEEE_Eclsr3stdE16forward_as_tuplespclsr3stdE7forwardIT_Efp_EEEEDpOSY_
>     @        0x115f2965d  _ZN5cpp176invokeIN6lambda8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS5_14HttpConnectionERKNS4_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEJEEEDTclclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSK_DpOSL_
>     @        0x115f29631  _ZN6lambda8internal6InvokeIvEclINS0_7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS6_14HttpConnectionERKNS5_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEJEEEvOT_DpOT0_
>     @        0x115f29526  _ZNO6lambda12CallableOnceIFvvEE10CallableFnINS_8internal7PartialIZN5mesos8internal30ResourceProviderManagerProcess9subscribeERKNS7_14HttpConnectionERKNS6_17resource_provider14Call_SubscribeEE3$_2JN7process6FutureI7NothingEEEEEEclEv
>     @        0x10b6ca690  _ZNO6lambda12CallableOnceIFvvEEclEv
>     @        0x10be09295  _ZZN7process8internal8DispatchIvEclIN6lambda12CallableOnceIFvvEEEEEvRKNS_4UPIDEOT_ENKUlOS7_PNS_11ProcessBaseEE_clESD_SF_
>     @        0x10be09180  _ZN5cpp176invokeIZN7process8internal8DispatchIvEclIN6lambda12CallableOnceIFvvEEEEEvRKNS1_4UPIDEOT_EUlOS9_PNS1_11ProcessBaseEE_JS9_SH_EEEDTclclsr3stdE7forwardISD_Efp_Espclsr3stdE7forwardIT0_Efp0_EEESE_DpOSJ_
>     @        0x10be0912b  _ZN6lambda8internal7PartialIZN7process8internal8DispatchIvEclINS_12CallableOnceIFvvEEEEEvRKNS2_4UPIDEOT_EUlOS9_PNS2_11ProcessBaseEE_JS9_NSt3__112placeholders4__phILi1EEEEE13invoke_expandISI_NSJ_5tupleIJS9_SM_EEENSP_IJOSH_EEEJLm0ELm1EEEEDTclsr5cpp17E6invokeclsr3stdE7forwardISD_Efp_Espcl6expandclsr3stdE3getIXT2_EEclsr3stdE7forwardIT0_Efp0_EEclsr3stdE7forwardIT1_Efp2_EEEESE_OST_N5cpp1416integer_sequenceImJXspT2_EEEEOSU_
>     @        0x10be0905f  _ZNO6lambda8internal7PartialIZN7process8internal8DispatchIvEclINS_12CallableOnceIFvvEEEEEvRKNS2_4UPIDEOT_EUlOS9_PNS2_11ProcessBaseEE_JS9_NSt3__112placeholders4__phILi1EEEEEclIJSH_EEEDTcl13invoke_expandclL_ZNSJ_4moveIRSI_EEONSJ_16remove_referenceISD_E4typeESE_EdtdefpT1fEclL_ZNSP_IRNSJ_5tupleIJS9_SM_EEEEESU_SE_EdtdefpT10bound_argsEcvN5cpp1416integer_sequenceImJLm0ELm1EEEE_Eclsr3stdE16forward_as_tuplespclsr3stdE7forwardIT_Efp_EEEEDpOS11_
>     @        0x10be08f4d  _ZN5cpp176invokeIN6lambda8internal7PartialIZN7process8internal8DispatchIvEclINS1_12CallableOnceIFvvEEEEEvRKNS4_4UPIDEOT_EUlOSB_PNS4_11ProcessBaseEE_JSB_NSt3__112placeholders4__phILi1EEEEEEJSJ_EEEDTclclsr3stdE7forwardISF_Efp_Espclsr3stdE7forwardIT0_Efp0_EEESG_DpOSQ_
>     @        0x10be08f11  _ZN6lambda8internal6InvokeIvEclINS0_7PartialIZN7process8internal8DispatchIvEclINS_12CallableOnceIFvvEEEEEvRKNS5_4UPIDEOT_EUlOSC_PNS5_11ProcessBaseEE_JSC_NSt3__112placeholders4__phILi1EEEEEEJSK_EEEvSH_DpOT0_
>     @        0x10be08d36  _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8internal8DispatchIvEclINS0_IFvvEEEEEvRKNS1_4UPIDEOT_EUlOSE_S3_E_JSE_NSt3__112placeholders4__phILi1EEEEEEEclEOS3_
>     @        0x11fd64bc9  _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEEclES3_
>     @        0x11fd64a69  process::ProcessBase::consume()
>     @        0x11fe20ac4  _ZNO7process13DispatchEvent7consumeEPNS_13EventConsumerE
>     @        0x113c77819  process::ProcessBase::serve()
>     @        0x11fd5b8c9  process::ProcessManager::resume()
>     @        0x11fe8260b  process::ProcessManager::init_threads()::$_1::operator()()
>     @        0x11fe82190  _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN7process14ProcessManager12init_threadsEvE3$_1EEEEEPvSB_
>     @     0x7fff64da56c1  _pthread_body
>     @     0x7fff64da556d  _pthread_start
>     @     0x7fff64da4c5d  thread_start
> Abort trap: 6
> {noformat}
> This is due to a race condition in {{resource_provider/manager.cpp}} when handling closed
HTTP connections of resource providers. If a resource provider resubscribes and its old HTTP
connection is still open, the resource provider manager will close it. This is unexpected
and will trigger closing the new HTTP connection which results in a failed {{CHECK}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message