mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gilbert Song (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (MESOS-8985) Posting to the operator api with 'accept recordio' header can crash the agent
Date Mon, 11 Jun 2018 22:29:00 GMT

     [ https://issues.apache.org/jira/browse/MESOS-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gilbert Song reassigned MESOS-8985:
-----------------------------------

    Assignee:     (was: Gilbert Song)

> Posting to the operator api with 'accept recordio' header can crash the agent
> -----------------------------------------------------------------------------
>
>                 Key: MESOS-8985
>                 URL: https://issues.apache.org/jira/browse/MESOS-8985
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 1.4.1, 1.5.1
>            Reporter: Philip Norman
>            Priority: Major
>         Attachments: mesos-slave-crash.log
>
>
> It's possible to crash the mesos agent by posting a reasonable request to the operator
API.
> h3. Background:
> Sending a request to the v1 api endpoint with an unsupported 'accept' header:
> {code:java}
> curl -X POST http://10.0.3.27:5051/api/v1 \
>   -H 'accept: application/atom+xml' \
>   -H 'content-type: application/json' \
>   -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": true,"show_standalone":
true}}'{code}
> Results in the following friendly error message:
> {code:java}
> Expecting 'Accept' to allow application/json or application/x-protobuf or application/recordio{code}
> h3. Reproducible crash:
> However, sending the same request with 'application/recordio' 'accept' header:
> {code:java}
> curl -X POST \
> http://10.0.3.27:5051/api/v1 \
>   -H 'accept: application/recordio' \
>   -H 'content-type: application/json' \
>   -d '{"type":"GET_CONTAINERS","get_containers":{"show_nested": true,"show_standalone":
true}}'{code}
> causes the agent to crash (no response is received).
> Crash log is shown below, full log from the agent is attached here:
> {code:java}
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: I0607 22:30:32.397320
3743 logfmt.cpp:178] type=audit timestamp=2018-06-07 22:30:32.397243904+00:00 reason="Error
in token 'Missing 'Authorization' header from HTTP request'. Allowing anonymous connection"
object="/slave(1)/api/v1" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36" authorizer="mesos-agent" action="POST"
result=allow srcip=10.0.6.99 dstport=5051 srcport=42084 dstip=10.0.3.27
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: W0607 22:30:32.397434
3743 authenticator.cpp:289] Error in token on request from '10.0.6.99:42084': Missing 'Authorization'
header from HTTP request
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: W0607 22:30:32.397466
3743 authenticator.cpp:291] Falling back to anonymous connection using user 'dcos_anonymous'
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: I0607 22:30:32.397629
3748 http.cpp:1099] HTTP POST for /slave(1)/api/v1 from 10.0.6.99:42084 with User-Agent='Mozilla/5.0
(Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181
Safari/537.36'
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: I0607 22:30:32.397784
3748 http.cpp:2030] Processing GET_CONTAINERS call
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: F0607 22:30:32.398736
3747 http.cpp:121] Serializing a RecordIO stream is not supported
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: *** Check
failure stack trace: ***
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f619478636d
google::LogMessage::Fail()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f619478819d
google::LogMessage::SendToLog()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6194785f5c
google::LogMessage::Flush()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6194788a99
google::LogMessageFatal::~LogMessageFatal()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935e2b9d
mesos::internal::serialize()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a4c0ef
_ZNO6lambda12CallableOnceIFN7process6FutureINS1_4http8ResponseEEERKN4JSON5ArrayEEE10CallableFnIZNK5mesos8internal5slave4Http13getContainersERKNSD_5agent4CallENSD_11ContentTypeERK6OptionINS3_14authentication9PrincipalEEEUlRKNS2_IS7_EEE0_EclES9_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a81d61
process::internal::thenf<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a59b15
_ZNO6lambda12CallableOnceIFvRKN7process6FutureIN4JSON5ArrayEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFNS2_INS1_4http8ResponseEEERKS4_EEESt10unique_ptrINS1_7PromiseISE_EESt14default_deleteISN_EES7_EJSJ_SQ_St12_PlaceholderILi1EEEEEEclES7_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a6e4e9
process::internal::run<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7fa28
process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe
process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe
process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe
process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a7f9fe
process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a84e00
process::Future<>::onReady()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a8509e
process::Promise<>::associate()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a856ac
process::internal::thenf<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a59935
_ZNO6lambda12CallableOnceIFvRKN7process6FutureISt5tupleIINS2_ISt4listINS2_IN5mesos15ContainerStatusEEESaIS7_EEEENS2_IS4_INS2_INS5_18ResourceStatisticsEEESaISC_EEEEEEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFNS2_IN4JSON5ArrayEEERKSG_EEESt10unique_ptrINS1_7PromiseISQ_EESt14default_deleteISZ_EESJ_EISV_S12_St12_PlaceholderILi1EEEEEEclESJ_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a81359
process::internal::run<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a83f12
_ZN7process6FutureISt5tupleIJNS0_ISt4listINS0_IN5mesos15ContainerStatusEEESaIS5_EEEENS0_IS2_INS0_INS3_18ResourceStatisticsEEESaISA_EEEEEEE4_setIRKSE_EEbOT_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a85f10
_ZNK7process6FutureISt5tupleIJNS0_ISt4listINS0_IN5mesos15ContainerStatusEEESaIS5_EEEENS0_IS2_INS0_INS3_18ResourceStatisticsEEESaISA_EEEEEEE7onReadyEON6lambda12CallableOnceIFvRKSE_EEE
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a861ae
process::Promise<>::associate()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a866ac
process::internal::thenf<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f6193a59875
_ZNO6lambda12CallableOnceIFvRKN7process6FutureISt4listINS2_I7NothingEESaIS5_EEEEEE10CallableFnINS_8internal7PartialIPFvONS0_IFNS2_ISt5tupleIINS2_IS3_INS2_IN5mesos15ContainerStatusEEESaISJ_EEEENS2_IS3_INS2_INSH_18ResourceStatisticsEEESaISO_EEEEEEEERKS7_EEESt10unique_ptrINS1_7PromiseISS_EESt14default_deleteIS11_EESA_EISX_S14_St12_PlaceholderILi1EEEEEEclESA_
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935c1a19
process::internal::run<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935cf25f
process::Future<>::_set<>()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61935cf44b
process::internal::AwaitProcess<>::waited()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61946d79d1
process::ProcessBase::consume()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61946e8dcc
process::ProcessManager::resume()
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61946ee7a6
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61918d8d73
(unknown)
> Jun 07 22:30:32 ip-10-0-3-27.us-west-2.compute.internal mesos-agent[3718]: @ 0x7f61913d952c
(unknown)
> Jun 07 22:30:34 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service:
Main process exited, code=killed, status=6/ABRT
> Jun 07 22:30:34 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service:
Unit entered failed state.
> Jun 07 22:30:34 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service:
Failed with result 'signal'.
> Jun 07 22:30:39 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: dcos-mesos-slave.service:
Service hold-off time over, scheduling restart.
> Jun 07 22:30:39 ip-10-0-3-27.us-west-2.compute.internal systemd[1]: Stopped Mesos Agent:
distributed systems kernel agent.{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message