mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Peach (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (MESOS-7017) HTTP API responses can crash the master.
Date Fri, 27 Jan 2017 23:11:24 GMT

    [ https://issues.apache.org/jira/browse/MESOS-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15843632#comment-15843632
] 

James Peach edited comment on MESOS-7017 at 1/27/17 11:10 PM:
--------------------------------------------------------------

Here's the partial stack trace:

{noformat}
#2  0x00007fb830734696 in google::DumpStackTraceAndExit () at src/utilities.cc:147
#3  0x00007fb83072c08d in google::LogMessage::Fail () at src/logging.cc:1458
#4  0x00007fb83072de1d in google::LogMessage::SendToLog (this=Unhandled dwarf expression opcode
0xf3
) at src/logging.cc:1412
#5  0x00007fb83072bc12 in google::LogMessage::Flush (this=0x7fb8227f3890) at src/logging.cc:1281
#6  0x00007fb83072e7f9 in google::LogMessageFatal::~LogMessageFatal (this=Unhandled dwarf
expression opcode 0xf3
) at src/logging.cc:1984
#7  0x00007fb82fb35113 in evolve<mesos::v1::master::Response> (response=...) at ../../src/internal/evolve.cpp:63
#8  mesos::internal::evolve (response=...) at ../../src/internal/evolve.cpp:218
#9  0x00007fb82fba8dd6 in mesos::internal::master::Master::Http::<lambda(const std::tuple<process::Owned<mesos::ObjectApprover>,
process::Owned<mesos::ObjectApprover> >&)>::operator()(const std::tuple<process::Owned<mesos::ObjectApprover>,
process::Owned<mesos::ObjectApprover> > &) const (__closure=0x7fb720c7a940, approvers=Unhandled
dwarf expression opcode 0xf3
)
    at ../../src/master/http.cpp:3772
#10 0x00007fb82fba9068 in operator() (__functor=Unhandled dwarf expression opcode 0xf3
) at ../../3rdparty/libprocess/include/process/deferred.hpp:225
#11 std::_Function_handler<process::Future<process::http::Response>(), process::_Deferred<G>::operator
std::function<R(P0)>() const::<lambda(P0)> [with R = process::Future<process::http::Response>;
P0 = const std::tuple<process::Owned<mesos::ObjectApprover>, process::Owned<mesos::ObjectApprover>
>&; F = mesos::internal::master::Master::Http::getTasks(const mesos::master::Call&,
const Option<std::basic_string<char> >&, mesos::ContentType) const::<lambda(const
std::tuple<process::Owned<mesos::ObjectApprover>, process::Owned<mesos::ObjectApprover>
>&)>]::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=Unhandled
dwarf expression opcode 0xf3
) at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2025
#12 0x00007fb82faf73b3 in operator() (__functor=Unhandled dwarf expression opcode 0xf3
) at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2439
#13 operator() (__functor=Unhandled dwarf expression opcode 0xf3
) at ../../3rdparty/libprocess/include/process/dispatch.hpp:112
#14 std::_Function_handler<void(process::ProcessBase*), process::internal::Dispatch<process::Future<T>
>::operator()(const process::UPID&, F&&) [with F = std::function<process::Future<process::http::Response>()>&;
R = process::http::Response]::<lambda(process::ProcessBase*)> >::_M_invoke(const
std::_Any_data &, process::ProcessBase *) (__functor=Unhandled dwarf expression opcode
0xf3
)
    at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2039
{noformat}

So the proximate cause of the crash is that {{evolve}} does a bidirectional serialization.
For large messages this causes 2 large allocations even if is doesn't trigger the {{CHECK}}.


was (Author: jamespeach):
Here's the partial stack trace:

{noformat}
#2  0x00007fb830734696 in google::DumpStackTraceAndExit () at src/utilities.cc:147
#3  0x00007fb83072c08d in google::LogMessage::Fail () at src/logging.cc:1458
#4  0x00007fb83072de1d in google::LogMessage::SendToLog (this=Unhandled dwarf expression opcode
0xf3
) at src/logging.cc:1412
#5  0x00007fb83072bc12 in google::LogMessage::Flush (this=0x7fb8227f3890) at src/logging.cc:1281
#6  0x00007fb83072e7f9 in google::LogMessageFatal::~LogMessageFatal (this=Unhandled dwarf
expression opcode 0xf3
) at src/logging.cc:1984
#7  0x00007fb82fb35113 in evolve<mesos::v1::master::Response> (response=...) at ../../src/internal/evolve.cpp:63
#8  mesos::internal::evolve (response=...) at ../../src/internal/evolve.cpp:218
#9  0x00007fb82fba8dd6 in mesos::internal::master::Master::Http::<lambda(const std::tuple<process::Owned<mesos::ObjectApprover>,
process::Owned<mesos::ObjectApprover> >&)>::operator()(const std::tuple<process::Owned<mesos::ObjectApprover>,
process::Owned<mesos::ObjectApprover> > &) const (__closure=0x7fb720c7a940, approvers=Unhandled
dwarf expression opcode 0xf3
)
    at ../../src/master/http.cpp:3772
#10 0x00007fb82fba9068 in operator() (__functor=Unhandled dwarf expression opcode 0xf3
) at ../../3rdparty/libprocess/include/process/deferred.hpp:225
#11 std::_Function_handler<process::Future<process::http::Response>(), process::_Deferred<G>::operator
std::function<R(P0)>() const::<lambda(P0)> [with R = process::Future<process::http::Response>;
P0 = const std::tuple<process::Owned<mesos::ObjectApprover>, process::Owned<mesos::ObjectApprover>
>&; F = mesos::internal::master::Master::Http::getTasks(const mesos::master::Call&,
const Option<std::basic_string<char> >&, mesos::ContentType) const::<lambda(const
std::tuple<process::Owned<mesos::ObjectApprover>, process::Owned<mesos::ObjectApprover>
>&)>]::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=Unhandled
dwarf expression opcode 0xf3
) at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2025
#12 0x00007fb82faf73b3 in operator() (__functor=Unhandled dwarf expression opcode 0xf3
) at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2439
#13 operator() (__functor=Unhandled dwarf expression opcode 0xf3
) at ../../3rdparty/libprocess/include/process/dispatch.hpp:112
#14 std::_Function_handler<void(process::ProcessBase*), process::internal::Dispatch<process::Future<T>
>::operator()(const process::UPID&, F&&) [with F = std::function<process::Future<process::http::Response>()>&;
R = process::http::Response]::<lambda(process::ProcessBase*)> >::_M_invoke(const
std::_Any_data &, process::ProcessBase *) (__functor=Unhandled dwarf expression opcode
0xf3
)
    at /opt/rh/devtoolset-3/root/usr/include/c++/4.9.2/functional:2039
{noformat}

So the proximate cause of the crash is that {{evolve}} does an unnecessary bidirectional serialization.
For large messages this causes 2 unnecessary large allocations even if is doesn't trigger
the {{CHECK}}.

> HTTP API responses can crash the master.
> ----------------------------------------
>
>                 Key: MESOS-7017
>                 URL: https://issues.apache.org/jira/browse/MESOS-7017
>             Project: Mesos
>          Issue Type: Bug
>          Components: HTTP API
>            Reporter: James Peach
>            Priority: Critical
>
> The master can crash when generating large responses to small API requests. One manifestation
of this is querying the tasks.
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:180] A protocol message was rejected
because it was too big (more than 67108864 bytes).  To increase the limit (or to disable these
warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F0126 18:34:18.790386 26230 evolve.cpp:63] Check failed: t.ParsePartialFromString(data)
Failed to parse mesos.v1.master.Response while evolving from mesos.master.Response
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message