Return-Path: X-Original-To: apmail-mesos-issues-archive@minotaur.apache.org Delivered-To: apmail-mesos-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3B748186EE for ; Tue, 6 Oct 2015 00:03:30 +0000 (UTC) Received: (qmail 26129 invoked by uid 500); 6 Oct 2015 00:03:27 -0000 Delivered-To: apmail-mesos-issues-archive@mesos.apache.org Received: (qmail 26115 invoked by uid 500); 6 Oct 2015 00:03:27 -0000 Mailing-List: contact issues-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@mesos.apache.org Delivered-To: mailing list issues@mesos.apache.org Received: (qmail 26106 invoked by uid 99); 6 Oct 2015 00:03:26 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2015 00:03:26 +0000 Date: Tue, 6 Oct 2015 00:03:26 +0000 (UTC) From: "James Mulcahy (JIRA)" To: issues@mesos.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (MESOS-3589) Mesos tasks with many ports cause significant master performance problems MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 James Mulcahy created MESOS-3589: ------------------------------------ Summary: Mesos tasks with many ports cause significant master performance problems Key: MESOS-3589 URL: https://issues.apache.org/jira/browse/MESOS-3589 Project: Mesos Issue Type: Bug Affects Versions: 0.23.0 Reporter: James Mulcahy Today, I used a framework to fire off some tasks which each requested a lot of ports. When doing so, the meson-master performance drops heavily. With 70 tasks each requesting 100 ports, the master seems to spend the majority (~90%) of its time dealing with merging and coalescing Value_Range objects. I'll attach some text views of the output of 'perf' record/report from the meson-master during this. The call-graph trace for the most frequently sampled call is: {{ - 4.42% mesos-master mesos-master [.] google::protobuf::internal::RepeatedPtrFieldBase::size() const - google::protobuf::internal::RepeatedPtrFieldBase::size() const - 37.05% mesos::Value_Ranges::range_size() const - 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range const&) - mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges const&) - 68.35% mesos::operator+=(mesos::Value_Ranges&, mesos::Value_Ranges const&) - 99.46% Option mesos::Resources::get(std::string const&) const - mesos::internal::model(mesos::Resources const&) - 97.58% mesos::internal::model(mesos::internal::Task const&) mesos::internal::master::model(mesos::internal::master::Framework const&) mesos::internal::master::Master::Http::state(process::http::Request const&) const mesos::internal::master::Master::initialize()::{lambda(process::http::Request const&)#9}::operator()(process::http::Request const&) const std::_Function_handler (process::http::Request const&), mesos::internal::master::Master::initialize()::{lambda(process::http::Request const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&) std::function (process::http::Request const&)>::operator()(process::http::Request const&) const process::ProcessBase::visit(process::HttpEvent const&) process::HttpEvent::visit(process::EventVisitor*) const process::ProcessBase::serve(process::Event const&) process::ProcessManager::resume(process::ProcessBase*) process::schedule(void*) start_thread }} The top ~80 call sides pretty much all relate to Value_Range handling, and account for 90% of the runtime of the mesos-master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)