mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Mulcahy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MESOS-3589) Mesos tasks with many ports cause significant master performance problems
Date Tue, 06 Oct 2015 00:05:27 GMT

     [ https://issues.apache.org/jira/browse/MESOS-3589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

James Mulcahy updated MESOS-3589:
---------------------------------
    Attachment: mesos-master-perf-top-call-expanded.txt

> Mesos tasks with many ports cause significant master performance problems
> -------------------------------------------------------------------------
>
>                 Key: MESOS-3589
>                 URL: https://issues.apache.org/jira/browse/MESOS-3589
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: James Mulcahy
>         Attachments: mesos-master-perf-top-call-expanded.txt, mesos-master-perf.txt
>
>
> Today, I used a framework to fire off some tasks which each requested a lot of ports.
 When doing so, the meson-master performance drops heavily.
> With 70 tasks each requesting 100 ports, the master seems to spend the majority (~90%)
of its time dealing with merging and coalescing Value_Range objects.  
> I'll attach some text views of the output of 'perf' record/report from the meson-master
during this.  The call-graph trace for the most frequently sampled call is:
> {{
> -   4.42%  mesos-master  mesos-master         [.] google::protobuf::internal::RepeatedPtrFieldBase::size()
const                                                                                    
                                                                                       
>    - google::protobuf::internal::RepeatedPtrFieldBase::size() const                 
                                                                                         
                                                                                         
                   
>       - 37.05% mesos::Value_Ranges::range_size() const                              
                                                                                         
                                                                                         
                   
>          - 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range const&)
                                                                                         
                                                                                         
                     
>             - mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges const&) 
                                                                                         
                                                                                         
                       
>                - 68.35% mesos::operator+=(mesos::Value_Ranges&, mesos::Value_Ranges
const&)                                                                              
                                                                                         
                        
>                   - 99.46% Option<mesos::Value_Ranges> mesos::Resources::get<mesos::Value_Ranges>(std::string
const&) const                                                                        
                                                                                        
>                      - mesos::internal::model(mesos::Resources const&)          
                                                                                         
                                                                                         
                       
>                         - 97.58% mesos::internal::model(mesos::internal::Task const&)
                                                                                         
                                                                                         
                  
>                              mesos::internal::master::model(mesos::internal::master::Framework
const&)                                                                              
                                                                                         
             
>                              mesos::internal::master::Master::Http::state(process::http::Request
const&) const                                                                        
                                                                                         
           
>                              mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}::operator()(process::http::Request const&) const                      
                                                                                         
     
>                              std::_Function_handler<process::Future<process::http::Response>
(process::http::Request const&), mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)
   
>                              std::function<process::Future<process::http::Response>
(process::http::Request const&)>::operator()(process::http::Request const&) const
                                                                                         
                            
>                              process::ProcessBase::visit(process::HttpEvent const&)
                                                                                         
                                                                                         
                    
>                              process::HttpEvent::visit(process::EventVisitor*) const
                                                                                         
                                                                                         
                   
>                              process::ProcessBase::serve(process::Event const&) 
                                                                                         
                                                                                         
                       
>                              process::ProcessManager::resume(process::ProcessBase*) 
                                                                                         
                                                                                         
                   
>                              process::schedule(void*)                               
                                                                                         
                                                                                         
                   
>                              start_thread                                           
                                                                                         
                                                                                         
                   
> }}
> The top ~80 call sides pretty much all relate to Value_Range handling, and account for
90% of the runtime of the mesos-master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message