mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Mulcahy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MESOS-3589) Mesos tasks with many ports cause significant master performance problems
Date Tue, 06 Oct 2015 00:03:26 GMT
James Mulcahy created MESOS-3589:
------------------------------------

             Summary: Mesos tasks with many ports cause significant master performance problems
                 Key: MESOS-3589
                 URL: https://issues.apache.org/jira/browse/MESOS-3589
             Project: Mesos
          Issue Type: Bug
    Affects Versions: 0.23.0
            Reporter: James Mulcahy


Today, I used a framework to fire off some tasks which each requested a lot of ports.  When
doing so, the meson-master performance drops heavily.

With 70 tasks each requesting 100 ports, the master seems to spend the majority (~90%) of
its time dealing with merging and coalescing Value_Range objects.  

I'll attach some text views of the output of 'perf' record/report from the meson-master during
this.  The call-graph trace for the most frequently sampled call is:

{{
-   4.42%  mesos-master  mesos-master         [.] google::protobuf::internal::RepeatedPtrFieldBase::size()
const                                                                                    
                                                                                       
   - google::protobuf::internal::RepeatedPtrFieldBase::size() const                      
                                                                                         
                                                                                         
              
      - 37.05% mesos::Value_Ranges::range_size() const                                   
                                                                                         
                                                                                         
              
         - 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range const&)   
                                                                                         
                                                                                         
                  
            - mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges const&)      
                                                                                         
                                                                                         
                  
               - 68.35% mesos::operator+=(mesos::Value_Ranges&, mesos::Value_Ranges const&)
                                                                                         
                                                                                         
            
                  - 99.46% Option<mesos::Value_Ranges> mesos::Resources::get<mesos::Value_Ranges>(std::string
const&) const                                                                        
                                                                                        
                     - mesos::internal::model(mesos::Resources const&)               
                                                                                         
                                                                                         
                  
                        - 97.58% mesos::internal::model(mesos::internal::Task const&)
                                                                                         
                                                                                         
                  
                             mesos::internal::master::model(mesos::internal::master::Framework
const&)                                                                              
                                                                                         
             
                             mesos::internal::master::Master::Http::state(process::http::Request
const&) const                                                                        
                                                                                         
           
                             mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}::operator()(process::http::Request const&) const                      
                                                                                         
     
                             std::_Function_handler<process::Future<process::http::Response>
(process::http::Request const&), mesos::internal::master::Master::initialize()::{lambda(process::http::Request
const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)
   
                             std::function<process::Future<process::http::Response>
(process::http::Request const&)>::operator()(process::http::Request const&) const
                                                                                         
                            
                             process::ProcessBase::visit(process::HttpEvent const&)  
                                                                                         
                                                                                         
                  
                             process::HttpEvent::visit(process::EventVisitor*) const     
                                                                                         
                                                                                         
              
                             process::ProcessBase::serve(process::Event const&)      
                                                                                         
                                                                                         
                  
                             process::ProcessManager::resume(process::ProcessBase*)      
                                                                                         
                                                                                         
              
                             process::schedule(void*)                                    
                                                                                         
                                                                                         
              
                             start_thread                                                
                                                                                         
                                                                                         
              
}}

The top ~80 call sides pretty much all relate to Value_Range handling, and account for 90%
of the runtime of the mesos-master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message