mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mesos ReviewBot" <...@mesos.apache.org>
Subject Re: Review Request 18758: Performance optimizations to the Registrar.
Date Thu, 06 Mar 2014 20:23:18 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18758/#review36421
-----------------------------------------------------------


Bad patch!

Reviews applied: [18758]

Failed command: git apply --index 18758.patch

Error:
 error: patch failed: src/master/registrar.cpp:108
error: src/master/registrar.cpp: patch does not apply


- Mesos ReviewBot


On March 6, 2014, 8:13 p.m., Ben Mahler wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18758/
> -----------------------------------------------------------
> 
> (Updated March 6, 2014, 8:13 p.m.)
> 
> 
> Review request for mesos, Benjamin Hindman and Vinod Kone.
> 
> 
> Bugs: MESOS-764
>     https://issues.apache.org/jira/browse/MESOS-764
> 
> 
> Repository: mesos-git
> 
> 
> Description
> -------
> 
> The bottleneck in the Registrar is the copying of the Registry. For a large number of
slaves and operations, this becomes prohibitively expensive. This patch therefore eliminates
copying when applying the operations. Optimizations to Future also eliminated copying in _update.
> 
> Further optimizations will be explored for larger clusters, this was currently tested
with 10,000 slaves.
> 
> 
> Diffs
> -----
> 
>   src/master/registrar.cpp 37337c07b24a96e71910b7c83085d159361a1188 
> 
> Diff: https://reviews.apache.org/r/18758/diff/
> 
> 
> Testing
> -------
> 
> Prior to this patch and the Future::get optimizations, the performance benchmark I wrote
took over an hour for 10,000 slaves.
> 
> With these optimizations 10,000 slaves can be handled quite efficiently:
> 
> Key numbers with gcc 4.1.2 on a 2.4GHz 12 core machine:
> 
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
> 
> 
> [ RUN      ] RegistrarTest.DISABLED_performance
> I0304 23:36:55.685456 33261 registrar.cpp:218] Recovering registrar
> I0304 23:36:55.801048 33252 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 1 operations in: 1.235673ms
>     Stored Registry in: 36.199255ms
> I0304 23:36:55.837477 33271 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 1 promises in: 66813ns
> I0304 23:36:55.972564 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 1 operations in: 1.573455ms
>     Stored Registry in: 30.215382ms
> I0304 23:36:56.002907 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 1 promises in: 113129ns
> I0304 23:36:56.003094 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 146 operations in: 8.579203ms
>     Stored Registry in: 174.710644ms
> I0304 23:36:56.179323 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 146 promises in: 4.656175ms
> I0304 23:36:56.184154 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 574 operations in: 73.828182ms
>     Stored Registry in: 527.540062ms
> I0304 23:36:56.744689 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 574 promises in: 13.873625ms
> I0304 23:36:56.760572 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 2400 operations in: 907.070761ms
>     Stored Registry in: 268.604653ms
> I0304 23:36:57.778074 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 2400 promises in: 69.867013ms
> I0304 23:36:57.858157 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 6879 operations in: 7.910334133secs
>     Stored Registry in: 990.759579ms
> I0304 23:37:05.795042 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 6879 promises in: 221.701612ms
> I0304 23:37:06.024024 33235 registrar_tests.cpp:229] Admitted 10,000 slaves in 10.04498586secs
> I0304 23:37:06.125613 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 1 operations in: 852.076566ms
>     Stored Registry in: 1.384153523secs
> I0304 23:37:07.576215 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 1 promises in: 191.668241ms
> I0304 23:37:07.768013 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 993 operations in: 923.901654ms
>     Stored Registry in: 953.378897ms
> I0304 23:37:08.827229 33266 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 993 promises in: 164.437762ms
> I0304 23:37:08.992169 33266 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 9006 operations in: 6.081593589secs
>     Stored Registry in: 896.142182ms
> I0304 23:37:15.397238 33257 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 9006 promises in: 427.267246ms
> I0304 23:37:15.853411 33235 registrar_tests.cpp:237] Readmitted 10,000 slaves in 9.829011689secs
> I0304 23:37:15.876085 33257 registrar.cpp:218] Recovering registrar
> I0304 23:37:15.876322 33235 registrar_tests.cpp:243] Recovered 10,000 slaves in 22.715892ms
> I0304 23:37:15.876656 33264 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 1 operations in: 819.303422ms
>     Stored Registry in: 1.383689258secs
>     I0304 23:37:17.329473 33264 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 1 promises in: 189.505006ms
> I0304 23:37:17.541276 33264 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 1 operations in: 921.761221ms
>     Stored Registry in: 986.688995ms
> I0304 23:37:18.588706 33264 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 1 promises in: 179.771104ms
> I0304 23:37:18.768606 33264 registrar.cpp:381] Attempting to update the 'registry'
>     Applied 9998 operations in: 441.652539ms
>     Stored Registry in: 209.866548ms
> I0304 23:37:19.298931 33265 registrar.cpp:425] Successfully updated 'registry'
>     Transitioned 9998 promises in: 154.775647ms
> I0304 23:37:19.453868 33235 registrar_tests.cpp:251] Removed 10,000 slaves in 3.577429332secs
> [       OK ] RegistrarTest.DISABLED_performance (23791 ms)
> 
> 
> Thanks,
> 
> Ben Mahler
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message