mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guangya Liu <gyliu...@gmail.com>
Subject Re: Review Request 50017: WIP: Validated the resources when parsing it.
Date Fri, 15 Jul 2016 08:55:50 GMT


> On 七月 14, 2016, 5:56 p.m., Benjamin Mahler wrote:
> > include/mesos/resources.hpp, lines 135-147
> > <https://reviews.apache.org/r/50017/diff/1/?file=1443931#file1443931line135>
> >
> >     This suggests a change of semantics to `Resources`.
> >     
> >     Currently: `Resources` always refers to a valid collection resources. This function
would never return Some(Error) unless we change the semantics.
> >     
> >     If we want to remove the unnecessary validation, we can do this by making '`operator
+= (const Resources& that)`' avoid validating '`that`' since it's already valid. Currently
it just calls '`operator += (const Resource& r)`' which is unaware of '`r`' already being
valid since it comes from a '`Resources`'.
> >     
> >     However, we have to keep the validation in '`operator += (const Resource&
r)`', since '`r`' is just a protobuf coming from an arbitrary caller, it may be invalid and
we need to validate.
> >     
> >     Make sense?

Hi Ben, does there are any example in mesos code for `Resource` object come from protobuf?
I found that we are always using `Resource::parse` to get a `Resource` or `Resources` object
and I have added some validation logic in `Resource::parse` in when getting `Resource` or
`Resources` object.


- Guangya


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50017/#review142257
-----------------------------------------------------------


On 七月 14, 2016, 3:59 p.m., Guangya Liu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50017/
> -----------------------------------------------------------
> 
> (Updated 七月 14, 2016, 3:59 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Klaus Ma.
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The "validation" API was called in a huge number of times, but this
> was only needed when parsing resources and we do not need to do
> other validation when doing resources operations, such as add etc.
> 
> 
> Diffs
> -----
> 
>   include/mesos/resources.hpp a557e97c65194d4aad879fb88d8edefd1c95b8d8 
>   include/mesos/v1/resources.hpp a5ba8fec4c9c3643646308f75a4b28cefe0b3df3 
>   src/common/resources.cpp f6ff92b591c15bc8e93fd85e1896349c3a7bb968 
>   src/v1/resources.cpp 8c3f2d1c1529915a59d47fe37bb3fc7a3267079a 
> 
> Diff: https://reviews.apache.org/r/50017/diff/
> 
> 
> Testing
> -------
> 
> The `qcachegrind` result here: https://docs.google.com/document/d/1oilen04e8trIOgYbj-jxAd7esTRDaySll8y2BQ6p5nU/edit?usp=sharing
> 
> After the fix, the performance of `adding` resources increased 30+% when adding 50000
agents to the cluster.
> 
> Before fix:
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/30
> Using 50000 agents and 1 clients
> Added 1 clients in 47us
> **Added 50000 agents in 1.312497secs**
> Sorted 1 clients in 43us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/30 (1321 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/31
> Using 50000 agents and 50 clients
> Added 50 clients in 948us
> **Added 50000 agents in 1.325987secs**
> Sorted 50 clients in 1165us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/31 (1340 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/32
> Using 50000 agents and 100 clients
> Added 100 clients in 1697us
> **Added 50000 agents in 1.409478secs**
> Sorted 100 clients in 2876us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/32 (1432 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/33
> Using 50000 agents and 200 clients
> Added 200 clients in 4553us
> **Added 50000 agents in 1.371473secs**
> Sorted 200 clients in 5371us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/33 (1412 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/34
> Using 50000 agents and 500 clients
> Added 500 clients in 8836us
> **Added 50000 agents in 1.304245secs**
> Sorted 500 clients in 14697us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/34 (1387 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/35
> Using 50000 agents and 1000 clients
> Added 1000 clients in 19508us
> **Added 50000 agents in 1.270555secs**
> Sorted 1000 clients in 32575us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/35 (1433 ms)
> 
> 
> After the fix:
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/30
> Using 50000 agents and 1 clients
> Added 1 clients in 42us
> **Added 50000 agents in 891266us**
> Sorted 1 clients in 59us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/30 (902 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/31
> Using 50000 agents and 50 clients
> Added 50 clients in 933us
> **Added 50000 agents in 885006us**
> Sorted 50 clients in 1220us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/31 (899 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/32
> Using 50000 agents and 100 clients
> Added 100 clients in 1879us
> **Added 50000 agents in 903112us**
> Sorted 100 clients in 2800us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/32 (922 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/33
> Using 50000 agents and 200 clients
> Added 200 clients in 3893us
> **Added 50000 agents in 881240us**
> Sorted 200 clients in 5802us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/33 (912 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/34
> Using 50000 agents and 500 clients
> Added 500 clients in 10712us
> **Added 50000 agents in 877442us**
> Sorted 500 clients in 17887us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/34 (949 ms)
> [ RUN      ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/35
> Using 50000 agents and 1000 clients
> Added 1000 clients in 21472us
> **Added 50000 agents in 916653us**
> Sorted 1000 clients in 37369us
> [       OK ] AgentAndClientCount/Sorter_BENCHMARK_Test.FullSort/35 (1057 ms)
> 
> 
> Thanks,
> 
> Guangya Liu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message