commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Manni-Bucau <rmannibu...@gmail.com>
Subject Re: [math] Multithreaded performances
Date Wed, 20 Nov 2013 15:26:59 GMT
next version (rewrite/fork):
https://svn.apache.org/repos/asf/incubator/sirona/trunk/core/src/main/java/org/apache/sirona/counters/OptimizedStatistics.java

was easier to centralize everything in a single class
Romain Manni-Bucau
Twitter: @rmannibucau
Blog: http://rmannibucau.wordpress.com/
LinkedIn: http://fr.linkedin.com/in/rmannibucau
Github: https://github.com/rmannibucau



2013/11/20 Phil Steitz <phil.steitz@gmail.com>:
> On 11/20/13, 12:43 AM, Romain Manni-Bucau wrote:
>> Hi
>>
>> A quick mail to give some feedbacks of my tests.
>>
>> I started to hack a bit to get rid of not used stats by sirona,
>> typically I do ATM:
>>
>>         setSumsqImpl(NoopStat.INSTANCE);
>>         setSumLogImpl(NoopStat.INSTANCE);
>>         setGeoMeanImpl(NoopStat.INSTANCE);
>>
>> (NoopStat is a mock of StorelessUnivariateStatistic doijg nothing)
>>
>> Another point which could be improoved is the duplication of info
>> accross sub StorelessUnivariateStatistic (typically n computed several
>> times for instance).
>
> Good point.  Its kind of funny that simplest way to solve the
> problem you mention at the end is to remove the flexibility that you
> use in the beginning  - i.e., to no longer use separate stats
> instances to compute the bundled statistics.  The setup is the way
> it is precisely so that you can plug in alternative impls.  I had
> never thought of no-op-ing instances to suppress things, but it does
> work.  Having stats share state data is a little tricky but in
> theory possible.  The moment stats at least are set up to support
> this.   Patches are welcome.  If you don't mind opening a JIRA to
> suggest eliminating repeated computations that would be great.
>
> Phil
>> Romain Manni-Bucau
>> Twitter: @rmannibucau
>> Blog: http://rmannibucau.wordpress.com/
>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> Github: https://github.com/rmannibucau
>>
>>
>>
>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>>> On 11/6/13 9:05 AM, Romain Manni-Bucau wrote:
>>>> Great!
>>>>
>>>> Btw not sure for sirona we oculd use it. One constraint on sirona-core
>>>> is to stay self contained. We already shade math3 so shading pool2 too
>>>> would start to create a big jar for this need. I'll try to bench
>>>> deeper next week too.
>>> OK - and any ideas you have about how to implement something
>>> lightweight inside [math] much appreciated.
>>>
>>> Phil
>>>> Romain Manni-Bucau
>>>> Twitter: @rmannibucau
>>>> Blog: http://rmannibucau.wordpress.com/
>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>>> Github: https://github.com/rmannibucau
>>>>
>>>>
>>>>
>>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>>>>> On 11/6/13 8:47 AM, Romain Manni-Bucau wrote:
>>>>>> well pool are based on locks so I'm not sure (it would need deep
>>>>>> benchs on a real app) it does worth it
>>>>> Commons pool2 uses pretty lightweight locking and using a pool of
>>>>> instances achieves the basic objective of reducing contention for
>>>>> the single sync lock on one SummaryStatistics object.   I bet it
>>>>> would improve throughput over the single-instance approach if
>>>>> maxActive, maxIdle were tuned.  If I get some time to play with
>>>>> this, I will report back with some benchmarks.
>>>>>
>>>>> Phil
>>>>>> Romain Manni-Bucau
>>>>>> Twitter: @rmannibucau
>>>>>> Blog: http://rmannibucau.wordpress.com/
>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>>>>> Github: https://github.com/rmannibucau
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>>>>>>> On 11/5/13 11:26 PM, Romain Manni-Bucau wrote:
>>>>>>>> Hehe, right.
>>>>>>>>
>>>>>>>> I looked a bit more today and LongAdder is only a part of
the
>>>>>>>> solution. The stat computation still needs to lock to get
acces to
>>>>>>>> previous values (N -> N+1). Basically the gain wouldn't
be as
>>>>>>>> important as I thought :(.
>>>>>>> Right, but I think your original idea of maintaining a pool of
>>>>>>> instances (fewer that one per thread) to be periodically aggregated
>>>>>>> is a good one.  See below.
>>>>>>>> As I said before we'll wait a bit to gather feedbacks, if
it blocks
>>>>>>>> I'll come back trying to find + propose a solution.
>>>>>>>>
>>>>>>>> Thanks in all cases for your answers!
>>>>>>> A workaround that I have started playing with (partly for other
>>>>>>> benchmarking reasons) might be to actually use a pool for the
stats
>>>>>>> objects that the monitoring threads use.  Using a pool would
allow
>>>>>>> you to monitor and tune the parameters.  We now have (well, once
the
>>>>>>> VOTE in progress completes :) a decently performing pool
>>>>>>> implementation.  The tricky bit is locking the instances during
>>>>>>> aggregation.  One way to handle this would be to have the factory's
>>>>>>> passivate method and the aggregation thread contend for locks
on the
>>>>>>> pooled stats instances.  The only contention would be when
>>>>>>> aggregation is copying individual instances and contention would
be
>>>>>>> with at most one client thread (waiting to proceed in passivate).
>>>>>>>
>>>>>>> Phil
>>>>>>>> Romain Manni-Bucau
>>>>>>>> Twitter: @rmannibucau
>>>>>>>> Blog: http://rmannibucau.wordpress.com/
>>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>>>>>>> Github: https://github.com/rmannibucau
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/11/5 Phil Steitz <phil.steitz@gmail.com>:
>>>>>>>>> On 11/5/13 9:57 AM, Romain Manni-Bucau wrote:
>>>>>>>>>> @Phil: hmm can be but the framework would create
its own overhead which
>>>>>>>>>> would be avoided with a dedicated solution, no? Well
thought gain was great
>>>>>>>>>> for small investment but ok to postpone it
>>>>>>>>> As I said, patches welcome.  Go for it.  My point about
the
>>>>>>>>> framework was that when you actually get this implemented
inside,
>>>>>>>>> e.g. SummaryStatistics,  you will have built a mini-framework.
>>>>>>>>> Whatever overhead it has, it will have ;)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Phil
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Le 5 nov. 2013 18:54, "Romain Manni-Bucau" <rmannibucau@gmail.com>
a écrit :
>>>>>>>>>>
>>>>>>>>>>> Well I didnt test sirona in prod but when using
jamon (same kind of
>>>>>>>>>>> framework) locks were creating a serious overhead
on some benches. Not the
>>>>>>>>>>> most important but enough to try to solve it.
>>>>>>>>>>>
>>>>>>>>>>> That said we are not yet in 1.0 so Im ok to wait
for more serious
>>>>>>>>>>> feedbacks if you think it is better
>>>>>>>>>>> Le 5 nov. 2013 18:48, "Ted Dunning" <ted.dunning@gmail.com>
a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Nov 4, 2013 at 10:09 PM, Romain Manni-Bucau
>>>>>>>>>>>> <rmannibucau@gmail.com>wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Oh sorry, that's what I said early, in
a real app no or not enough to
>>>>>>>>>>>> be an
>>>>>>>>>>>>> issue buy on simple apps or very high
thrououtput apps yes.
>>>>>>>>>>>>>  Le 5 nov. 2013 07:00, "Ted Dunning"
<ted.dunning@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>>> That isn't what I meant.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you really think that more than
one metric has to update
>>>>>>>>>>>> (increment,
>>>>>>>>>>>>>> say) at precisely the same time?
>>>>>>>>>>>>>>
>>>>>>>>>>>> I realize that is what you said.  Do you
have any serious examples where
>>>>>>>>>>>> metrics have to be updated all or nothing?
>>>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> For additional commands, e-mail: dev-help@commons.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message