commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phil Steitz <phil.ste...@gmail.com>
Subject Re: [math] Multithreaded performances
Date Wed, 20 Nov 2013 16:35:43 GMT
On 11/20/13, 7:26 AM, Romain Manni-Bucau wrote:
> next version (rewrite/fork):
> https://svn.apache.org/repos/asf/incubator/sirona/trunk/core/src/main/java/org/apache/sirona/counters/OptimizedStatistics.java
>
> was easier to centralize everything in a single class

That is exactly what I meant about the moment statistics.  They can
share state.  The problem is this approach eliminates the
pluggability and also cuts out the other stats.  We can probably
have it both ways by allowing you to turn off stuff you don't want
and just having the moment stats share state.

Phil
> Romain Manni-Bucau
> Twitter: @rmannibucau
> Blog: http://rmannibucau.wordpress.com/
> LinkedIn: http://fr.linkedin.com/in/rmannibucau
> Github: https://github.com/rmannibucau
>
>
>
> 2013/11/20 Phil Steitz <phil.steitz@gmail.com>:
>> On 11/20/13, 12:43 AM, Romain Manni-Bucau wrote:
>>> Hi
>>>
>>> A quick mail to give some feedbacks of my tests.
>>>
>>> I started to hack a bit to get rid of not used stats by sirona,
>>> typically I do ATM:
>>>
>>>         setSumsqImpl(NoopStat.INSTANCE);
>>>         setSumLogImpl(NoopStat.INSTANCE);
>>>         setGeoMeanImpl(NoopStat.INSTANCE);
>>>
>>> (NoopStat is a mock of StorelessUnivariateStatistic doijg nothing)
>>>
>>> Another point which could be improoved is the duplication of info
>>> accross sub StorelessUnivariateStatistic (typically n computed several
>>> times for instance).
>> Good point.  Its kind of funny that simplest way to solve the
>> problem you mention at the end is to remove the flexibility that you
>> use in the beginning  - i.e., to no longer use separate stats
>> instances to compute the bundled statistics.  The setup is the way
>> it is precisely so that you can plug in alternative impls.  I had
>> never thought of no-op-ing instances to suppress things, but it does
>> work.  Having stats share state data is a little tricky but in
>> theory possible.  The moment stats at least are set up to support
>> this.   Patches are welcome.  If you don't mind opening a JIRA to
>> suggest eliminating repeated computations that would be great.
>>
>> Phil
>>> Romain Manni-Bucau
>>> Twitter: @rmannibucau
>>> Blog: http://rmannibucau.wordpress.com/
>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>> Github: https://github.com/rmannibucau
>>>
>>>
>>>
>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>>>> On 11/6/13 9:05 AM, Romain Manni-Bucau wrote:
>>>>> Great!
>>>>>
>>>>> Btw not sure for sirona we oculd use it. One constraint on sirona-core
>>>>> is to stay self contained. We already shade math3 so shading pool2 too
>>>>> would start to create a big jar for this need. I'll try to bench
>>>>> deeper next week too.
>>>> OK - and any ideas you have about how to implement something
>>>> lightweight inside [math] much appreciated.
>>>>
>>>> Phil
>>>>> Romain Manni-Bucau
>>>>> Twitter: @rmannibucau
>>>>> Blog: http://rmannibucau.wordpress.com/
>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>>>> Github: https://github.com/rmannibucau
>>>>>
>>>>>
>>>>>
>>>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>>>>>> On 11/6/13 8:47 AM, Romain Manni-Bucau wrote:
>>>>>>> well pool are based on locks so I'm not sure (it would need deep
>>>>>>> benchs on a real app) it does worth it
>>>>>> Commons pool2 uses pretty lightweight locking and using a pool of
>>>>>> instances achieves the basic objective of reducing contention for
>>>>>> the single sync lock on one SummaryStatistics object.   I bet it
>>>>>> would improve throughput over the single-instance approach if
>>>>>> maxActive, maxIdle were tuned.  If I get some time to play with
>>>>>> this, I will report back with some benchmarks.
>>>>>>
>>>>>> Phil
>>>>>>> Romain Manni-Bucau
>>>>>>> Twitter: @rmannibucau
>>>>>>> Blog: http://rmannibucau.wordpress.com/
>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>>>>>> Github: https://github.com/rmannibucau
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>>>>>>>> On 11/5/13 11:26 PM, Romain Manni-Bucau wrote:
>>>>>>>>> Hehe, right.
>>>>>>>>>
>>>>>>>>> I looked a bit more today and LongAdder is only a part
of the
>>>>>>>>> solution. The stat computation still needs to lock to
get acces to
>>>>>>>>> previous values (N -> N+1). Basically the gain wouldn't
be as
>>>>>>>>> important as I thought :(.
>>>>>>>> Right, but I think your original idea of maintaining a pool
of
>>>>>>>> instances (fewer that one per thread) to be periodically
aggregated
>>>>>>>> is a good one.  See below.
>>>>>>>>> As I said before we'll wait a bit to gather feedbacks,
if it blocks
>>>>>>>>> I'll come back trying to find + propose a solution.
>>>>>>>>>
>>>>>>>>> Thanks in all cases for your answers!
>>>>>>>> A workaround that I have started playing with (partly for
other
>>>>>>>> benchmarking reasons) might be to actually use a pool for
the stats
>>>>>>>> objects that the monitoring threads use.  Using a pool would
allow
>>>>>>>> you to monitor and tune the parameters.  We now have (well,
once the
>>>>>>>> VOTE in progress completes :) a decently performing pool
>>>>>>>> implementation.  The tricky bit is locking the instances
during
>>>>>>>> aggregation.  One way to handle this would be to have the
factory's
>>>>>>>> passivate method and the aggregation thread contend for locks
on the
>>>>>>>> pooled stats instances.  The only contention would be when
>>>>>>>> aggregation is copying individual instances and contention
would be
>>>>>>>> with at most one client thread (waiting to proceed in passivate).
>>>>>>>>
>>>>>>>> Phil
>>>>>>>>> Romain Manni-Bucau
>>>>>>>>> Twitter: @rmannibucau
>>>>>>>>> Blog: http://rmannibucau.wordpress.com/
>>>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>>>>>>>>> Github: https://github.com/rmannibucau
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/11/5 Phil Steitz <phil.steitz@gmail.com>:
>>>>>>>>>> On 11/5/13 9:57 AM, Romain Manni-Bucau wrote:
>>>>>>>>>>> @Phil: hmm can be but the framework would create
its own overhead which
>>>>>>>>>>> would be avoided with a dedicated solution, no?
Well thought gain was great
>>>>>>>>>>> for small investment but ok to postpone it
>>>>>>>>>> As I said, patches welcome.  Go for it.  My point
about the
>>>>>>>>>> framework was that when you actually get this implemented
inside,
>>>>>>>>>> e.g. SummaryStatistics,  you will have built a mini-framework.
>>>>>>>>>> Whatever overhead it has, it will have ;)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Phil
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Le 5 nov. 2013 18:54, "Romain Manni-Bucau" <rmannibucau@gmail.com>
a écrit :
>>>>>>>>>>>
>>>>>>>>>>>> Well I didnt test sirona in prod but when
using jamon (same kind of
>>>>>>>>>>>> framework) locks were creating a serious
overhead on some benches. Not the
>>>>>>>>>>>> most important but enough to try to solve
it.
>>>>>>>>>>>>
>>>>>>>>>>>> That said we are not yet in 1.0 so Im ok
to wait for more serious
>>>>>>>>>>>> feedbacks if you think it is better
>>>>>>>>>>>> Le 5 nov. 2013 18:48, "Ted Dunning" <ted.dunning@gmail.com>
a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Nov 4, 2013 at 10:09 PM, Romain
Manni-Bucau
>>>>>>>>>>>>> <rmannibucau@gmail.com>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oh sorry, that's what I said early,
in a real app no or not enough to
>>>>>>>>>>>>> be an
>>>>>>>>>>>>>> issue buy on simple apps or very
high thrououtput apps yes.
>>>>>>>>>>>>>>  Le 5 nov. 2013 07:00, "Ted Dunning"
<ted.dunning@gmail.com> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That isn't what I meant.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you really think that more
than one metric has to update
>>>>>>>>>>>>> (increment,
>>>>>>>>>>>>>>> say) at precisely the same time?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>> I realize that is what you said.  Do
you have any serious examples where
>>>>>>>>>>>>> metrics have to be updated all or nothing?
>>>>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> For additional commands, e-mail: dev-help@commons.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message