commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Manni-Bucau <rmannibu...@gmail.com>
Subject Re: [math] Multithreaded performances
Date Thu, 21 Nov 2013 10:14:13 GMT
Hi

first maybe the thread should move over sirona list (I wouldn't bother
commons ;)): http://sirona.incubator.apache.org/mail-lists.html

about hot topics I think we have:
1) Gui enhancement (a jira is about using another js library to draw
graphes and get an analysis/correlation view, another topic is the GUI
configuration: how to make dashboard etc...)
2) Storage backends (we support mainly cassandra and in memory ATM but
a rotating JDBC impl could be great too)
3) Alerting is not yet done and needs to be in final versions
4) Agent/Plugins enhancement (we have a few today)
5) Surely a better configuration
6) Counter implementation with less locks (even if thanks to the stat
impl we use we reduced the lock duration we still rely too much on
lock IMO and I'm sure we can do better).
7) etc.. ;)


It is a new project so all help is welcomed!

PS: we have an IRC channel on freenode: #apache-sirona




Romain Manni-Bucau
Twitter: @rmannibucau
Blog: http://rmannibucau.wordpress.com/
LinkedIn: http://fr.linkedin.com/in/rmannibucau
Github: https://github.com/rmannibucau



2013/11/21 Thomas Neidhart <thomas.neidhart@gmail.com>:
> I did take a look at sirona, and it looks really nice.
> Whats the best way to contribute to it? Any open issues you would like to
> work somebody on?
>
> Thomas
>
>
> On Wed, Nov 20, 2013 at 5:38 PM, Romain Manni-Bucau
> <rmannibucau@gmail.com>wrote:
>
>> yep but this is not enough, almost each "component" is doing n++. This
>> is fast but since done under a lock if it can be removed it should.
>> Romain Manni-Bucau
>> Twitter: @rmannibucau
>> Blog: http://rmannibucau.wordpress.com/
>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> Github: https://github.com/rmannibucau
>>
>>
>>
>> 2013/11/20 Phil Steitz <phil.steitz@gmail.com>:
>> > On 11/20/13, 7:26 AM, Romain Manni-Bucau wrote:
>> >> next version (rewrite/fork):
>> >>
>> https://svn.apache.org/repos/asf/incubator/sirona/trunk/core/src/main/java/org/apache/sirona/counters/OptimizedStatistics.java
>> >>
>> >> was easier to centralize everything in a single class
>> >
>> > That is exactly what I meant about the moment statistics.  They can
>> > share state.  The problem is this approach eliminates the
>> > pluggability and also cuts out the other stats.  We can probably
>> > have it both ways by allowing you to turn off stuff you don't want
>> > and just having the moment stats share state.
>> >
>> > Phil
>> >> Romain Manni-Bucau
>> >> Twitter: @rmannibucau
>> >> Blog: http://rmannibucau.wordpress.com/
>> >> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> >> Github: https://github.com/rmannibucau
>> >>
>> >>
>> >>
>> >> 2013/11/20 Phil Steitz <phil.steitz@gmail.com>:
>> >>> On 11/20/13, 12:43 AM, Romain Manni-Bucau wrote:
>> >>>> Hi
>> >>>>
>> >>>> A quick mail to give some feedbacks of my tests.
>> >>>>
>> >>>> I started to hack a bit to get rid of not used stats by sirona,
>> >>>> typically I do ATM:
>> >>>>
>> >>>>         setSumsqImpl(NoopStat.INSTANCE);
>> >>>>         setSumLogImpl(NoopStat.INSTANCE);
>> >>>>         setGeoMeanImpl(NoopStat.INSTANCE);
>> >>>>
>> >>>> (NoopStat is a mock of StorelessUnivariateStatistic doijg nothing)
>> >>>>
>> >>>> Another point which could be improoved is the duplication of info
>> >>>> accross sub StorelessUnivariateStatistic (typically n computed several
>> >>>> times for instance).
>> >>> Good point.  Its kind of funny that simplest way to solve the
>> >>> problem you mention at the end is to remove the flexibility that you
>> >>> use in the beginning  - i.e., to no longer use separate stats
>> >>> instances to compute the bundled statistics.  The setup is the way
>> >>> it is precisely so that you can plug in alternative impls.  I had
>> >>> never thought of no-op-ing instances to suppress things, but it does
>> >>> work.  Having stats share state data is a little tricky but in
>> >>> theory possible.  The moment stats at least are set up to support
>> >>> this.   Patches are welcome.  If you don't mind opening a JIRA to
>> >>> suggest eliminating repeated computations that would be great.
>> >>>
>> >>> Phil
>> >>>> Romain Manni-Bucau
>> >>>> Twitter: @rmannibucau
>> >>>> Blog: http://rmannibucau.wordpress.com/
>> >>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> >>>> Github: https://github.com/rmannibucau
>> >>>>
>> >>>>
>> >>>>
>> >>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>> >>>>> On 11/6/13 9:05 AM, Romain Manni-Bucau wrote:
>> >>>>>> Great!
>> >>>>>>
>> >>>>>> Btw not sure for sirona we oculd use it. One constraint
on
>> sirona-core
>> >>>>>> is to stay self contained. We already shade math3 so shading
pool2
>> too
>> >>>>>> would start to create a big jar for this need. I'll try
to bench
>> >>>>>> deeper next week too.
>> >>>>> OK - and any ideas you have about how to implement something
>> >>>>> lightweight inside [math] much appreciated.
>> >>>>>
>> >>>>> Phil
>> >>>>>> Romain Manni-Bucau
>> >>>>>> Twitter: @rmannibucau
>> >>>>>> Blog: http://rmannibucau.wordpress.com/
>> >>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> >>>>>> Github: https://github.com/rmannibucau
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>> >>>>>>> On 11/6/13 8:47 AM, Romain Manni-Bucau wrote:
>> >>>>>>>> well pool are based on locks so I'm not sure (it
would need deep
>> >>>>>>>> benchs on a real app) it does worth it
>> >>>>>>> Commons pool2 uses pretty lightweight locking and using
a pool of
>> >>>>>>> instances achieves the basic objective of reducing contention
for
>> >>>>>>> the single sync lock on one SummaryStatistics object.
  I bet it
>> >>>>>>> would improve throughput over the single-instance approach
if
>> >>>>>>> maxActive, maxIdle were tuned.  If I get some time to
play with
>> >>>>>>> this, I will report back with some benchmarks.
>> >>>>>>>
>> >>>>>>> Phil
>> >>>>>>>> Romain Manni-Bucau
>> >>>>>>>> Twitter: @rmannibucau
>> >>>>>>>> Blog: http://rmannibucau.wordpress.com/
>> >>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> >>>>>>>> Github: https://github.com/rmannibucau
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 2013/11/6 Phil Steitz <phil.steitz@gmail.com>:
>> >>>>>>>>> On 11/5/13 11:26 PM, Romain Manni-Bucau wrote:
>> >>>>>>>>>> Hehe, right.
>> >>>>>>>>>>
>> >>>>>>>>>> I looked a bit more today and LongAdder
is only a part of the
>> >>>>>>>>>> solution. The stat computation still needs
to lock to get acces
>> to
>> >>>>>>>>>> previous values (N -> N+1). Basically
the gain wouldn't be as
>> >>>>>>>>>> important as I thought :(.
>> >>>>>>>>> Right, but I think your original idea of maintaining
a pool of
>> >>>>>>>>> instances (fewer that one per thread) to be
periodically
>> aggregated
>> >>>>>>>>> is a good one.  See below.
>> >>>>>>>>>> As I said before we'll wait a bit to gather
feedbacks, if it
>> blocks
>> >>>>>>>>>> I'll come back trying to find + propose
a solution.
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks in all cases for your answers!
>> >>>>>>>>> A workaround that I have started playing with
(partly for other
>> >>>>>>>>> benchmarking reasons) might be to actually use
a pool for the
>> stats
>> >>>>>>>>> objects that the monitoring threads use.  Using
a pool would
>> allow
>> >>>>>>>>> you to monitor and tune the parameters.  We
now have (well, once
>> the
>> >>>>>>>>> VOTE in progress completes :) a decently performing
pool
>> >>>>>>>>> implementation.  The tricky bit is locking the
instances during
>> >>>>>>>>> aggregation.  One way to handle this would be
to have the
>> factory's
>> >>>>>>>>> passivate method and the aggregation thread
contend for locks on
>> the
>> >>>>>>>>> pooled stats instances.  The only contention
would be when
>> >>>>>>>>> aggregation is copying individual instances
and contention would
>> be
>> >>>>>>>>> with at most one client thread (waiting to proceed
in passivate).
>> >>>>>>>>>
>> >>>>>>>>> Phil
>> >>>>>>>>>> Romain Manni-Bucau
>> >>>>>>>>>> Twitter: @rmannibucau
>> >>>>>>>>>> Blog: http://rmannibucau.wordpress.com/
>> >>>>>>>>>> LinkedIn: http://fr.linkedin.com/in/rmannibucau
>> >>>>>>>>>> Github: https://github.com/rmannibucau
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> 2013/11/5 Phil Steitz <phil.steitz@gmail.com>:
>> >>>>>>>>>>> On 11/5/13 9:57 AM, Romain Manni-Bucau
wrote:
>> >>>>>>>>>>>> @Phil: hmm can be but the framework
would create its own
>> overhead which
>> >>>>>>>>>>>> would be avoided with a dedicated
solution, no? Well thought
>> gain was great
>> >>>>>>>>>>>> for small investment but ok to postpone
it
>> >>>>>>>>>>> As I said, patches welcome.  Go for
it.  My point about the
>> >>>>>>>>>>> framework was that when you actually
get this implemented
>> inside,
>> >>>>>>>>>>> e.g. SummaryStatistics,  you will have
built a mini-framework.
>> >>>>>>>>>>> Whatever overhead it has, it will have
;)
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Phil
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Le 5 nov. 2013 18:54, "Romain Manni-Bucau"
<
>> rmannibucau@gmail.com> a écrit :
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Well I didnt test sirona in
prod but when using jamon (same
>> kind of
>> >>>>>>>>>>>>> framework) locks were creating
a serious overhead on some
>> benches. Not the
>> >>>>>>>>>>>>> most important but enough to
try to solve it.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> That said we are not yet in
1.0 so Im ok to wait for more
>> serious
>> >>>>>>>>>>>>> feedbacks if you think it is
better
>> >>>>>>>>>>>>> Le 5 nov. 2013 18:48, "Ted Dunning"
<ted.dunning@gmail.com>
>> a écrit :
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Mon, Nov 4, 2013 at 10:09
PM, Romain Manni-Bucau
>> >>>>>>>>>>>>>> <rmannibucau@gmail.com>wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Oh sorry, that's what
I said early, in a real app no or
>> not enough to
>> >>>>>>>>>>>>>> be an
>> >>>>>>>>>>>>>>> issue buy on simple
apps or very high thrououtput apps yes.
>> >>>>>>>>>>>>>>>  Le 5 nov. 2013 07:00,
"Ted Dunning" <
>> ted.dunning@gmail.com> a écrit :
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> That isn't what
I meant.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Do you really think
that more than one metric has to
>> update
>> >>>>>>>>>>>>>> (increment,
>> >>>>>>>>>>>>>>>> say) at precisely
the same time?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I realize that is what you
said.  Do you have any serious
>> examples where
>> >>>>>>>>>>>>>> metrics have to be updated
all or nothing?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>>>>>
>> >>>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>>>
>> >>>>>>
>> ---------------------------------------------------------------------
>> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>>
>> >>>>>>
>> >>>>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>>
>> >>>>
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >>> For additional commands, e-mail: dev-help@commons.apache.org
>> >>>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> >> For additional commands, e-mail: dev-help@commons.apache.org
>> >>
>> >>
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> > For additional commands, e-mail: dev-help@commons.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> For additional commands, e-mail: dev-help@commons.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message