activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francesco Nigro <nigro....@gmail.com>
Subject Re: [DISCUSS] Artemis IOPS Limiter strategy
Date Fri, 12 May 2017 21:25:12 GMT
And the most weird thing is that I'm having better results with NIO then
ASYNCIO (with small sized messages) and I'm not getting why....

Time to go, 'm off; in Italy is 23:30 now..have fun guys and thanks: always
great stuff :)

2017-05-12 23:02 GMT+02:00 nigro_franz <nigro.fra@gmail.com>:

> ahah np Michael!!!
>
> Anyway I was really expecting some more spike compared to the old
> TimedBuffer: Is one of the reasons why I wanted to put the IO compensation.
> The new one seems to work well (with ASYNCIO) in this regard by the tests
> I've done: I'm hoping it will help to remove that little spikes too!
>
>
> 2017-05-12 22:54 GMT+02:00 Michael André Pearce [via ActiveMQ] <
> ml+s2283324n4726101h6@n4.nabble.com>:
>
> > Correct, :) seems like tester error. Apologies
> >
> > Sent from my iPhone
> >
> > > On 12 May 2017, at 21:24, nigro_franz <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=4726101&i=0>> wrote:
> > >
> > > Hence the new TimedBuffer didn't change drastically the latencies?
> > > I've got some graph after the stress test with a maxed IO, if you're
> > > curious :)
> > >
> > > 2017-05-12 22:19 GMT+02:00 Michael André Pearce [via ActiveMQ] <
> > > [hidden email] <http:///user/SendEmail.jtp?
> type=node&node=4726101&i=1>>:
> >
> > >
> > >> Yup deleting the data on version change removes the issue.
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 12 May 2017, at 20:54, Clebert Suconic <[hidden email]
> > >> <http:///user/SendEmail.jtp?type=node&node=4726097&i=0>>
wrote:
> > >>>
> > >>> Can you cleanup your data (as a test only?) rm -rf data.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> perhaps the spikes you have is the compacting running?
> > >>>
> > >>> On Fri, May 12, 2017 at 3:51 PM, Michael André Pearce
> > >>> <[hidden email] <http:///user/SendEmail.jtp?
> type=node&node=4726097&i=1>>
> >
> > >> wrote:
> > >>>> All I'm doing is changing Artemis.profile of the instance to the
> > >> different versions.
> > >>>>
> > >>>>
> > >>>> Sent from my iPhone
> > >>>>
> > >>>>> On 12 May 2017, at 20:35, nigro_franz <[hidden email]
> > >> <http:///user/SendEmail.jtp?type=node&node=4726097&i=2>>
wrote:
> > >>>>>
> > >>>>> Too strange...have you tried with NIO too?It uses always the
same
> > >> buffer to
> > >>>>> batch writes AFAIK....
> > >>>>>
> > >>>>> 2017-05-12 21:30 GMT+02:00 Michael André Pearce [via ActiveMQ]
<
> > >>>>> [hidden email] <http:///user/SendEmail.jtp?
> > type=node&node=4726097&i=3>>:
> > >>
> > >>>>>
> > >>>>>> Ok so it isn't the times buffer changes. Just deployed
2.1.0 with
> > and
> > >>>>>> without the timed buffer changes. Running the same tests
no
> > >> difference.
> > >>>>>>
> > >>>>>> Must be something else. Causing the behaviour change.
> > >>>>>>
> > >>>>>> Sent from my iPhone
> > >>>>>>
> > >>>>>>> On 12 May 2017, at 18:10, Michael André Pearce <[hidden
email]
> > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=0>>
wrote:
> > >>>>>>>
> > >>>>>>> I've left for the day now. If I get time over the weekend
I'll
> try
> > >> see
> > >>>>>> if I can make a build of 2.1.0 without that change and
see if it
> > >> makes any
> > >>>>>> difference.
> > >>>>>>>
> > >>>>>>> Can someone point me to the PR for that change, so
I know what
> I'm
> > >>>>>> unpicking locally?
> > >>>>>>>
> > >>>>>>> Sent from my iPhone
> > >>>>>>>
> > >>>>>>>> On 12 May 2017, at 17:58, Clebert Suconic <[hidden
email]
> > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=1>>
wrote:
> > >>>>>>>>
> > >>>>>>>> As the only thing that could affect this is the
Change on timed
> > >> buffer.
> > >>>>>>>> Afaik
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic
<[hidden email]
> > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=2>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I'm considering only keeping the pooled buffer
part and switch
> > >> back to
> > >>>>>> the
> > >>>>>>>>> ole sleep or an improved sleep we had.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Fri, May 12, 2017 at 12:49 PM Michael André
Pearce <
> > >>>>>>>>> [hidden email] <http:///user/SendEmail.jtp?
> > >> type=node&node=4726091&i=3>>
> > >>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> As it seems  I can't send images to mail
list, just sent to
> >  you
> > >> both
> > >>>>>> via
> > >>>>>>>>>> email. Some graphs we have comparing versions.
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure what changes might cause it.
> > >>>>>>>>>>
> > >>>>>>>>>>> On 12 May 2017, at 17:37, Clebert Suconic
<[hidden email]
> > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=4>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> There is a class we use on producer.
 TokenLimiter.  Perhaps
> > you
> > >>>>>> could
> > >>>>>>>>>>> reuse that one ?
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Fri, May 12, 2017 at 11:00 AM
nigro_franz <[hidden email]
> > >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=5>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I was thinking of a similar solution
but I've discovered
> that
> > >>>>>> couldn't
> > >>>>>>>>>> work
> > >>>>>>>>>>>> (in the old or the new TimedBuffer
too), because of the
> > >>>>>>>>>>>> TimedBuffer::checkSize method that
could force a flush if
> the
> > >> batch
> > >>>>>>>>>> buffer
> > >>>>>>>>>>>> if not big enough to receive new
data, going IOPS.
> > >>>>>>>>>>>> Sadly TimedBuffer::checkSize is
outside any timeout, but
> > >> depends on
> > >>>>>> the
> > >>>>>>>>>>>> writers.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> That's why I've implemented the
"compensation" right after
> > any
> > >>>>>> flush,
> > >>>>>>>>>> in
> > >>>>>>>>>>>> order to work with forced flushes
too:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/
> > >>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/
> > >>>>>> src/main/java/org/apache/activemq/artemis/core/io/
> > >>>>>> buffer/TimedBuffer.java#L338
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Doing it on TimedBuffer::flush
all the flushes on disk will
> > be
> > >>>>>>>>>> compensated
> > >>>>>>>>>>>> (half of the story: ASYNCIO is
async so depends on libAIO
> > >>>>>> partially!)
> > >>>>>>>>>>>> Regarding the IOPS computation
I've built this, as you've
> > >>>>>> suggested:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/
> > >>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/
> > >>>>>> src/main/java/org/apache/activemq/artemis/core/io/
> > >>>>>> buffer/TimedBuffer.java#L119
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The performance seems pretty good,
it compensates well but
> it
> > >> is
> > >>>>>> faster
> > >>>>>>>>>>>> than
> > >>>>>>>>>>>> the original version, limiting
IOPS too!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>> Franz
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> View this message in context:
> > >>>>>>>>>>>>
> > >>>>>>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> > >>>>>> IOPS-Limiter-strategy-tp4725875p4726057.html
> > >>>>>>>>>>>> Sent from the ActiveMQ - Dev mailing
list archive at
> > >> Nabble.com.
> > >>>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Clebert Suconic
> > >>>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> Clebert Suconic
> > >>>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Clebert Suconic
> > >>>>>>
> > >>>>>>
> > >>>>>> ------------------------------
> > >>>>>> If you reply to this email, your message will be added
to the
> > >> discussion
> > >>>>>> below:
> > >>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> > >>>>>> IOPS-Limiter-strategy-tp4725875p4726091.html
> > >>>>>> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy,
click
> > >> here
> > >>>>>> <
> > >>
> > >>>>>> .
> > >>>>>> NAML
> > >>>>>> <http://activemq.2283324.n4.nabble.com/template/
> > >> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > >> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > >> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > >> nabble.view.web.template.NodeNamespace&breadcrumbs=
> > >> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > >> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> View this message in context: http://activemq.2283324.n4.
> > >> nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-
> tp4725875p4726093.html
> > >>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Clebert Suconic
> > >>
> > >>
> > >> ------------------------------
> > >> If you reply to this email, your message will be added to the
> > discussion
> > >> below:
> > >> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> > >> IOPS-Limiter-strategy-tp4725875p4726097.html
> > >> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click
> here
> > >> <
> > >> .
> > >> NAML
> > >> <http://activemq.2283324.n4.nabble.com/template/
> > NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > nabble.view.web.template.NodeNamespace&breadcrumbs=
> > notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > >>
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context: http://activemq.2283324.n4.
> > nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726099.html
> > > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> > http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> > IOPS-Limiter-strategy-tp4725875p4726101.html
> > To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click here
> > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=
> unsubscribe_by_code&node=4725875&code=bmlncm8uZnJhQGdtYWlsLmNvbXw0Nz
> I1ODc1fC0zNzI3NjE2NQ==>
> > .
> > NAML
> > <http://activemq.2283324.n4.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726102.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message