activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nigro_franz <nigro....@gmail.com>
Subject Re: [DISCUSS] Artemis IOPS Limiter strategy
Date Fri, 12 May 2017 21:02:45 GMT
ahah np Michael!!!

Anyway I was really expecting some more spike compared to the old
TimedBuffer: Is one of the reasons why I wanted to put the IO compensation.
The new one seems to work well (with ASYNCIO) in this regard by the tests
I've done: I'm hoping it will help to remove that little spikes too!


2017-05-12 22:54 GMT+02:00 Michael André Pearce [via ActiveMQ] <
ml+s2283324n4726101h6@n4.nabble.com>:

> Correct, :) seems like tester error. Apologies
>
> Sent from my iPhone
>
> > On 12 May 2017, at 21:24, nigro_franz <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=4726101&i=0>> wrote:
> >
> > Hence the new TimedBuffer didn't change drastically the latencies?
> > I've got some graph after the stress test with a maxed IO, if you're
> > curious :)
> >
> > 2017-05-12 22:19 GMT+02:00 Michael André Pearce [via ActiveMQ] <
> > [hidden email] <http:///user/SendEmail.jtp?type=node&node=4726101&i=1>>:
>
> >
> >> Yup deleting the data on version change removes the issue.
> >>
> >> Sent from my iPhone
> >>
> >>> On 12 May 2017, at 20:54, Clebert Suconic <[hidden email]
> >> <http:///user/SendEmail.jtp?type=node&node=4726097&i=0>> wrote:
> >>>
> >>> Can you cleanup your data (as a test only?) rm -rf data.
> >>>
> >>>
> >>>
> >>>
> >>> perhaps the spikes you have is the compacting running?
> >>>
> >>> On Fri, May 12, 2017 at 3:51 PM, Michael André Pearce
> >>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=4726097&i=1>>
>
> >> wrote:
> >>>> All I'm doing is changing Artemis.profile of the instance to the
> >> different versions.
> >>>>
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On 12 May 2017, at 20:35, nigro_franz <[hidden email]
> >> <http:///user/SendEmail.jtp?type=node&node=4726097&i=2>> wrote:
> >>>>>
> >>>>> Too strange...have you tried with NIO too?It uses always the same
> >> buffer to
> >>>>> batch writes AFAIK....
> >>>>>
> >>>>> 2017-05-12 21:30 GMT+02:00 Michael André Pearce [via ActiveMQ]
<
> >>>>> [hidden email] <http:///user/SendEmail.jtp?
> type=node&node=4726097&i=3>>:
> >>
> >>>>>
> >>>>>> Ok so it isn't the times buffer changes. Just deployed 2.1.0
with
> and
> >>>>>> without the timed buffer changes. Running the same tests no
> >> difference.
> >>>>>>
> >>>>>> Must be something else. Causing the behaviour change.
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>>
> >>>>>>> On 12 May 2017, at 18:10, Michael André Pearce <[hidden
email]
> >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=0>>
wrote:
> >>>>>>>
> >>>>>>> I've left for the day now. If I get time over the weekend
I'll try
> >> see
> >>>>>> if I can make a build of 2.1.0 without that change and see if
it
> >> makes any
> >>>>>> difference.
> >>>>>>>
> >>>>>>> Can someone point me to the PR for that change, so I know
what I'm
> >>>>>> unpicking locally?
> >>>>>>>
> >>>>>>> Sent from my iPhone
> >>>>>>>
> >>>>>>>> On 12 May 2017, at 17:58, Clebert Suconic <[hidden
email]
> >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=1>>
wrote:
> >>>>>>>>
> >>>>>>>> As the only thing that could affect this is the Change
on timed
> >> buffer.
> >>>>>>>> Afaik
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden
email]
> >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=2>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I'm considering only keeping the pooled buffer part
and switch
> >> back to
> >>>>>> the
> >>>>>>>>> ole sleep or an improved sleep we had.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, May 12, 2017 at 12:49 PM Michael André
Pearce <
> >>>>>>>>> [hidden email] <http:///user/SendEmail.jtp?
> >> type=node&node=4726091&i=3>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> As it seems  I can't send images to mail list,
just sent to
>  you
> >> both
> >>>>>> via
> >>>>>>>>>> email. Some graphs we have comparing versions.
> >>>>>>>>>>
> >>>>>>>>>> Not sure what changes might cause it.
> >>>>>>>>>>
> >>>>>>>>>>> On 12 May 2017, at 17:37, Clebert Suconic
<[hidden email]
> >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=4>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> There is a class we use on producer.  TokenLimiter.
 Perhaps
> you
> >>>>>> could
> >>>>>>>>>>> reuse that one ?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> On Fri, May 12, 2017 at 11:00 AM nigro_franz
<[hidden email]
> >>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=5>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I was thinking of a similar solution
but I've discovered that
> >>>>>> couldn't
> >>>>>>>>>> work
> >>>>>>>>>>>> (in the old or the new TimedBuffer too),
because of the
> >>>>>>>>>>>> TimedBuffer::checkSize method that could
force a flush if the
> >> batch
> >>>>>>>>>> buffer
> >>>>>>>>>>>> if not big enough to receive new data,
going IOPS.
> >>>>>>>>>>>> Sadly TimedBuffer::checkSize is outside
any timeout, but
> >> depends on
> >>>>>> the
> >>>>>>>>>>>> writers.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That's why I've implemented the "compensation"
right after
> any
> >>>>>> flush,
> >>>>>>>>>> in
> >>>>>>>>>>>> order to work with forced flushes too:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/
> >>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/
> >>>>>> src/main/java/org/apache/activemq/artemis/core/io/
> >>>>>> buffer/TimedBuffer.java#L338
> >>>>>>>>>>>>
> >>>>>>>>>>>> Doing it on TimedBuffer::flush all the
flushes on disk will
> be
> >>>>>>>>>> compensated
> >>>>>>>>>>>> (half of the story: ASYNCIO is async
so depends on libAIO
> >>>>>> partially!)
> >>>>>>>>>>>> Regarding the IOPS computation I've
built this, as you've
> >>>>>> suggested:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/
> >>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/
> >>>>>> src/main/java/org/apache/activemq/artemis/core/io/
> >>>>>> buffer/TimedBuffer.java#L119
> >>>>>>>>>>>>
> >>>>>>>>>>>> The performance seems pretty good, it
compensates well but it
> >> is
> >>>>>> faster
> >>>>>>>>>>>> than
> >>>>>>>>>>>> the original version, limiting IOPS
too!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Franz
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> View this message in context:
> >>>>>>>>>>>>
> >>>>>>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> >>>>>> IOPS-Limiter-strategy-tp4725875p4726057.html
> >>>>>>>>>>>> Sent from the ActiveMQ - Dev mailing
list archive at
> >> Nabble.com.
> >>>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Clebert Suconic
> >>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Clebert Suconic
> >>>>>>>>>
> >>>>>>>> --
> >>>>>>>> Clebert Suconic
> >>>>>>
> >>>>>>
> >>>>>> ------------------------------
> >>>>>> If you reply to this email, your message will be added to the
> >> discussion
> >>>>>> below:
> >>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> >>>>>> IOPS-Limiter-strategy-tp4725875p4726091.html
> >>>>>> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy,
click
> >> here
> >>>>>> <
> >>
> >>>>>> .
> >>>>>> NAML
> >>>>>> <http://activemq.2283324.n4.nabble.com/template/
> >> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> >> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> >> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> >> nabble.view.web.template.NodeNamespace&breadcrumbs=
> >> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> >> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> View this message in context: http://activemq.2283324.n4.
> >> nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726093.html
> >>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> >>>
> >>>
> >>>
> >>> --
> >>> Clebert Suconic
> >>
> >>
> >> ------------------------------
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> >> IOPS-Limiter-strategy-tp4725875p4726097.html
> >> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click here
> >> <
> >> .
> >> NAML
> >> <http://activemq.2283324.n4.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >>
> >
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726099.html
> > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
> IOPS-Limiter-strategy-tp4725875p4726101.html
> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click here
> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4725875&code=bmlncm8uZnJhQGdtYWlsLmNvbXw0NzI1ODc1fC0zNzI3NjE2NQ==>
> .
> NAML
> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726102.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message