activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael André Pearce <michael.andre.pea...@me.com>
Subject Re: [DISCUSS] Artemis IOPS Limiter strategy
Date Fri, 12 May 2017 21:06:46 GMT
Correct, :) seems like tester error. Apologies 

Sent from my iPhone

> On 12 May 2017, at 21:24, nigro_franz <nigro.fra@gmail.com> wrote:
> 
> Hence the new TimedBuffer didn't change drastically the latencies?
> I've got some graph after the stress test with a maxed IO, if you're
> curious :)
> 
> 2017-05-12 22:19 GMT+02:00 Michael André Pearce [via ActiveMQ] <
> ml+s2283324n4726097h82@n4.nabble.com>:
> 
>> Yup deleting the data on version change removes the issue.
>> 
>> Sent from my iPhone
>> 
>>> On 12 May 2017, at 20:54, Clebert Suconic <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=4726097&i=0>> wrote:
>>> 
>>> Can you cleanup your data (as a test only?) rm -rf data.
>>> 
>>> 
>>> 
>>> 
>>> perhaps the spikes you have is the compacting running?
>>> 
>>> On Fri, May 12, 2017 at 3:51 PM, Michael André Pearce
>>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=4726097&i=1>>
>> wrote:
>>>> All I'm doing is changing Artemis.profile of the instance to the
>> different versions.
>>>> 
>>>> 
>>>> Sent from my iPhone
>>>> 
>>>>> On 12 May 2017, at 20:35, nigro_franz <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=4726097&i=2>> wrote:
>>>>> 
>>>>> Too strange...have you tried with NIO too?It uses always the same
>> buffer to
>>>>> batch writes AFAIK....
>>>>> 
>>>>> 2017-05-12 21:30 GMT+02:00 Michael André Pearce [via ActiveMQ] <
>>>>> [hidden email] <http:///user/SendEmail.jtp?type=node&node=4726097&i=3>>:
>> 
>>>>> 
>>>>>> Ok so it isn't the times buffer changes. Just deployed 2.1.0 with
and
>>>>>> without the timed buffer changes. Running the same tests no
>> difference.
>>>>>> 
>>>>>> Must be something else. Causing the behaviour change.
>>>>>> 
>>>>>> Sent from my iPhone
>>>>>> 
>>>>>>> On 12 May 2017, at 18:10, Michael André Pearce <[hidden email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=0>>
wrote:
>>>>>>> 
>>>>>>> I've left for the day now. If I get time over the weekend I'll
try
>> see
>>>>>> if I can make a build of 2.1.0 without that change and see if it
>> makes any
>>>>>> difference.
>>>>>>> 
>>>>>>> Can someone point me to the PR for that change, so I know what
I'm
>>>>>> unpicking locally?
>>>>>>> 
>>>>>>> Sent from my iPhone
>>>>>>> 
>>>>>>>> On 12 May 2017, at 17:58, Clebert Suconic <[hidden email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=1>>
wrote:
>>>>>>>> 
>>>>>>>> As the only thing that could affect this is the Change on
timed
>> buffer.
>>>>>>>> Afaik
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden
email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=2>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I'm considering only keeping the pooled buffer part and
switch
>> back to
>>>>>> the
>>>>>>>>> ole sleep or an improved sleep we had.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Fri, May 12, 2017 at 12:49 PM Michael André Pearce
<
>>>>>>>>> [hidden email] <http:///user/SendEmail.jtp?
>> type=node&node=4726091&i=3>>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> As it seems  I can't send images to mail list, just
sent to  you
>> both
>>>>>> via
>>>>>>>>>> email. Some graphs we have comparing versions.
>>>>>>>>>> 
>>>>>>>>>> Not sure what changes might cause it.
>>>>>>>>>> 
>>>>>>>>>>> On 12 May 2017, at 17:37, Clebert Suconic <[hidden
email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=4>>
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> There is a class we use on producer.  TokenLimiter.
 Perhaps you
>>>>>> could
>>>>>>>>>>> reuse that one ?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, May 12, 2017 at 11:00 AM nigro_franz
<[hidden email]
>>>>>> <http:///user/SendEmail.jtp?type=node&node=4726091&i=5>>
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> I was thinking of a similar solution but
I've discovered that
>>>>>> couldn't
>>>>>>>>>> work
>>>>>>>>>>>> (in the old or the new TimedBuffer too),
because of the
>>>>>>>>>>>> TimedBuffer::checkSize method that could
force a flush if the
>> batch
>>>>>>>>>> buffer
>>>>>>>>>>>> if not big enough to receive new data, going
IOPS.
>>>>>>>>>>>> Sadly TimedBuffer::checkSize is outside any
timeout, but
>> depends on
>>>>>> the
>>>>>>>>>>>> writers.
>>>>>>>>>>>> 
>>>>>>>>>>>> That's why I've implemented the "compensation"
right after any
>>>>>> flush,
>>>>>>>>>> in
>>>>>>>>>>>> order to work with forced flushes too:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/
>>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/
>>>>>> src/main/java/org/apache/activemq/artemis/core/io/
>>>>>> buffer/TimedBuffer.java#L338
>>>>>>>>>>>> 
>>>>>>>>>>>> Doing it on TimedBuffer::flush all the flushes
on disk will be
>>>>>>>>>> compensated
>>>>>>>>>>>> (half of the story: ASYNCIO is async so depends
on libAIO
>>>>>> partially!)
>>>>>>>>>>>> Regarding the IOPS computation I've built
this, as you've
>>>>>> suggested:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> https://github.com/franz1981/activemq-artemis/blob/
>>>>>> 4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/
>>>>>> src/main/java/org/apache/activemq/artemis/core/io/
>>>>>> buffer/TimedBuffer.java#L119
>>>>>>>>>>>> 
>>>>>>>>>>>> The performance seems pretty good, it compensates
well but it
>> is
>>>>>> faster
>>>>>>>>>>>> than
>>>>>>>>>>>> the original version, limiting IOPS too!
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Franz
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>> 
>>>>>>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
>>>>>> IOPS-Limiter-strategy-tp4725875p4726057.html
>>>>>>>>>>>> Sent from the ActiveMQ - Dev mailing list
archive at
>> Nabble.com.
>>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Clebert Suconic
>>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Clebert Suconic
>>>>>>>>> 
>>>>>>>> --
>>>>>>>> Clebert Suconic
>>>>>> 
>>>>>> 
>>>>>> ------------------------------
>>>>>> If you reply to this email, your message will be added to the
>> discussion
>>>>>> below:
>>>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
>>>>>> IOPS-Limiter-strategy-tp4725875p4726091.html
>>>>>> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click
>> here
>>>>>> <
>> 
>>>>>> .
>>>>>> NAML
>>>>>> <http://activemq.2283324.n4.nabble.com/template/
>> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
>> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
>> BasicNamespace-nabble.view.web.template.NabbleNamespace-
>> nabble.view.web.template.NodeNamespace&breadcrumbs=
>> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
>> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> View this message in context: http://activemq.2283324.n4.
>> nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726093.html
>>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>> 
>>> 
>>> 
>>> --
>>> Clebert Suconic
>> 
>> 
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-
>> IOPS-Limiter-strategy-tp4725875p4726097.html
>> To unsubscribe from [DISCUSS] Artemis IOPS Limiter strategy, click here
>> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4725875&code=bmlncm8uZnJhQGdtYWlsLmNvbXw0NzI1ODc1fC0zNzI3NjE2NQ==>
>> .
>> NAML
>> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>> 
> 
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726099.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Mime
View raw message