felix-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erwin Hogeweg <erwin.hoge...@seecago.com>
Subject Re: Is EventAdmin-1.4.6 really loosing events?
Date Wed, 29 Nov 2017 15:45:56 GMT
Carsten,

Ugh… good point.

Well, case closed I guess? :-)

Thanks for your help Carsten.


Erwin


> On Nov 29, 2017, at 10:02, Carsten Ziegeler <cziegeler@apache.org> wrote:
>
> Hi,
>
> actually I think this is more or less ok. The events are posted from the
> same thread and event admin requires them to be delivered in the order
> they are posted. So if one listener is blocking, then delivery of all
> other events initiated from the same thread are blocked.
>
> Regards
>
> Carsten
>
>
> Carsten Ziegeler wrote
>> Hi Erwin,
>>
>> ah sorry, right :) I was reading modulo 10 instead of equals 10...
>>
>> Yes, you're absolutely right in this case, that should not happen.
>>
>> This looks like a problem in postEvent somewhere
>>
>> Carsten
>>
>>
>> Erwin Hogeweg wrote
>>> Hi Carsten,
>>>
>>> Sorry, I think didn’t make myself clear.
>>>
>>> I am only replacing ONE listener with a BlockingListener, and then I am blocking
ONE event delivery. So I do expect to loose ONE event, not 70k…
>>>
>>> Or am I missing something?
>>>
>>> Erwin
>>>
>>>
>>>    @Test
>>>    public void testEventing() throws Exception {
>>>        // this.addListener(PREFIX + "/0", null);
>>>        this.addBlockingListener(PREFIX + "/0", null);
>>>        this.addListener(PREFIX + "/1", null);
>>>        this.addListener(PREFIX + "/2", null);
>>> ...
>>>
>>>
>>>
>>> On Nov 29, 2017, at 09:21, Carsten Ziegeler <cziegeler@apache.org<mailto:cziegeler@apache.org>>
wrote:
>>>
>>> Thanks
>>>
>>> I'm not sure if event admin could/should do anything in this case. Each
>>> blocking listener takes away one thread from the thread pool. As soon as
>>> you have as many blocking listeners as the pool has threads, event admin
>>> will not deliver any event anymore
>>>
>>> Regards
>>>
>>> Carsten
>>>
>>>
>>> Erwin Hogeweg wrote
>>> Hi Carsten,
>>>
>>> After analyzing a bunch of log files we found some evidence that suggested that
an EventAdminThread might be blocked.
>>>
>>> I was able to duplicate it with a JUnit test. I created a BlockingListener with
extends Listener and override the handleEvent() method. As you can see below we are loosing
almost 70000 events because of this one block.
>>>
>>> I know, eventHandlers are not supposed to block, so we definitely have a design
issue here, but I figured I share the results with you because I am not sure this is what’s
expected.
>>>
>>>
>>> Kind Regards,
>>>
>>> Erwin
>>>
>>>
>>> @Override
>>> public void handleEvent(final Event event) {
>>> this.test.handleEvent(event, this.payload);
>>> count++;
>>> if (count == 10) {
>>> logger.info<http://logger.info>("{} is blocking forever", Thread.currentThread().getName());
>>> while (true) {
>>> try {
>>> Thread.sleep(60000);
>>> } catch (InterruptedException e) {
>>> //
>>> }
>>> }
>>> }
>>>
>>> }
>>>
>>>
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Starting eventing test StressTestIT
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Preparing test with 15 threads
and 10000 events per thread.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Expecting 1050000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started test with 15 threads
and 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 0 events so far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 0 events so far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : [org.apache.felix.eventadmin.ittests.StressTestIT]
: Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : [org.apache.felix.eventadmin.ittests.StressTestIT]
: [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestITStarted thread
>>> Started thread
>>> ] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 12 events so far.
>>> [org.apache.felix.eventadmin.ittests.BlockingListener] : EventAdminThread #6
is blocking forever
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> [org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
>>> …
>>>
>>>
>>>
>>> On Nov 15, 2017, at 21:04, Carsten Ziegeler <cziegeler@apache.org<mailto:cziegeler@apache.org><mailto:cziegeler@apache.org>>
wrote:
>>>
>>> Hi,
>>>
>>> we haven't experienced anything like this so far and I wouldn't know if
>>> any better out of the box way to trouble shoot. I guess the best would
>>> be to add additional logging to event admin, so you can follow what's
>>> happening inside event admin.
>>>
>>> If it is somehow reproducible, then switching from postEvent to
>>> sendEvent could also help in identifying whether there is a problem in
>>> the postEvent handling (which is more complicated than sendEvent).
>>>
>>> Regards
>>>
>>> Carsten
>>>
>>>
>>> Erwin Hogeweg wrote
>>> Hi,
>>>
>>> I have a really bizarre problem. It looks like the eventAdmin is intermittent
but frequently loosing events. I find this very hard to believe but the evidence seems to
support my suspicion.
>>>
>>> I spit out a log msgs just before the event is posted and also when the event
is handled. There is only one eventHandler for this topic, and sure enough every now and again
I see and event be posted but it never comes out at the other end.
>>>
>>> I have the event admin log level set to 4, but there aren’t any msgs to indicate
that something wrong.
>>>
>>> I am really at a loss here. Any suggestions for further trouble shooting would
be greatly appreciated.
>>>
>>> BTW… this seems to happen (and gradually increase) after the system has been
up for several weeks which points in the direction of a resource leak, but I don’t see that
either.
>>>
>>>
>>> Regards,
>>>
>>> Erwin
>>>
>>> Java8
>>> EventAdmin-1.4.6
>>> Equinox: 3.10.2.v20150203-1939
>>>
>>> Producer side:
>>> ...
>>>  if (eventAdmin != null) {
>>>      if (LOG.isDebugEnabled()) {
>>>          LOG.debug("Posting: " + newEvent);
>>>      }
>>>      eventAdmin.postEvent(newEvent);
>>>  } else {
>>>      LOG.error("eventAdmin is null.");
>>>  }
>>>
>>>
>>> Consumer side:
>>>  @Override
>>>  public void handleEvent( Event event ) {
>>>  LOG.info<http://LOG.info><http://LOG.info><http://LOG.info>("handleEvent:
" + event);
>>> ...
>>>
>>> Erwin Hogeweg
>>> CTO
>>> 3690 Airport Road
>>> Boca Raton, FL 33431
>>> P. +1 (954) 556-6565
>>> M. +1 (561) 306-7395
>>> F. +1 (561) 948-2730
>>> [Seecago]<http://www.seecago.com>
>>>
>>> --
>>> Carsten Ziegeler
>>> Adobe Research Switzerland
>>> cziegeler@apache.org<mailto:cziegeler@apache.org><mailto:cziegeler@apache.org>
>>>
>>> Erwin Hogeweg
>>> CTO
>>> 3690 Airport Road
>>> Boca Raton, FL 33431
>>> P. +1 (954) 556-6565
>>> M. +1 (561) 306-7395
>>> F. +1 (561) 948-2730
>>> [Seecago]<http://www.seecago.com>
>>>
>>> --
>>> Carsten Ziegeler
>>> Adobe Research Switzerland
>>> cziegeler@apache.org<mailto:cziegeler@apache.org>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
>>> For additional commands, e-mail: users-help@felix.apache.org
>>>
>>>
>>> Erwin Hogeweg
>>> CTO
>>> 3690 Airport Road
>>> Boca Raton, FL 33431
>>> P. +1 (954) 556-6565
>>> M. +1 (561) 306-7395
>>> F. +1 (561) 948-2730
>>> [Seecago]<http://www.seecago.com>
>>>
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziegeler@apache.org

Erwin Hogeweg
CTO
3690 Airport Road
Boca Raton, FL 33431
P. +1 (954) 556-6565
M. +1 (561) 306-7395
F. +1 (561) 948-2730
[Seecago]<http://www.seecago.com>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@felix.apache.org
For additional commands, e-mail: users-help@felix.apache.org


Mime
View raw message