atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nixon Rodrigues <nixon.rodrig...@freestoneinfotech.com>
Subject Re: Review Request 61274: ATLAS-1944 - Fix for ConcurrentModificationException Exception in HookConsumer (KafkaConsumer) while stopping Atlas server
Date Wed, 02 Aug 2017 08:46:39 GMT


> On Aug. 1, 2017, 8:12 p.m., Ashutosh Mestry wrote:
> > Can you please let me know the testing performed to verify this. I am little hesitent
because this removes _ShutdownableThread_.
> > 
> > See attached my patch that merely calls _shutdown_ on consumer thread. I have done
some testing on this and did not see the problems we had seen before.

With calling consumer.stop(); there timeout in thread shutdown with message in log "Timed
out waiting for consumer threads to shut down, exiting uncleanly"

But after Adding extended HookConsumer from ShutdownableThread and calling consumer.shutdown(),
closing of kafkaconsumer was happening fine before timeout of consumer thread.Now added back
the ShutdownableThread implementation.


- Nixon


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61274/#review181916
-----------------------------------------------------------


On Aug. 1, 2017, 11:07 a.m., Nixon Rodrigues wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61274/
> -----------------------------------------------------------
> 
> (Updated Aug. 1, 2017, 11:07 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and Sarath
Subramanian.
> 
> 
> Bugs: ATLAS-1944
>     https://issues.apache.org/jira/browse/ATLAS-1944
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Background:- NotificationHookConsumer was throwing ConcurrentModificationException exception
while shutting down the consumer thread. The stop method of thread was calling the kafkacosumer.stop
method while in run method of HookConsumer which is busy in cosuming the kafka message by
polling kafka server. This simaltanous action of stoping and polling causes ConcurrentModificationException.
> 
> Fix:- 
> 
> * Added try finally block in run method which wraps the while loop and when loop ends
the finally blocks calls the consumer close method is called which ensures that close is called
after consuming is ended. 
> 
> * Replaced the ShutdownThread interface which was delaying the shutdown process of consumer
with Runnable interface used earlier.
> 
> 
> Reference : - https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> 
> 
> Diffs
> -----
> 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d431176 
>   notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java
0bd75e1 
>   notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
bcee00c 
>   webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java b8255b3

> 
> 
> Diff: https://reviews.apache.org/r/61274/diff/1/
> 
> 
> Testing
> -------
> 
> Tested the Atlas kafkaconsumer from hive hook and shutdown of atlas server to check if
there is issue while shutdown process.
> Unit test and integration test are passing.
> 
> 
> Thanks,
> 
> Nixon Rodrigues
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message