atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shwetha GS <sshivalingamur...@hortonworks.com>
Subject Re: Review Request 51035: ATLAS-1111 Data loss is observed when atlas is restarted while hive_table metadata ingestion into kafka topic is in-progress
Date Fri, 12 Aug 2016 12:00:15 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51035/
-----------------------------------------------------------

(Updated Aug. 12, 2016, noon)


Review request for atlas.


Changes
-------

Added prefix for failed messages

have renamed KafkaNotificationTest to KafkaNotificationMockTest and added KafkaNotificationTest.
But the diff looks big for KafkaNotificationTest


Bugs: ATLAS-1111
    https://issues.apache.org/jira/browse/ATLAS-1111


Repository: atlas


Description
-------

kafkaconsumer.next() always returns the next message irrespective of auto commit or not. With
auto commit disabled, the the offset has to be committed manually, as opposed to auto commit
on next(). The last saved offset is used across consumer restarts.
During graceful shutdown, because of graph shutdown, all the message processing failed and
we did commits even in case of failures. So, after restart, atlas started at last committed
offset and ignored message failures during shutdown and hence lost messages.
The patch has the following changes:
1. No commit in case of message failure
2. Failed messages are written to different log file


Diffs (updated)
-----

  distro/src/conf/atlas-log4j.xml eaa4ec5 
  notification/src/main/java/org/apache/atlas/kafka/KafkaConsumer.java 270215d 
  notification/src/main/java/org/apache/atlas/notification/NotificationConsumer.java 2e861cb

  notification/src/main/java/org/apache/atlas/notification/hook/HookNotification.java 88a0322

  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java PRE-CREATION

  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationTest.java 2a49634 
  notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
0c8990f 
  plugin-classloader/src/main/java/org/apache/atlas/plugin/classloader/AtlasPluginClassLoader.java
92cc406 
  typesystem/src/test/resources/atlas-application.properties a3b6c90 
  webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 901b1ed

  webapp/src/main/java/org/apache/atlas/web/listeners/GuiceServletConfig.java 0a7c5df 
  webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
6fd1939 
  webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java d22c5f1


Diff: https://reviews.apache.org/r/51035/diff/


Testing
-------

UTs and end to end test


Thanks,

Shwetha GS


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message