atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shwetha GS <>
Subject Re: Review Request 51035: ATLAS-1111 Data loss is observed when atlas is restarted while hive_table metadata ingestion into kafka topic is in-progress
Date Fri, 12 Aug 2016 12:00:15 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 12, 2016, noon)

Review request for atlas.


Added prefix for failed messages

have renamed KafkaNotificationTest to KafkaNotificationMockTest and added KafkaNotificationTest.
But the diff looks big for KafkaNotificationTest

Bugs: ATLAS-1111

Repository: atlas

------- always returns the next message irrespective of auto commit or not. With
auto commit disabled, the the offset has to be committed manually, as opposed to auto commit
on next(). The last saved offset is used across consumer restarts.
During graceful shutdown, because of graph shutdown, all the message processing failed and
we did commits even in case of failures. So, after restart, atlas started at last committed
offset and ignored message failures during shutdown and hence lost messages.
The patch has the following changes:
1. No commit in case of message failure
2. Failed messages are written to different log file

Diffs (updated)

  distro/src/conf/atlas-log4j.xml eaa4ec5 
  notification/src/main/java/org/apache/atlas/kafka/ 270215d 
  notification/src/main/java/org/apache/atlas/notification/ 2e861cb

  notification/src/main/java/org/apache/atlas/notification/hook/ 88a0322

  notification/src/test/java/org/apache/atlas/kafka/ PRE-CREATION

  notification/src/test/java/org/apache/atlas/kafka/ 2a49634 
  typesystem/src/test/resources/ a3b6c90 
  webapp/src/main/java/org/apache/atlas/notification/ 901b1ed

  webapp/src/main/java/org/apache/atlas/web/listeners/ 0a7c5df 
  webapp/src/test/java/org/apache/atlas/notification/ d22c5f1



UTs and end to end test


Shwetha GS

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message