atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhan Neethiraj <mad...@apache.org>
Subject Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages
Date Fri, 29 Sep 2017 06:30:31 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/
-----------------------------------------------------------

(Updated Sept. 29, 2017, 6:30 a.m.)


Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath Subramanian,
and Suma Shivaprasad.


Changes
-------

Addressed review comments. Added unit tests.


Bugs: ATLAS-2075
    https://issues.apache.org/jira/browse/ATLAS-2075


Repository: atlas


Description
-------

Update notification module to compress and split large messages (> 1mb) that can't be sent
via Kafka. When such a message is sent, the message is first compressed via GZip. If compressed
message is within the size limit, the message is sent via Kafka. If the compressed size is
larger, then the message is split into multiple small messages and sent via Kafka.

Notification consumer is updated to merge and uncompress messages before the message is processed
further.


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49e 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef8 
  notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java
ec99372d 
  notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc68

  notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java
PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java
PRE-CREATION 
  notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407ac 
  notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 956c85e0

  notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb46

  notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java
cc2099e7 
  notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 9b712f44 
  notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java b7474a0e

  notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
3b2a093b 
  notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9f

  notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java 587b7ebd

  notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java
be324277 
  notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java
3724fd5f 


Diff: https://reviews.apache.org/r/62681/diff/3/

Changes: https://reviews.apache.org/r/62681/diff/2-3/


Testing
-------

- verified that large messages are compressed and split by Atlas hook, before sending to Kafka
- verified that Atlas server processes compressed, multi-part messages correctly


Thanks,

Madhan Neethiraj


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message