atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhay Kulkarni <akulka...@hortonworks.com>
Subject Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages
Date Sun, 01 Oct 2017 16:40:29 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62681/#review186808
-----------------------------------------------------------


Ship it!




- Abhay Kulkarni


On Sept. 30, 2017, 9:58 p.m., Madhan Neethiraj wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62681/
> -----------------------------------------------------------
> 
> (Updated Sept. 30, 2017, 9:58 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath Subramanian,
and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-2075
>     https://issues.apache.org/jira/browse/ATLAS-2075
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> Update notification module to compress and split large messages (> 1mb) that can't
be sent via Kafka. When such a message is sent, the message is first compressed via GZip.
If compressed message is within the size limit, the message is sent via Kafka. If the compressed
size is larger, then the message is split into multiple small messages and sent via Kafka.
> 
> Notification consumer is updated to merge and uncompress messages before the message
is processed further.
> 
> 
> Diffs
> -----
> 
>   notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java
1f9404de 
>   notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
b1ac2fa0 
> 
> 
> Diff: https://reviews.apache.org/r/62681/diff/6/
> 
> 
> Testing
> -------
> 
> - verified that large messages are compressed and split by Atlas hook, before sending
to Kafka
> - verified that Atlas server processes compressed, multi-part messages correctly
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message