Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 997C1200D1D for ; Sat, 30 Sep 2017 00:32:31 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9807D1609ED; Fri, 29 Sep 2017 22:32:31 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B68A41609D1 for ; Sat, 30 Sep 2017 00:32:30 +0200 (CEST) Received: (qmail 16908 invoked by uid 500); 29 Sep 2017 22:32:29 -0000 Mailing-List: contact dev-help@atlas.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@atlas.apache.org Delivered-To: mailing list dev@atlas.apache.org Received: (qmail 16892 invoked by uid 99); 29 Sep 2017 22:32:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Sep 2017 22:32:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 83D171A4393; Fri, 29 Sep 2017 22:32:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.999 X-Spam-Level: ** X-Spam-Status: No, score=2.999 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id uqD3-Fq-kFKz; Fri, 29 Sep 2017 22:32:26 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 5E4085FD33; Fri, 29 Sep 2017 22:32:26 +0000 (UTC) Received: from reviews.apache.org (unknown [10.41.0.12]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id DF6A2E0041; Fri, 29 Sep 2017 22:32:25 +0000 (UTC) Received: from reviews-vm2.apache.org (localhost [IPv6:::1]) by reviews.apache.org (ASF Mail Server at reviews-vm2.apache.org) with ESMTP id 2E251C402D9; Fri, 29 Sep 2017 22:32:24 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============1305216342144748092==" MIME-Version: 1.0 Subject: Re: Review Request 62681: ATLAS-2075: notification enhancement to handle large messages, using compression and multi-part messages From: Madhan Neethiraj To: Ashutosh Mestry , Apoorv Naik , Abhay Kulkarni , Sarath Subramanian Cc: Madhan Neethiraj , atlas Date: Fri, 29 Sep 2017 22:32:24 -0000 Message-ID: <20170929223224.860.86408@reviews-vm2.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Madhan Neethiraj X-ReviewGroup: atlas X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/62681/ X-Sender: Madhan Neethiraj References: <20170929063031.860.37701@reviews-vm2.apache.org> In-Reply-To: <20170929063031.860.37701@reviews-vm2.apache.org> X-ReviewBoard-Diff-For: notification/src/test/java/org/apache/atlas/notification/AtlasNotificationMessageTest.java X-ReviewBoard-Diff-For: notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessage.java X-ReviewBoard-Diff-For: notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java X-ReviewBoard-Diff-For: notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java X-ReviewBoard-Diff-For: notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java X-ReviewBoard-Diff-For: notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java X-ReviewBoard-Diff-For: notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java X-ReviewBoard-Diff-For: notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java Reply-To: Madhan Neethiraj X-ReviewRequest-Repository: atlas archived-at: Fri, 29 Sep 2017 22:32:31 -0000 --===============1305216342144748092== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62681/ ----------------------------------------------------------- (Updated Sept. 29, 2017, 10:32 p.m.) Review request for atlas, Apoorv Naik, Ashutosh Mestry, Abhay Kulkarni, Sarath Subramanian, and Suma Shivaprasad. Changes ------- Updated for the review comments. Udpated to use utf-8 bytes length of the string (instead of string length) in tests for message-max-limit Bugs: ATLAS-2075 https://issues.apache.org/jira/browse/ATLAS-2075 Repository: atlas Description ------- Update notification module to compress and split large messages (> 1mb) that can't be sent via Kafka. When such a message is sent, the message is first compressed via GZip. If compressed message is within the size limit, the message is sent via Kafka. If the compressed size is larger, then the message is split into multiple small messages and sent via Kafka. Notification consumer is updated to merge and uncompress messages before the message is processed further. Diffs (updated) ----- common/src/main/java/org/apache/atlas/AtlasConfiguration.java 9a9bb76a notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49e notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef8 notification/src/main/java/org/apache/atlas/notification/AbstractMessageDeserializer.java ec99372d notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc68 notification/src/main/java/org/apache/atlas/notification/AtlasNotificationBaseMessage.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/AtlasNotificationStringMessage.java PRE-CREATION notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407ac notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java 956c85e0 notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb46 notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java cc2099e7 notification/src/test/java/org/apache/atlas/kafka/KafkaConsumerTest.java 9b712f44 notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java b7474a0e notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 3b2a093b notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9f notification/src/test/java/org/apache/atlas/notification/VersionedMessageTest.java 587b7ebd notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java be324277 notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java 3724fd5f Diff: https://reviews.apache.org/r/62681/diff/4/ Changes: https://reviews.apache.org/r/62681/diff/3-4/ Testing ------- - verified that large messages are compressed and split by Atlas hook, before sending to Kafka - verified that Atlas server processes compressed, multi-part messages correctly Thanks, Madhan Neethiraj --===============1305216342144748092==--