atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Mestry (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ATLAS-2064) Compress Hook Messages Posted to Kafka
Date Fri, 18 Aug 2017 22:55:00 GMT

     [ https://issues.apache.org/jira/browse/ATLAS-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ashutosh Mestry updated ATLAS-2064:
-----------------------------------
    Summary: Compress Hook Messages Posted to Kafka  (was: Compressed HiveHook Messages)

> Compress Hook Messages Posted to Kafka
> --------------------------------------
>
>                 Key: ATLAS-2064
>                 URL: https://issues.apache.org/jira/browse/ATLAS-2064
>             Project: Atlas
>          Issue Type: Improvement
>          Components: atlas-intg
>    Affects Versions: 0.8-incubating
>            Reporter: Ashutosh Mestry
>            Assignee: Ashutosh Mestry
>             Fix For: trunk
>
>         Attachments: ATLAS-2064-compressed-hook-message-envelope.patch
>
>
> *Background*
> Messages posted by hooks to Atlas Kafka topics are JSON format. 
> Kafka imposes a 1MB limit on the message size.
> Occasionally, depending on operations performed, this threshold is reached. This results
in messages being dropped.
> The entities are thus not reflected in Atlas.
> *Solution*
> Applying compression to these message will alleviate the problem.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message