atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Mestry (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (ATLAS-2075) Support Arbitrarily Large Size Messages from Hooks
Date Mon, 21 Aug 2017 18:18:00 GMT

     [ https://issues.apache.org/jira/browse/ATLAS-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ashutosh Mestry reassigned ATLAS-2075:
--------------------------------------

    Assignee: Ashutosh Mestry

> Support Arbitrarily Large Size Messages from Hooks
> --------------------------------------------------
>
>                 Key: ATLAS-2075
>                 URL: https://issues.apache.org/jira/browse/ATLAS-2075
>             Project: Atlas
>          Issue Type: Improvement
>          Components:  atlas-core
>    Affects Versions: 0.8-incubating
>            Reporter: Ashutosh Mestry
>            Assignee: Ashutosh Mestry
>             Fix For: trunk
>
>
> *Background*
> Messages produced by Hooks have potential to be larger than 1MB, which is the size threshold
imposed by Kafka.
> Although, compressing the messages (see [ATLAS-2064](https://issues.apache.org/jira/browse/ATLAS-2064))
alleviates the the problem, it is not a complete solution. It is possible even for compressed
messages to exceed the size threshold.
> *Solution*
> If the compressed message produced exceeds the size threshold, split the messages. Accumulate
the message at consumer end. 
> Account for cases such as:
> - Messages are not received in order they are produced.
> - Atlas server is shutdown before it can consume all the split messages.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message