flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2942) AvroEventDeserializer ignores header from spool source
Date Wed, 21 Dec 2016 08:14:58 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15766433#comment-15766433

ASF GitHub Bot commented on FLUME-2942:

GitHub user sebastian-alfers opened a pull request:


    Read header and footer if available

    This commit fixes an issue, that header-values can not be restored correctly after an
event was avro-serialized to disk.
    The problem was, that deserializing the event from disk, the body of the event contained
the binary and the header.
    See: https://issues.apache.org/jira/browse/FLUME-2942
    Discussion welcome!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sebastian-alfers/flume FLUME-2942

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #99
commit 84a1783217428f683a21c6199d83035792f7d718
Author: sa <sebastian.alfesr@plista.com>
Date:   2016-12-21T08:09:26Z

    Read header and footer if available


> AvroEventDeserializer ignores header from spool source
> ------------------------------------------------------
>                 Key: FLUME-2942
>                 URL: https://issues.apache.org/jira/browse/FLUME-2942
>             Project: Flume
>          Issue Type: Bug
>    Affects Versions: v1.6.0
>            Reporter: Sebastian Alfers
>         Attachments: FLUME-2942-0.patch
> I have a spool file source and use avro for de-/serialization
> In detail, serialized events store the topic of the kafka sink in the header.
> When I load the events from the spool directory, the header are ignored. 
> Please see: https://github.com/apache/flume/blob/caa64a1a6d4bc97be5993cb468516e9ffe862794/flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java#L122
> You can see, it uses the whole event as body but does not distinguish between the header
and body encoded by avro.
> Please verify that this is a bug.
> I fixed this but by using the record that stores header and body separately.

This message was sent by Atlassian JIRA

View raw message