incubator-flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-734) escapedFormatDfs goes into a file creation frenzy
Date Sat, 01 Oct 2011 00:03:45 GMT

    [ https://issues.apache.org/jira/browse/FLUME-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118581#comment-13118581
] 

Jonathan Hsieh commented on FLUME-734:
--------------------------------------

committed.
                
> escapedFormatDfs goes into a file creation frenzy
> -------------------------------------------------
>
>                 Key: FLUME-734
>                 URL: https://issues.apache.org/jira/browse/FLUME-734
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v0.9.4
>         Environment: CentOS 5.6
>            Reporter: Eran Kutner
>            Assignee: Jonathan Hsieh
>            Priority: Critical
>             Fix For: v0.9.5
>
>         Attachments: 0001-FLUME-734-escapedFormatDfs-goes-into-a-file-creation.patch,
FLUME-734-draft.patch, flume.log
>
>
> Using this configuration:
> collectorSource(54001) | collector(600000) { escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
"events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) }
> The expected behavior is to see a new file created every 10 minutes. However, once in
a while the collector would go into a file creation frenzy, creating new files every second.
> The log indicates that writing has failed with error: "OutputFormat instance can only
write to the same OutputStream" causing the file to be closed a new one to be opened just
to be closed again.
> Looking at the code I'm not even sure how the output stream could change but the behavior
I'm seeing feels like some sort of a race condition. It is happening much more under heavy
load than under low load.
> See attached log excerpt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message