flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Percy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2653) Allow inUseSuffix to be null/empty
Date Wed, 20 Jul 2016 19:04:20 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386457#comment-15386457

Mike Percy commented on FLUME-2653:

Hmm, one more thing. I wonder what happens if you specify this in the configuration file:

my_agent.sinks.my_hdfs_sink.hdfs.inUseSuffix =

Won't that achieve a similar effect? Maybe this patch is not needed, and we already have this
feature (but people just don't know how to use it)? I haven't investigated this in depth but
that question also needs to be answered.

Regarding the comment by [~yobibytes] about the Cloudera Manager validation, someone from
Cloudera should investigate whether that is a real Flume bug, a Flume validation bug, or a
Cloudera Manager validation bug. If I recall correctly, Cloudera Manager relies on information
from Flume itself for most of its configuration validation. If it's a Flume-related bug, we
can fix it here, otherwise that needs to be handled elsewhere (not as part of the Apache Flume

> Allow inUseSuffix to be null/empty
> ----------------------------------
>                 Key: FLUME-2653
>                 URL: https://issues.apache.org/jira/browse/FLUME-2653
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: v1.5.1
>            Reporter: Andrew Jones
>            Assignee: bimal tandel
>              Labels: docs-missing, hdfssink
>             Fix For: v1.7.0
>         Attachments: FLUME-2653.patch
> At the moment, it doesn't seem possible to set the null/empty. We've tried {{''}} which
just adds the quotes to the end, and setting to nothing, which just uses the default {{.tmp}}.
> We want the _in use_ file to have the same name as the _closed_ file, so we can read
from files that are in use without the file moving from underneath us. In our use case, we
know that an in use file is still readable and parseable, because it is just text with a JSON
document per line.
> It looks like [the HDFS sink code|https://github.com/apache/flume/blob/542b1695033d330eb00ae81713fdc838b88332b6/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L618]
can handle this change already, but at the moment there is no way to set the {{bucketPath}}
and {{targetPath}} to be the same.

This message was sent by Atlassian JIRA

View raw message