flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Capwell <dcapw...@gmail.com>
Subject Re: unable to write to HDFS with flume 1.2.0
Date Fri, 03 Aug 2012 02:29:38 GMT
Thanks!  I have everything working just fine now with the following
settings (writing to s3)

agent.sources = seqGenSrc
agent.channels = memoryChannel
agent.sinks = hdfsSink

# For each one of the sources, the type is defined
agent.sources.seqGenSrc.type = seq
agent.sources.seqGenSrc.channels = memoryChannel
agent.sources.seqGenSrc.interceptors = a b
agent.sources.seqGenSrc.interceptors.a.type =
org.apache.flume.interceptor.HostInterceptor$Builder
agent.sources.seqGenSrc.interceptors.a.preserveExisting = false
agent.sources.seqGenSrc.interceptors.a.hostHeader = hostname
agent.sources.seqGenSrc.interceptors.b.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder

# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 1000
agent.channels.memoryChannel.transactionCapacity = 1000

agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H
agent.sinks.hdfsSink.hdfs.filePrefix = events-
agent.sinks.hdfsSink.hdfs.batchSize = 100000
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.rollSize = 0
agent.sinks.hdfsSink.hdfs.round = true
agent.sinks.hdfsSink.hdfs.roundValue = 10
agent.sinks.hdfsSink.hdfs.roundUnit = minute


On Thu, Aug 2, 2012 at 4:21 PM, Hari Shreedharan
<hshreedharan@cloudera.com>wrote:

>  agent.sinks.hdfsSink.channels = memoryChannel   <-needs to be
> "agent.sinks.hdfsSink.channel = memoryChannel"
>
> Hari
> --
> Hari Shreedharan
>
> On Thursday, August 2, 2012 at 4:19 PM, David Capwell wrote:
>
> I downloaded the latest release (1.2.0) of flume and tried to get it
> sending data to HDFS; for some reason it isn't adding hdfs as a sink.  Here
> is my properties file:
>
>
> agent.sources = seqGenSrc
> agent.channels = memoryChannel
> agent.sinks = hdfsSink
>
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
> agent.sources.seqGenSrc.channels = memoryChannel
>
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
> agent.sinks.loggerSink.channel = memoryChannel
>
> # Each channel's type is defined.
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 100
>
> agent.sinks.hdfsSink.type = hdfs
>
> agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
> agent.sinks.hdfsSink.hdfs.filePrefix =
> events-#agent.sinks.hdfsSink.hdfs.round = true
> agent.sinks.hdfsSink.hdfs.roundValue =
> 10#agent.sinks.hdfsSink.hdfs.roundUnit = second
>
>
> Here is what I see in the logs with debugging enabled:
>
> 2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]
> 649 Creating sink: hdfsSink using HDFS
> 2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN -
> org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)]
> 668 Configuration empty for: hdfsSink.Removed.
>
> Going over the code I wasn't sure where the ConfigurationException was
> thrown so I did a test and commented out agent.sinks.hdfsSink.channels (to
> get linke 655 to fail [sinkConf.configure(sinkContext);] ) but still get
> the same error.
>
>
> am I missing something?
>
> Thanks for your time reading this email.
>
>
>

Mime
View raw message