flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: unable to write to HDFS with flume 1.2.0
Date Thu, 02 Aug 2012 23:21:45 GMT
agent.sinks.hdfsSink.channels = memoryChannel   <-needs to be "agent.sinks.hdfsSink.channel
= memoryChannel" 

Hari
-- 
Hari Shreedharan


On Thursday, August 2, 2012 at 4:19 PM, David Capwell wrote:

> I downloaded the latest release (1.2.0) of flume and tried to get it sending data to
HDFS; for some reason it isn't adding hdfs as a sink.  Here is my properties file:
> 
> 
> agent.sources = seqGenSrc 
> agent.channels = memoryChannel
> agent.sinks = hdfsSink
> 
> # For each one of the sources, the type is defined
> agent.sources.seqGenSrc.type = seq
> agent.sources.seqGenSrc.channels = memoryChannel
> 
> # Each sink's type must be defined
> agent.sinks.loggerSink.type = logger
> agent.sinks.loggerSink.channel = memoryChannel
> 
> # Each channel's type is defined. 
> agent.channels.memoryChannel.type = memory
> agent.channels.memoryChannel.capacity = 100
> 
> agent.sinks.hdfsSink.type = hdfs
> 
> agent.sinks.hdfsSink.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
> agent.sinks.hdfsSink.hdfs.filePrefix = events-#agent.sinks.hdfsSink.hdfs.round = true
> agent.sinks.hdfsSink.hdfs.roundValue = 10#agent.sinks.hdfsSink.hdfs.roundUnit = second
> 
> 
> 
> Here is what I see in the logs with debugging enabled:
> 
> 2012-08-02 16:11:31,961 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)]
649 Creating sink: hdfsSink using HDFS 
> 2012-08-02 16:11:31,963 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:668)]
668 Configuration empty for: hdfsSink.Removed.
> 
> 
> Going over the code I wasn't sure where the ConfigurationException was thrown so I did
a test and commented out agent.sinks.hdfsSink.channels (to get linke 655 to fail [sinkConf.configure(sinkContext);]
) but still get the same error. 
> 
> 
> am I missing something?  
> 
> Thanks for your time reading this email. 


Mime
View raw message