flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanzhi.liu" <404307...@qq.com>
Subject Re:Unable to use HDFS sink
Date Sat, 29 Sep 2012 08:13:12 GMT
Hello:
    Upgrade hadoop enough.
  

  
  

 

 ------------------ Original ------------------
  From:  "Lichen"<kevinli.li@huawei.com>;
 Date:  Sat, Sep 29, 2012 03:57 PM
 To:  "user@flume.apache.org"<user@flume.apache.org>; 
 
 Subject:  Unable to use HDFS sink

 

  
Hi. I’m using flume 1.2 with hadoop-0.20.2 but unable to use HDFS sink. Flume stuck during
creating HDFS sink, which is OK using file-roll or logger sinks. I’ve included $HADOOP_HOME
and $HADOOP_HOME/lib in FLUME_CLASSPATH. Is it because my hadoop version not compatible?
 
 
 
Configuration file:
 
agent_foo.sources = netcat-source
 
agent_foo.sinks = hdfs-sink
 
agent_foo.channels = mem-channel-1
 
 
 
# properties of netcat-source
 
agent_foo.sources.netcat-source.channels = mem-channel-1
 
agent_foo.sources.netcat-source.type = netcat
 
agent_foo.sources.netcat-source.bind = localhost
 
agent_foo.sources.netcat-source.port = 44444
 
 
 
# properties of hdfs-sink
 
agent_foo.sinks.hdfs-sink.channel = mem-channel-1
 
agent_foo.sinks.hdfs-sink.type = hdfs
 
agent_foo.sinks.hdfs-sink.hdfs.path = hdfs://hadoop0:9000/flume/
 
 
 
# properties of mem-channel-1
 
agent_foo.channels.mem-channel-1.type = memory
 
agent_foo.channels.mem-channel-1.capacity = 1000
 
agent_foo.channels.mem-channel-1.transactionCapacity = 100
 
 
 
Log:
 
$ bin/flume-ng agent -c conf/ -f conf/test.conf -n agent_foo
 
2012-09-29 15:19:47,765 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 1
 
2012-09-29 15:19:47,768 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)]
Flume node starting - agent_foo
 
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)]
Node manager starting
 
2012-09-29 15:19:47,772 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)]
Starting lifecycle supervisor 11
 
2012-09-29 15:19:47,774 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)]
Configuration provider starting
 
2012-09-29 15:19:47,776 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)]
Reloading configuration file:conf/test.conf
 
2012-09-29 15:19:47,780 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hdfs-sink
 
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hdfs-sink
 
2012-09-29 15:19:47,788 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)]
Added sinks: hdfs-sink Agent: agent_foo
 
2012-09-29 15:19:47,789 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)]
Processing:hdfs-sink
 
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)]
Post-validation flume configuration contains configuration  for agents: [agent_foo]
 
2012-09-29 15:19:47,816 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)]
Creating channels
 
2012-09-29 15:19:47,858 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)]
Monitoried counter group for type: CHANNEL, name: mem-channel-1, registered successfully.
 
2012-09-29 15:19:47,859 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)]
created channel mem-channel-1
 
2012-09-29 15:19:47,879 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)]
Creating instance of sink: hdfs-sink, type: hdfs
 
 
 
There is no more output after the last line.
Mime
View raw message