flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shengyi.pan" <shengyi....@gmail.com>
Subject Re: HDFS sink to a remote HDFS node
Date Tue, 30 Sep 2014 01:04:18 GMT
you need hadoop-common-x.x.x.jar and hadoop-hdfs-x.x.x.jar under your flume-ng classpath, and
the dependent hadoop jar version must match your hadoop system.

if sink to hadoop-2.0.0,  you should use "protobuf-java-2.4.1.jar" (defaultly, flume-1.5.0
uses "protobuf-java-2.5.0.jar", the jar file is under flume lib directory ), because the pb
interface of hdfs-2.0 is compiled wtih protobuf-2.4, while using protobuf-2.5 the flume-ng
will fail to start....



发件人:Ed Judge <ejudgie@gmail.com>
发送时间:2014-09-29 22:38
主题:HDFS sink to a remote HDFS node

I am trying to run the flume-ng agent on one node with an HDFS sink pointing to an HDFS filesystem
on another node.
Is this possible?  What packages/jar files are needed on the flume agent node for this to
work?  Secondary goal is to install only what is needed on the flume-ng node.

# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://<remote IP address>/tmp/

View raw message