flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@gmail.com>
Subject Re: Tried NG 1.1.0 and trunk
Date Fri, 20 Jul 2012 10:38:11 GMT
Hi,

On Jul 20, 2012, at 12:13 PM, Wang, Yongkun | Yongkun | BDD wrote:

inline

> Hi,
> 
> Currently we are using a modified version of flume-0.9.3 in production.
> 
> I tried release 1.1.0. The basic command is still bin/flume-ng node, not bin/flume-ng
agent (by doc).

Did you build Flume from git? In 1.1.x the notation was changed into agent, node is deprecated.


> No document for HDFS sink, I need to read the source code to config it. HDFS sink of
1.1.0 seems be developed with hadoop 0.20.205.0. It can rotate files by count of events, but
the last hadoop file cannot be closed.

http://archive.cloudera.com/cdh4/cdh/4/flume-ng/

> 
> Rotating the hadoop files by time interval does not work. It seems that problem is here:
> A mistake by timing 1000 in BucketWriter:
> 
> lastProcessTime = System.currentTimeMillis() * 1000;
> 
> So the lines in 'if' will never get executed:
> 
> if ((rollInterval > 0)
> 
>        && (rollInterval < (System.currentTimeMillis() - lastProcessTime) /
1000))
> 
>      doRotate = true;
> 
> But the code has been changed aggressively in later versions, so it is no longer a bug
in the trunk.
> 
> I tested NG trunk (1.3 snapshot). The code is aggressively changed compared to 1.1.0.
It seems that it is developed with hadoop 0.23.1 or 1.0.1. It can work with hadoop 0.20.205.0
(time-interval-based rotation works perfectly, last hadoop file closed correctly).
> 

Flume looks into the exported classpath and determine the installed hadoop version to use
it.


> Will keep an eye on NG and wait for later releases.


cheers,
 Alex


--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message