hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@cloudera.com>
Subject Re: HDFS as a logfile ??
Date Fri, 10 Apr 2009 04:14:19 GMT
This is a great idea and a common application, Ricky.  Scribe is probably
useful for you as well:

<http://sourceforge.net/projects/scribeserver/>
<
http://images.google.com/imgres?imgurl=http://farm3.static.flickr.com/2211/2197670659_b42810b8ba.jpg&imgrefurl=http://www.flickr.com/photos/niallkennedy/2197670659/&usg=__WLc-p9Gi_p3AdA-YuKLRZ-bdgvo=&h=375&w=500&sz=131&hl=en&start=2&sig2=P22LVO1KObby6_DDy8ujYg&um=1&tbnid=QudxiEyFOk1EpM:&tbnh=98&tbnw=130&prev=/images%3Fq%3Dfacebook%2Bscribe%2Bhadoop%26hl%3Den%26client%3Dfirefox-a%26rls%3Dorg.mozilla:en-US:official%26sa%3DN%26um%3D1&ei=48beSa74L4H-swORnPmjDg
>

Scribe is what Facebook uses to get its Apache logs to Hadoop.
Unfortunately, HDFS doesn't (yet) have append, so you'll have to batch log
files and load them into HDFS in bulk.

Alex

On Thu, Apr 9, 2009 at 9:06 PM, Ricky Ho <rho@adobe.com> wrote:

> I want to analyze the traffic pattern and statistics of a distributed
> application.  I am thinking of having the application write the events as
> log entries into HDFS and then later I can use a Map/Reduce task to do the
> analysis in parallel.  Is this a good approach ?
>
> In this case, does HDFS support concurrent write (append) to a file ?
>  Another question is whether the write API thread-safe ?
>
> Rgds,
> Ricky
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message