hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Babu <manoj...@gmail.com>
Subject Re: Reg: parsing all files & file append
Date Mon, 10 Sep 2012 05:33:09 GMT
Thank you Bejoy.

Does file append is production stable?


Cheers!
Manoj.



On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <bejoy.hadoop@gmail.com> wrote:

> **
> Hi Manoj
>
> You can load daily logs into a individual directories in hdfs and process
> them daily. Keep those results in hdfs or hbase or dbs etc. Every day do
> the processing, get the results and aggregate the same with the previously
> aggregated results till date.
>
> Regards
> Bejoy KS
>
> Sent from handheld, please excuse typos.
> ------------------------------
> *From: * Manoj Babu <manoj444@gmail.com>
> *Date: *Sun, 9 Sep 2012 21:28:54 +0530
> *To: *<mapreduce-user@hadoop.apache.org>
> *ReplyTo: * mapreduce-user@hadoop.apache.org
> *Subject: *Reg: parsing all files & file append
>
> Hi All,
>
> I have two questions, providing info on it will be helpful.
>
> 1, I am using hadoop to analyze and to find top n search term metric's
> from logs.
> If any new log file is added to HDFS then again we are running the job to
> find the metrics.
> Daily we will be getting log files and we are parsing the whole file and
> getting the metric's.
> All the log file's are parsed daily to get the latest metric's is there
> any way is there any way to avoid this?
>
> 2, Does file append is production stable?
>
> Cheers!
> Manoj.
>
>

Mime
View raw message