hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bejoy Ks <bejoy.had...@gmail.com>
Subject Re: Reg: parsing all files & file append
Date Mon, 10 Sep 2012 08:06:45 GMT
Hi Manoj

>From my limited knowledge on file appends in hdfs , i have seen more
recommendations to use sync() in the latest releases than using append().
Let us wait for some commiter to authoritatively comment on 'the production
readiness of append()' . :)

Regards
Bejoy KS

On Mon, Sep 10, 2012 at 11:03 AM, Manoj Babu <manoj444@gmail.com> wrote:

> Thank you Bejoy.
>
> Does file append is production stable?
>
>
> Cheers!
> Manoj.
>
>
>
> On Sun, Sep 9, 2012 at 10:19 PM, Bejoy KS <bejoy.hadoop@gmail.com> wrote:
>
>> **
>> Hi Manoj
>>
>> You can load daily logs into a individual directories in hdfs and process
>> them daily. Keep those results in hdfs or hbase or dbs etc. Every day do
>> the processing, get the results and aggregate the same with the previously
>> aggregated results till date.
>>
>> Regards
>> Bejoy KS
>>
>> Sent from handheld, please excuse typos.
>> ------------------------------
>> *From: * Manoj Babu <manoj444@gmail.com>
>> *Date: *Sun, 9 Sep 2012 21:28:54 +0530
>> *To: *<mapreduce-user@hadoop.apache.org>
>> *ReplyTo: * mapreduce-user@hadoop.apache.org
>> *Subject: *Reg: parsing all files & file append
>>
>> Hi All,
>>
>> I have two questions, providing info on it will be helpful.
>>
>> 1, I am using hadoop to analyze and to find top n search term metric's
>> from logs.
>> If any new log file is added to HDFS then again we are running the job to
>> find the metrics.
>> Daily we will be getting log files and we are parsing the whole file and
>> getting the metric's.
>> All the log file's are parsed daily to get the latest metric's is there
>> any way is there any way to avoid this?
>>
>> 2, Does file append is production stable?
>>
>> Cheers!
>> Manoj.
>>
>>
>

Mime
View raw message