hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Stack <st...@duboce.net>
Subject Re: HDFS, appending-writes status
Date Fri, 01 Jun 2007 16:26:07 GMT
HBase is another application that needs write-append.

Every HBase update is written both to a RAM-based and file-system-based 
log.  On a period the RAM-based log is flushed to the filesystem.  The 
RAM-based log and its flushes are used fielding queries.

The sympathetic file-system-based log is meant to act as insurance 
against machine crashes.  On recovery, the file-system-based log is 
supposed to be replayed to catch-up on all updates that happened after 
the last RAM-based log flush and before machine crash.

Currently the file-system-based log is being written to HDFS though its 
near-to-useless since the log is only seen after its closed.

Thanks,
St.Ack

P.S. Bdbje writing its logs to HDFS would be pretty cool.


Ali Salehi wrote:
> Hi,
>  Thanks for your response.
>  Actually I want to use it with Berkeley Db java ed. [bdbj].
>  BDBJ only uses appends to create its db so I though using hadoop
>  would be a viable option considering the scalability requirements
>  I'm facing.
>
> Cheers,
> AliS
>
>   
>> Hi Ali,
>>
>> >From what I know, there are quite a few applications that need
>> append-to-exisiting-file feature. Since there is a demand for it, HDFS
>> will
>> probably implement it in the near future. I do not know of precise
>> time-frames yet.
>>
>> Do you have an application that requires appending-writes to files. If so,
>> it will be helpful to understand the type and behaviour of this
>> application.
>>
>> Thanks,
>> dhruba
>>
>> -----Original Message-----
>> From: Ali Salehi [mailto:ali.salehi@epfl.ch]
>> Sent: Thursday, May 31, 2007 11:46 AM
>> To: hadoop-user@lucene.apache.org
>> Subject: HDFS, appending-writes status
>>
>> Hi,
>>
>>  While I was reading the HDFS docs from the wiki, I bump into this
>>  sentence:
>>
>>  "There is a plan to support appending-writes to a file in future"
>>
>>  I would like to know, if possible, what is the time line for implementing
>>  this future.
>>
>>
>> Cheers,
>> AliS
>>
>>
>> **************************************************************
>> Ali Salehi, LSIR - Distributed Information Systems Laboratory
>> EPFL-IC-IIF-LSIR, B√Ętiment BC, Station 14, CH-1015 Lausanne, Switzerland.
>> http://lsirwww.epfl.ch/
>> email: ali.salehi@epfl.ch
>> Tel: +41-21-6936656 Fax: +41-21-6938115
>>
>>
>>     
>
>
> **************************************************************
> Ali Salehi, LSIR - Distributed Information Systems Laboratory
> EPFL-IC-IIF-LSIR, B√Ętiment BC, Station 14, CH-1015 Lausanne, Switzerland.
> http://lsirwww.epfl.ch/
> email: ali.salehi@epfl.ch
> Tel: +41-21-6936656 Fax: +41-21-6938115
>
>   


Mime
View raw message