bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Re: BookKeeper as low-latency distributed data storage
Date Fri, 12 Aug 2016 12:09:02 GMT
Thanks Sijie. I will make some bench and  report the results

Enrico

Il Mar 9 Ago 2016 21:50 Sijie Guo <sijie@apache.org> ha scritto:

> In general, reading an entry from bookie is pretty fast as most of the
> index pages will be cached in memory, so it typically only needs one disk
> access.
>
> You might see evaluated latency when index pages begin swapping in and
> out. So ideally I would suggest using small index page size and large
> memory, so you can keep most of the index pages in memory.
>
> - Sijie
>
> On Wed, Aug 3, 2016 at 7:10 AM, Enrico Olivelli <eolivelli@gmail.com>
> wrote:
>
>> Hi BookKeepers,
>>
>> Recently JV explained how BookKeeper can support a long term
>> low-latency data store.
>>
>> I'm working on a new project which need a fast replicated "object
>> store" on local LAN, replication is needed only for high-availability.
>>
>> Usually the object (byte[]) size will be between 10KB and 10MB and
>> typically they are kept only for a couple of days, for a medium total
>> storage size of 700GB.
>>
>> My idea is to write these objects to BookKeeper and store the entry
>> coordinates (ledgerId/offset) as a pointer to the object.
>>
>> My question is about random access reading from Bookies, can I hope to
>> achieve low latency while reading a single entry from a ledger ?
>>
>> Both writes and reads will run concurrently with an high degree of
>> parallelism (thousands of concurrent writes/reads)
>>
>> In my use case it is very likely that reads will be done on recently
>> written data, so I think that the bookie cache would come to help
>>
>> What do you think ?
>>
>
> --


-- Enrico Olivelli

Mime
View raw message