bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <guosi...@gmail.com>
Subject Re: Using BK as WAL and accessing ledger metadata
Date Wed, 06 Feb 2013 06:10:33 GMT
Whitney,

Thanks for replies.

I think the case depends on how you reader consume the entries in the
ledgers.

1) if a reader consumes the entries added by a writer in order, so you just
keep a ledger id list, for the last ledger, it is an writing ledger, you
should use #openLedgerNoRecovery to read, for the ledgers created before
last ledger, they are closed ledgers, you could #openLedger to read.

2) if the readers just randomly pick a ledger to read, you might need to
check the state.


so exposing api like LedgerHandle#isClosed() to verify a ledger is closed
or not, is it enough for your case?

-Sijie



On Tue, Feb 5, 2013 at 11:11 AM, Whitney Sorenson <wsorenson@hubspot.com>wrote:

> Sijie,
>
> The problem is I have many writers (all with their own ledgers.) They are
> constantly closing and creating new ledgers.
>
> Then I have many readers which want to read the ledgers. How should the
> readers know what the ledgers are that exist to be read - they want to read
> ALL ledgers that are closed, essentially.
>
> Does this make sense?
>
>
> On Tue, Feb 5, 2013 at 12:09 AM, Sijie Guo <guosijie@gmail.com> wrote:
>
>> Hello Whitney,
>>
>>
>> On Mon, Feb 4, 2013 at 10:44 AM, Whitney Sorenson <wsorenson@hubspot.com>wrote:
>>
>>> Thank you for responding.
>>>
>>> Forgive me if I'm missing something, but if I have a writer and separate
>>> readers, why would I want to have to communicate ledger ids between them?
>>> More specifically, we have a series of writers writing to a write-ahead log
>>> and a separate set of readers that are consuming these ledgers to move them
>>> into long term storage and send them to queues / workflows to be processed.
>>>
>>
>> I am just curious about the case you mentioned that you have a series of
>> writers writing to a write-ahead log. If the write-ahead log means a
>> ledger, I couldn't image how you implemented a series of writers writing to
>> it, since bookkeeper just allow one writer writing to a ledger.
>>
>> if the write-head log is formed by several ledgers, it means that you
>> might already have a mechanism to map the writer to the ledger, so when a
>> writer #openLedger, it means that the ledger would be closed. basically,
>> the close state could be distinguished by different calls : the ledger
>> handle returned by #createLedger is an opened ledger while the ledger
>> handle returned by #openLedger is a closed ledger. If you want to write
>> entries, you had to create a new ledger. Either the ledger is closed or the
>> writer is crashed, the ledger could not be written again. So I don't think
>> you need to keep the state is your end. If I don't understand your case,
>> please let me know.
>>
>>
>>> This means I have to keep the state about which ledgers are available,
>>> and which are closed, which seems to be a complete duplication of the state
>>> that is already in BK.
>>>
>>> I'm not sure named ledgers are helpful in this situation, except that we
>>> could keep less state (perhaps a sequential id.)
>>>
>>> On Mon, Feb 4, 2013 at 1:27 PM, Sijie Guo <guosijie@gmail.com> wrote:
>>>
>>>>
>>>> Hello, Whitney:
>>>>
>>>> please check the replies inline.
>>>>
>>>> On Mon, Feb 4, 2013 at 8:47 AM, Whitney Sorenson <wsorenson@hubspot.com
>>>> > wrote:
>>>>
>>>>> Hey all,
>>>>>
>>>>> A couple questions about running BK stand-alone:
>>>>>
>>>>> 1) If I call openLedgerNoRecovery am I blocking writes or not? What
>>>>> are the guarantees I lose - just ordering? Can I use this to essentially
>>>>> read / tail an active ledger?
>>>>>
>>>>
>>>> open a ledger using openLedgerNoRecovery doesn't block any writes to
>>>> it. And you don't lose the ordering guarantee. You could use it to
>>>> read/tail an active ledger, but please keep in mind that you need to call
>>>> #readLastConfirmed to catch up to the latest confirmed entries added by the
>>>> writer. And the entries you could read from an openLedgerNoRecovery ledger,
>>>> is just between 0 and last confirmed.
>>>>
>>>> you could check:
>>>> http://zookeeper.apache.org/bookkeeper/docs/r4.2.0/apidocs/org/apache/bookkeeper/client/BookKeeper.html#asyncOpenLedgerNoRecovery(long,
>>>> org.apache.bookkeeper.client.BookKeeper.DigestType, byte[],
>>>> org.apache.bookkeeper.client.AsyncCallback.OpenCallback, java.lang.Object)
>>>>
>>>>
>>>>>
>>>>> 2) How can I access BK's metadata so that I can determine a list of
>>>>> ledgers, and which ledgers are closed/open? It doesn't appear in the
client
>>>>> documentation (
>>>>> http://zookeeper.apache.org/bookkeeper/docs/r4.2.0/apidocs/org/apache/bookkeeper/client/)
>>>>> Is this not an intended operation? Are clients supposed to track ledger
ids
>>>>> on their own (we are currently doing this but it seems suboptimal)
>>>>>
>>>>>
>>>> currently we don't expose the API for client. Is there any special case
>>>> you are considering? We'd happy to expose it if necessary.
>>>>
>>>>  Since most of the cases are working in following styles: a *standby*
>>>> writer observes the *active* writer state, if the *active* writer failed,
>>>> the *standby* writer would take over the responsibility, closed the ledger
>>>> written by *active* writer, replayed the ledger and created a new ledger
to
>>>> write new entries. For now, clients needs to track ledger ids on their end.
>>>>
>>>> There is one proposal working on providing *named* ledgers on top of
>>>> bookkeeper to ease user's experience tracking ledger ids. You could check
:
>>>> https://issues.apache.org/jira/browse/BOOKKEEPER-220 . And we are
>>>> under discussion on whether to provide ledger name internally in bookkeeper
>>>> for metadata access concerns. We'd like to hear your feedback on the usage
>>>> of API and make it better.
>>>>
>>>>
>>>>
>>>>> Thank you;
>>>>>
>>>>> -Whitney Sorenson
>>>>> HubSpot
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message