bookkeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Nagro <jna...@hubspot.com>
Subject Re: exception in 4.1 server logs
Date Wed, 06 Jun 2012 12:51:32 GMT
Thanks for looking into it.

-John

On Wed, Jun 6, 2012 at 1:13 AM, Sijie Guo <guosijie@gmail.com> wrote:

> This issue is introduced by LedgerCacheImpl#deleteLedger. It doesn't
> return the ledger pages that deleted ledgers occupied back to LedgerCache.
> So the ledger cache pages are exhausted.
>
> I attached a patch to fix it on
> https://issues.apache.org/jira/browse/BOOKKEEPER-287 .
>
> Thanks John for reporting this issue.
>
> -
> Sijie
>
>
> On Wed, Jun 6, 2012 at 6:57 AM, John Nagro <jnagro@hubspot.com> wrote:
>
>> I'm sorry, but I don't think so.... basically, the server was running
>> fine (4.1), there were fencing messages but that seems to be normal, and
>> then the exceptions started:
>>
>> 2012-06-05 15:12:38,107 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759036 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:12:43,818 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759043 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:12:44,672 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759057 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:12:54,018 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759066 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:12:54,124 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759068 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:13:02,857 - WARN
>>  [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server
>> socket loop: /0.0.0.0
>> java.util.NoSuchElementException
>>         at java.util.LinkedList.getFirst(LinkedList.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199)
>>         at
>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81)
>>         at
>> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656)
>>         at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691)
>>         at
>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123)
>> 2012-06-05 15:13:04,033 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759078 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:13:04,034 - WARN
>>  [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server
>> socket loop: /0.0.0.0
>> java.util.NoSuchElementException
>>         at java.util.LinkedList.getFirst(LinkedList.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169)
>>          at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.getEntryOffset(LedgerCacheImpl.java:217)
>>         at
>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:126)
>>         at
>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:86)
>>         at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:715)
>>         at
>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:400)
>>          at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123)
>> 2012-06-05 15:13:08,868 - WARN
>>  [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server
>> socket loop: /0.0.0.0
>> java.util.NoSuchElementException
>>         at java.util.LinkedList.getFirst(LinkedList.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199)
>>         at
>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81)
>>         at
>> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656)
>>         at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691)
>>         at
>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123)
>> 2012-06-05 15:13:10,950 - WARN
>>  [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server
>> socket loop: /0.0.0.0
>>  java.util.NoSuchElementException
>>         at java.util.LinkedList.getFirst(LinkedList.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199)
>>         at
>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81)
>>         at
>> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656)
>>         at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691)
>>         at
>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123)
>> 2012-06-05 15:13:14,649 - WARN  [NIOServerFactory-3181:BookieServer@389]
>> - Ledger 759087 fenced by
>> Socket[addr=/***REDACTED***,port=54294,localport=3181]
>> 2012-06-05 15:13:14,649 - WARN
>>  [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server
>> socket loop: /0.0.0.0
>>  java.util.NoSuchElementException
>>         at java.util.LinkedList.getFirst(LinkedList.java:109)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169)
>>         at
>> org.apache.bookkeeper.bookie.LedgerCacheImpl.getEntryOffset(LedgerCacheImpl.java:217)
>>          at
>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.getEntry(InterleavedLedgerStorage.java:126)
>>         at
>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.readEntry(LedgerDescriptorImpl.java:86)
>>         at org.apache.bookkeeper.bookie.Bookie.readEntry(Bookie.java:715)
>>         at
>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:400)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208)
>>         at
>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123)
>>
>>  On Tue, Jun 5, 2012 at 6:27 PM, Flavio Junqueira <fpj@yahoo-inc.com>wrote:
>>
>>> Thanks for reporting, John. It is a write: at
>>> org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691)
>>>
>>> Is this all you have or you have more logs/information, John?
>>>
>>> -Flavio
>>>
>>>
>>> On Jun 5, 2012, at 11:05 PM, John Nagro wrote:
>>>
>>> 2012-06-05 16:24:29,596 - WARN
>>>  [NIOServerFactory-3181:NIOServerFactory@128] - Exception in server
>>> socket loop: /0.0.0.0
>>> java.util.NoSuchElementException
>>>         at java.util.LinkedList.getFirst(LinkedList.java:109)
>>>         at
>>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabCleanPage(LedgerCacheImpl.java:478)
>>>         at
>>> org.apache.bookkeeper.bookie.LedgerCacheImpl.grabLedgerEntryPage(LedgerCacheImpl.java:169)
>>>         at
>>> org.apache.bookkeeper.bookie.LedgerCacheImpl.putEntryOffset(LedgerCacheImpl.java:199)
>>>         at
>>> org.apache.bookkeeper.bookie.InterleavedLedgerStorage.addEntry(InterleavedLedgerStorage.java:109)
>>>         at
>>> org.apache.bookkeeper.bookie.LedgerDescriptorImpl.addEntry(LedgerDescriptorImpl.java:81)
>>>         at
>>> org.apache.bookkeeper.bookie.Bookie.addEntryInternal(Bookie.java:656)
>>>         at org.apache.bookkeeper.bookie.Bookie.addEntry(Bookie.java:691)
>>>         at
>>> org.apache.bookkeeper.proto.BookieServer.processPacket(BookieServer.java:368)
>>>         at
>>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.readRequest(NIOServerFactory.java:310)
>>>         at
>>> org.apache.bookkeeper.proto.NIOServerFactory$Cnxn.doIO(NIOServerFactory.java:208)
>>>         at
>>> org.apache.bookkeeper.proto.NIOServerFactory.run(NIOServerFactory.java:123)
>>>
>>> unfortunately, i am unsure about what causes this (a read or a write).
>>>
>>>
>>>
>>>
>>
>

Mime
View raw message