cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ingram Chen <ingramc...@gmail.com>
Subject Re: busy thread on IncomingStreamReader ?
Date Tue, 20 Apr 2010 03:03:57 GMT
I check system.log both, but there is no exception logged.

On Tue, Apr 20, 2010 at 10:40, Jonathan Ellis <jbellis@gmail.com> wrote:

> I don't see csArena-tmp-6-Index.db in the incoming files list.  If
> it's not there, that means that it did break out of that while loop.
>
> Did you check both logs for exceptions?
>
> On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <ingramchen@gmail.com> wrote:
> > Ouch ! I talk too early !
> >
> > We still suffer same problems after upgrade to 1.6.0_20.
> >
> > In JMX StreamingService, I see several wired incoming/outgoing transfer:
> >
> > In Host A, 192.168.2.87
> >
> > StreamingService Status:
> > Done with transfer to /192.168.2.88
> >
> > StreamingService StreamSources:
> > [/192.168.2.88]
> >
> > StreamingService StreamDestinations:
> > [/192.168.2.88]
> >
> > StreamingService getIncomingFiles=192.168.2.88
> > [
> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
> > 0/5718,
> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
> > 0/325,
> > UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
> > 0/29831,
> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
> > 0/47623,
> >
> > ... omit several 0 received pending files.....
> >
> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
> > 0/355041,
> >
> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
> > 27711/2173906,
> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
> > 27711/18821998,
> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
> > 27711/743037,
> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
> > 27711/189214,
> > UserState:
> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
> > 27711/1892375,
> > UserState:
> /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
> > 27711/143216,
> > UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
> > 27711/201188,
> > UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
> > 27711/354923,
> > UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
> > 27711/1260768,
> > UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
> > 27711/332649,
> > UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
> > 27711/39739
> > ]
> >
> > lots of files stalled after receiving 27711 bytes. this strange number is
> > the length of first file to income, see Host B
> >
> > Host B, 192.168.2.88
> >
> > StreamingService Status:
> > Receiving stream
> >
> > StreamingService StreamSources:
> > StreamSources: [/192.168.2.87]
> >
> > StreamingService StreamDestinations:
> >  [/192.168.2.87]
> >
> > StreamingService getOutgoingFiles=192.168.2.87
> > [
> > /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
> > /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
> > /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
> >
> > ... omit pending outgoing files ....
> > ]
> >
> > It seems that outgoing files does not terminate properly. and cause the
> > receiver goes into infinite loop to cause busy thread. From thread dump,
> it
> > looks like fc.transferFrom() in IncomingStreamReader never return:
> >
> >     while (bytesRead < pendingFile.getExpectedBytes()) {
> >                 bytesRead += fc.transferFrom(socketChannel, bytesRead,
> > FileStreamTask.CHUNK_SIZE);
> >                 pendingFile.update(bytesRead);
> >     }
> >
> >
> > On Tue, Apr 20, 2010 at 05:48, Rob Coli <rcoli@digg.com> wrote:
> >>
> >> On 4/17/10 6:47 PM, Ingram Chen wrote:
> >>>
> >>> after upgrading jdk from  1.6.0_16 to  1.6.0_20, the problem solved.
> >>
> >> FYI, this sounds like it might be :
> >>
> >> https://issues.apache.org/jira/browse/CASSANDRA-896
> >>
> >>
> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
> >>
> >> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
> >> storming which hoses performance.
> >>
> >> =Rob
> >
> >
> >
> >
> >
>



-- 
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen

Mime
View raw message