cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: busy thread on IncomingStreamReader ?
Date Tue, 20 Apr 2010 02:40:37 GMT
I don't see csArena-tmp-6-Index.db in the incoming files list.  If
it's not there, that means that it did break out of that while loop.

Did you check both logs for exceptions?

On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <ingramchen@gmail.com> wrote:
> Ouch ! I talk too early !
>
> We still suffer same problems after upgrade to 1.6.0_20.
>
> In JMX StreamingService, I see several wired incoming/outgoing transfer:
>
> In Host A, 192.168.2.87
>
> StreamingService Status:
> Done with transfer to /192.168.2.88
>
> StreamingService StreamSources:
> [/192.168.2.88]
>
> StreamingService StreamDestinations:
> [/192.168.2.88]
>
> StreamingService getIncomingFiles=192.168.2.88
> [
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
> 0/5718,
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
> 0/325,
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
> 0/29831,
> UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
> 0/47623,
>
> ... omit several 0 received pending files.....
>
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
> 0/355041,
>
> UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
> 27711/2173906,
> UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
> 27711/18821998,
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
> 27711/743037,
> UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
> 27711/189214,
> UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.db
> 27711/1892375,
> UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.db
> 27711/143216,
> UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
> 27711/201188,
> UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
> 27711/354923,
> UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
> 27711/1260768,
> UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
> 27711/332649,
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
> 27711/39739
> ]
>
> lots of files stalled after receiving 27711 bytes. this strange number is
> the length of first file to income, see Host B
>
> Host B, 192.168.2.88
>
> StreamingService Status:
> Receiving stream
>
> StreamingService StreamSources:
> StreamSources: [/192.168.2.87]
>
> StreamingService StreamDestinations:
>  [/192.168.2.87]
>
> StreamingService getOutgoingFiles=192.168.2.87
> [
> /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
> /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
> /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
>
> ... omit pending outgoing files ....
> ]
>
> It seems that outgoing files does not terminate properly. and cause the
> receiver goes into infinite loop to cause busy thread. From thread dump, it
> looks like fc.transferFrom() in IncomingStreamReader never return:
>
>     while (bytesRead < pendingFile.getExpectedBytes()) {
>                 bytesRead += fc.transferFrom(socketChannel, bytesRead,
> FileStreamTask.CHUNK_SIZE);
>                 pendingFile.update(bytesRead);
>     }
>
>
> On Tue, Apr 20, 2010 at 05:48, Rob Coli <rcoli@digg.com> wrote:
>>
>> On 4/17/10 6:47 PM, Ingram Chen wrote:
>>>
>>> after upgrading jdk from  1.6.0_16 to  1.6.0_20, the problem solved.
>>
>> FYI, this sounds like it might be :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-896
>>
>> http://bugs.sun.com/view_bug.do;jsessionid=60c39aa55d3666c0c84dd70eb826?bug_id=6805775
>>
>> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
>> storming which hoses performance.
>>
>> =Rob
>
>
>
>
>

Mime
View raw message