Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 74.125.82.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=oC6NRw3hiLQUUE/ygVRPPd8rFjb5zvGO+p3cv1SAgpZOyiycIp6sO/uuO49fwXRlTN
         H/wct0z8zl4DIZ9MuuoDBPLKyIIQuiVxm/QSHtkc+gxjvLTyGPBNzGINeAunBfmi+6K4
         FIu295cMDoyg5Dl5pJQ0+CYqVt5HG8MABrQFQ=
MIME-Version: 1.0
In-Reply-To: <y2r18306c3b1004191936p13f5636l6aaaab91e7f8984@mail.gmail.com>
References: <h2p18306c3b1004150933vbdf95ba3o6989c3dfad4a3d9a@mail.gmail.com>
	<u2q18306c3b1004171847r1c528660sb66e640bca1be03d@mail.gmail.com>
	<4BCCCF9E.80002@digg.com>
 <y2r18306c3b1004191936p13f5636l6aaaab91e7f8984@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Mon, 19 Apr 2010 21:40:37 -0500
Message-ID: <y2ze06563881004191940t9d3f81b1s900d66778df9bbcc@mail.gmail.com>
Subject: Re: busy thread on IncomingStreamReader ?
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I don't see csArena-tmp-6-Index.db in the incoming files list.  If
it's not there, that means that it did break out of that while loop.

Did you check both logs for exceptions?

On Mon, Apr 19, 2010 at 9:36 PM, Ingram Chen <ingramchen@gmail.com> wrote:
> Ouch ! I talk too early !
>
> We still suffer same problems after upgrade to 1.6.0_20.
>
> In JMX StreamingService, I see several wired incoming/outgoing transfer:
>
> In Host A, 192.168.2.87
>
> StreamingService Status:
> Done with transfer to /192.168.2.88
>
> StreamingService StreamSources:
> [/192.168.2.88]
>
> StreamingService StreamDestinations:
> [/192.168.2.88]
>
> StreamingService getIncomingFiles=3D192.168.2.88
> [
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Index.db
> 0/5718,
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Filter.db
> 0/325,
> UserState: /var/lib/cassandra/data/UserState/multiMine-tmp-11-Data.db
> 0/29831,
> UserState: /var/lib/cassandra/data/UserState/csArena-tmp-13-Index.db
> 0/47623,
>
> ... omit several 0 received pending files.....
>
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-19-Data.db
> 0/355041,
>
> UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Data.db
> 27711/2173906,
> UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Data.db
> 27711/18821998,
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Data.db
> 27711/743037,
> UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Index.db
> 27711/189214,
> UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Data.d=
b
> 27711/1892375,
> UserState: /var/lib/cassandra/data/UserState/facebookPoker99-tmp-6-Index.=
db
> 27711/143216,
> UserState: /var/lib/cassandra/data/UserState/csArena-tmp-6-Data.db
> 27711/201188,
> UserState: /var/lib/cassandra/data/UserState/darkChess-tmp-12-Index.db
> 27711/354923,
> UserState: /var/lib/cassandra/data/UserState/big2-tmp-12-Data.db
> 27711/1260768,
> UserState: /var/lib/cassandra/data/UserState/mahjong-tmp-12-Index.db
> 27711/332649,
> UserState: /var/lib/cassandra/data/UserState/battleCity2-tmp-6-Index.db
> 27711/39739
> ]
>
> lots of files stalled after receiving 27711 bytes. this strange number is
> the length of first file to income, see Host B
>
> Host B, 192.168.2.88
>
> StreamingService Status:
> Receiving stream
>
> StreamingService StreamSources:
> StreamSources: [/192.168.2.87]
>
> StreamingService StreamDestinations:
> =A0[/192.168.2.87]
>
> StreamingService getOutgoingFiles=3D192.168.2.87
> [
> /var/lib/cassandra/data/UserState/stream/csArena-6-Index.db 27711/27711,
> /var/lib/cassandra/data/UserState/stream/csArena-6-Filter.db 0/1165,
> /var/lib/cassandra/data/UserState/stream/csArena-6-Data.db 0/201188,
>
> ... omit pending outgoing files ....
> ]
>
> It seems that outgoing files does not terminate properly. and cause the
> receiver goes into infinite loop to cause busy thread. From thread dump, =
it
> looks like fc.transferFrom() in IncomingStreamReader never return:
>
> =A0=A0=A0 while (bytesRead < pendingFile.getExpectedBytes()) {
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 bytesRead +=3D fc.transferF=
rom(socketChannel, bytesRead,
> FileStreamTask.CHUNK_SIZE);
> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 pendingFile.update(bytesRea=
d);
> =A0=A0=A0 }
>
>
> On Tue, Apr 20, 2010 at 05:48, Rob Coli <rcoli@digg.com> wrote:
>>
>> On 4/17/10 6:47 PM, Ingram Chen wrote:
>>>
>>> after upgrading jdk from =A01.6.0_16 to =A01.6.0_20, the problem solved=
.
>>
>> FYI, this sounds like it might be :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-896
>>
>> http://bugs.sun.com/view_bug.do;jsessionid=3D60c39aa55d3666c0c84dd70eb82=
6?bug_id=3D6805775
>>
>> Where garbage collection issues in JVM/JDKs before 7.b70 leads to GC
>> storming which hoses performance.
>>
>> =3DRob
>
>
>
>
>