Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of ryan@twitter.com designates
 209.85.216.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=twitter.com; s=google;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=DIYPxK9SIYWCV8rpEVOTzEX9POlCZvAkhnO6/G4og9ZaUjEvvA5wZqAIPYkd1aqbnf
         S7qZXSUykz8nZ4BtO95KTVVssbfl9FUr++BD7MNV1pXIEo6LsPgrk0NjrGVfABCATcBa
         CF91XenX1tniBqV9xd+RlNoBJUzE5pgTms/0U=
MIME-Version: 1.0
In-Reply-To: <BANLkTikSCnSFJzEPeHea-C9DWbKkhQjoeg@mail.gmail.com>
References: <BANLkTikSCnSFJzEPeHea-C9DWbKkhQjoeg@mail.gmail.com>
Date: Thu, 7 Apr 2011 10:24:21 -0700
Message-ID: <BANLkTimnGUZEF8oKrqEFufbVLZFn1DMLgA@mail.gmail.com>
Subject: Re: problem with large batch mutation set
From: Ryan King <ryan@twitter.com>
To: user@cassandra.apache.org
Cc: Ross Black <ross.w.black@gmail.com>
Content-Type: multipart/alternative; boundary=0016e64de50441679a04a0576188

--0016e64de50441679a04a0576188
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Apr 6, 2011 at 11:49 PM, Ross Black <ross.w.black@gmail.com> wrote:

> Hi,
>
> I am using the thrift client batch_mutate method with Cassandra 0.7.0  on
> Ubuntu 10.10.
>
> When the size of the mutations gets too large, the client fails with the
> following exception:
>
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.SocketException: Connection reset
>     at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
>     at
> org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157)
>     at
> org.apache.cassandra.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:901)
>     at
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:889)
>     at
> com.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(BatchMutationTask.java:78)
>     at
> com.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(BatchMutationTask.java:30)
>     at
> com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassandraConnectionTemplate.java:316)
>     at
> com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassandraConnectionTemplate.java:257)
>     at
> com.cxense.cxad.core.persistence.cassandra.store.AbstractCassandraStore.writeMutations(AbstractCassandraStore.java:492)
>     ... 39 more
> Caused by: java.net.SocketException: Connection reset
>     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
>     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>     at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
>     at
> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
>     ... 47 more
>
>
> It took a while for me to discover that this obscure error message was as a
> result of the thrift message exceeding the maximum frame size specified for
> the Cassandra server. (default 15MB)
> [Using TFastFramedTransport with a max frame size of 15728640 bytes.]
>
> The poor error message looks like a bug in Thrift server code that assumes
> that any transport exception is a connection failure which should drop the
> connection.
>
> *
> *My main problem is how to ensure that this does not occur again in
> running code.
> I could configure the server with a larger frame size, but this size is
> effectively arbitrary so there is no guarantee in our code would stop a very
> large mutation being sent occasionally.
> My current work-around is to break the mutation list into multiple parts,
> but to do this correctly I need to track the size of each mutation which is
> fairly messy.
> *
> Is there some way to configure Thrift or Cassandra to deal with messages
> that are larger than the max frame size (at either client or server) ?*


The only way to do that is to set the frame size higher. Messages cannot be
bigger than the maximum frame size.

-ryan

--0016e64de50441679a04a0576188
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote">On Wed, Apr 6, 2011 at 11:49 PM, Ross Black <spa=
n dir=3D"ltr">&lt;<a href=3D"mailto:ross.w.black@gmail.com">ross.w.black@gm=
ail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D=
"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi,<br><br>I am using the thrift client batch_mutate method with Cassandra =
0.7.0=A0 on Ubuntu 10.10.<br><br>When the size of the mutations gets too la=
rge, the client fails with the following exception:<br><br>Caused by: org.a=
pache.thrift.transport.TTransportException: java.net.SocketException: Conne=
ction reset<br>

=A0=A0=A0 at org.apache.thrift.transport.TIOStreamTransport.write(TIOStream=
Transport.java:147)<br>=A0=A0=A0 at org.apache.thrift.transport.TFramedTran=
sport.flush(TFramedTransport.java:157)<br>=A0=A0=A0 at org.apache.cassandra=
.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:901)<br>

=A0=A0=A0 at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cass=
andra.java:889)<br>=A0=A0=A0 at com.cxense.cxad.core.persistence.cassandra.=
store.BatchMutationTask.apply(BatchMutationTask.java:78)<br>=A0=A0=A0 at co=
m.cxense.cxad.core.persistence.cassandra.store.BatchMutationTask.apply(Batc=
hMutationTask.java:30)<br>

=A0=A0=A0 at com.cxense.cassandra.conn.DefaultCassandraConnectionTemplate.e=
xecute(DefaultCassandraConnectionTemplate.java:316)<br>=A0=A0=A0 at com.cxe=
nse.cassandra.conn.DefaultCassandraConnectionTemplate.execute(DefaultCassan=
draConnectionTemplate.java:257)<br>

=A0=A0=A0 at com.cxense.cxad.core.persistence.cassandra.store.AbstractCassa=
ndraStore.writeMutations(AbstractCassandraStore.java:492)<br>=A0=A0=A0 ... =
39 more<br>Caused by: java.net.SocketException: Connection reset<br>=A0=A0=
=A0 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)<=
br>

=A0=A0=A0 at java.net.SocketOutputStream.write(SocketOutputStream.java:136)=
<br>=A0=A0=A0 at java.io.BufferedOutputStream.write(BufferedOutputStream.ja=
va:105)<br>=A0=A0=A0 at org.apache.thrift.transport.TIOStreamTransport.writ=
e(TIOStreamTransport.java:145)<br>

=A0=A0=A0 ... 47 more<br><br><br>It took a while for me to discover that th=
is obscure error message was as a result of the thrift message exceeding th=
e maximum frame size specified for the Cassandra server. (default 15MB)<br>=
[Using TFastFramedTransport with a max frame size of 15728640 bytes.] <br>

<br>The poor error message looks like a bug in Thrift server code that assu=
mes that any transport exception is a connection failure which should drop =
the connection.<br><br><b><br></b>My main problem is how to ensure that thi=
s does not occur again in running code.<br>

I could configure the server with a larger frame size, but this size is eff=
ectively arbitrary so there is no guarantee in our code would stop a very l=
arge mutation being sent occasionally.<br>My current work-around is to brea=
k the mutation list into multiple parts, but to do this correctly I need to=
 track the size of each mutation which is fairly messy.<br>

<b><br>Is there some way to configure Thrift or Cassandra to deal with mess=
ages that are larger than the max frame size (at either client or server) ?=
</b></blockquote><div><br></div><div>The only way to do that is to set the =
frame size higher. Messages cannot be bigger than the maximum frame size.</=
div>
<div><br></div><div>-ryan=A0</div></div>

--0016e64de50441679a04a0576188--