Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DB32D7BC3 for ; Wed, 21 Sep 2011 01:02:52 +0000 (UTC) Received: (qmail 33311 invoked by uid 500); 21 Sep 2011 01:02:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 33276 invoked by uid 500); 21 Sep 2011 01:02:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 33268 invoked by uid 99); 21 Sep 2011 01:02:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Sep 2011 01:02:50 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tyler@datastax.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Sep 2011 01:02:45 +0000 Received: by fxd18 with SMTP id 18so1182402fxd.31 for ; Tue, 20 Sep 2011 18:02:23 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.17.3 with SMTP id q3mr54400faa.71.1316566943591; Tue, 20 Sep 2011 18:02:23 -0700 (PDT) Received: by 10.152.24.65 with HTTP; Tue, 20 Sep 2011 18:02:23 -0700 (PDT) X-Originating-IP: [70.124.83.21] In-Reply-To: References: Date: Tue, 20 Sep 2011 20:02:23 -0500 Message-ID: Subject: Re: TransportException when storing large values From: Tyler Hobbs To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00151743f846feef9604ad692063 --00151743f846feef9604ad692063 Content-Type: text/plain; charset=ISO-8859-1 >From cassandra.yaml: # Frame size for thrift (maximum field length). # 0 disables TFramedTransport in favor of TSocket. This option # is deprecated; we strongly recommend using Framed mode. thrift_framed_transport_size_in_mb: 15 So you can either increase that limit, or split your write into multiple operations. On Tue, Sep 20, 2011 at 4:26 PM, Pete Warden wrote: > I'm running into consistent problems when storing values larger than 15MB > into Cassandra, and I was hoping for some advice on tracking down what's > going wrong. From the FAQ it seems like what I'm trying to do is possible, > so I assume I'm messing something up with my configuration. I have a minimal > set of code to reproduce the issue below, which I've run on the DataStax > 0.8.1 AMI I'm using in production (ami-9996c4dc) > > # To set up the test data structure on Cassandra: > > cassandra-cli > connect localhost/9160; > create keyspace TestKeyspace with > placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy' and > strategy_options = [{replication_factor:3}]; > use TestKeyspace; > create column family TestFamily with > comparator = UTF8Type and > column_metadata = > [ > {column_name: test_column, validation_class: UTF8Type}, > ]; > > # From bash on the same machine, with Ruby and the Cassandra gem installed: > > irb > require 'rubygems' > require 'cassandra/0.8' > > client = Cassandra.new('TestKeyspace', 'localhost:9160', :retries => 5, > :connect_timeout => 5, :timeout => 10) > > # With data this size, the call works > column_value = 'a' * (14*1024*1024) > row_value = { 'column_name' => column_value } > client.insert(:TestFamily, 'SomeKey', row_value) > > > # With data this size, the call fails with the exception below > column_value = 'a' * (15*1024*1024) > row_value = { 'column_name' => column_value } > client.insert(:TestFamily, 'SomeKey', row_value) > > # Results: > > This first call with a 14MB chunk of data succeeds, but the second one > fails with this exception: > > CassandraThrift::Cassandra::Client::TransportException: > CassandraThrift::Cassandra::Client::TransportException > from > /usr/lib/ruby/gems/1.8/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:53:in > `open' > from > /usr/lib/ruby/gems/1.8/gems/thrift-0.7.0/lib/thrift/transport/framed_transport.rb:37:in > `open' > from > /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/connection/socket.rb:11:in > `connect!' > from > /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:105:in > `connect!' > from > /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:144:in > `handled_proxy' > from > /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:60:in > `batch_mutate' > from > /usr/lib/ruby/gems/1.8/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:7:in > `_mutate' > from > /usr/lib/ruby/gems/1.8/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:459:in > `insert' > from (irb):6 > from :0 > > Any suggestions on how to dig deeper? I'll be reaching out to the Cassandra > gem folks, etc too of course. > > cheers, > Pete > -- Tyler Hobbs Software Engineer, DataStax Maintainer of the pycassa Cassandra Python client library --00151743f846feef9604ad692063 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable >From cassandra.yaml:

# Frame size for thrift (maximum field length).=
# 0 disables TFramedTransport in favor of TSocket. This option
# is = deprecated; we strongly recommend using Framed mode.
thrift_framed_trans= port_size_in_mb: 15

So you can either increase that limit, or split your write into multipl= e operations.

On Tue, Sep 20, 2011 at 4:2= 6 PM, Pete Warden <pete@petewarden.com> wrote:
I'm running into consistent problems wh= en storing values larger than 15MB into Cassandra, and I was hoping for som= e advice on tracking down what's going wrong. From the FAQ it seems lik= e what I'm trying to do is possible, so I assume I'm messing someth= ing up with my configuration. I have a minimal set of code to reproduce the= issue below, which I've run on the DataStax 0.8.1 AMI I'm using in= production (ami-9996c4dc)

# To set up the test data structure on Cassandra:
=
cassandra-cli
connect localhost/9160;
create keyspace TestKeyspace with=A0
=A0 placement_strategy = =3D 'org.apache.cassandra.locator.SimpleStrategy' and=A0
=A0 strategy_options =3D [{replication_factor:3}];
use TestK= eyspace;
create column family TestFamily with
=A0 compa= rator =3D UTF8Type and
=A0 column_metadata =3D
=A0 [
=A0 =A0 {column_name: test_column, validation_class: UTF8Type},
=A0 ];

# From bash on the same machine, with = Ruby and the Cassandra gem installed:

irb
require 'rubygems'
require 'cassandra/0.8'

client =3D Cassandra.new('TestKeyspace', 'l= ocalhost:9160', :retries =3D> 5, :connect_timeout =3D> 5, :timeou= t =3D> 10)

# With data this size, the call work= s
column_value =3D 'a' * (14*1024*1024)
row_value =3D = { 'column_name' =3D> column_value }
client.insert(:Tes= tFamily, 'SomeKey', row_value)


<= div> # With data this size, the call fails with the exception below
co= lumn_value =3D 'a' * (15*1024*1024)
row_value =3D { '= column_name' =3D> column_value }
client.insert(:TestFamily= , 'SomeKey', row_value)

# Results:

This first call wit= h a 14MB chunk of data succeeds, but the second one fails with this excepti= on:

CassandraThrift::Cassandra::Client::TransportE= xception: CassandraThrift::Cassandra::Client::TransportException
from /usr/lib/ruby/gems/1= .8/gems/thrift-0.7.0/lib/thrift/transport/socket.rb:53:in `open'
<= div> from /usr/lib/ruby/gems/1.= 8/gems/thrift-0.7.0/lib/thrift/transport/framed_transport.rb:37:in `open= 9;
from /usr/lib/ruby/gems/1= .8/gems/thrift_client-0.7.1/lib/thrift_client/connection/socket.rb:11:in `c= onnect!'
from /u= sr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_th= rift_client.rb:105:in `connect!'
from /usr/lib/ruby/gems/1= .8/gems/thrift_client-0.7.1/lib/thrift_client/abstract_thrift_client.rb:144= :in `handled_proxy'
from /usr/lib/ruby/gems/1.8/gems/thrift_client-0.7.1/lib/thrift_client/= abstract_thrift_client.rb:60:in `batch_mutate'
from /usr/lib/ruby/gems/1= .8/gems/cassandra-0.12.1/lib/cassandra/protocol.rb:7:in `_mutate'
=
from /usr/lib/ruby/gems/1= .8/gems/cassandra-0.12.1/lib/cassandra/cassandra.rb:459:in `insert'
from (irb):6
from :0

Any suggestions on how to dig deeper? I'll be reaching out to th= e Cassandra gem folks, etc too of course.

cheers,
=A0 =A0 =A0 =A0 =A0 =A0Pete



--
Tyler Hobbs
Software Engineer, DataS= tax
Maintainer of the pycassa Cassandra Python client library
 --00151743f846feef9604ad692063--