cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra with large number of columns per row
Date Tue, 21 Aug 2012 09:03:00 GMT
> I think the limit of the size per row in cassandra is 2G?
That was a pre 0.7 restriction http://wiki.apache.org/cassandra/CassandraLimitations


> and I insert 10000 columns into a row, each column has a 1MB data. 
So a single row with 10GB of data. That's what we call a big one.


> /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/socket.rb:109:in
`read': CassandraThrift::Cassandra::Client::TransportException
>        from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/base_transport.rb:87:in
`read_all'
I would expect to see a string description there as well. Check on the server side error logs.


> this script crashed again, same error message. And cassandra process remain in 100% cpu
usage.
Counting columns involves reading them, so you are asking cassandra to read 10GB of data.
This will take a while. 


It's probably the size of the row that is causing problems. You can easily have rows with
millions of columns (here is an experiment that uses 10MM cols in a row http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/)


In general you will want to avoid rows with more than say 32 or 64 MB of data. It's not a
hard restriction but big rows cause issues and it's often easier to avoid them.   

Hope that helps. 
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 20/08/2012, at 8:15 PM, Chuan-Heng Hsiao <hsiao.chuanheng@gmail.com> wrote:

> I think the limit of the size per row in cassandra is 2G?
> 
> 10000 x 1M = 10G.
> 
> Hsiao
> 
> On Mon, Aug 20, 2012 at 1:07 PM, oupfevph <oupfevph@yahoo.com> wrote:
> I setup cassandra with default configuration in clean AWS instance, and I insert 10000
columns into a row, each column has a 1MB data. I use this ruby(version 1.9.3) script:
> 
>     10000.times do
>         key = rand(36**8).to_s(36)
>         value = rand(36**1024).to_s(36) * 1024
>         Cas_client.insert(TestColumnFamily,TestRow,{key=>value})
>     end
> 
> every time I run this script, it will crash:
> 
> /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/socket.rb:109:in
`read': CassandraThrift::Cassandra::Client::TransportException
>         from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/base_transport.rb:87:in
`read_all'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/framed_transport.rb:104:in
`read_frame'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/transport/framed_transport.rb:69:in
`read_into_buffer'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/client.rb:45:in
`read_message_begin'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/thrift-0.8.0/lib/thrift/client.rb:45:in
`receive_message'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/vendor/0.8/gen-rb/cassandra.rb:251:in
`recv_batch_mutate'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/vendor/0.8/gen-rb/cassandra.rb:243:in
`batch_mutate'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/thrift_client-0.8.1/lib/thrift_client/abstract_thrift_client.rb:150:in
`handled_proxy'        from /usr/local/lib/ruby/gems/1.9.1/gems/thrift_client-0.8.1/lib/thrift_client/abstract_thrift_client.rb:60:in
`batch_mutate'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/lib/cassandra/protocol.rb:7:in
`_mutate'
>         from /usr/local/lib/ruby/gems/1.9.1/gems/cassandra-0.15.0/lib/cassandra/cassandra.rb:463:in
`insert'
>         from a.rb:6:in `block in <main>'
>         from a.rb:3:in `times'
>         from a.rb:3:in `<main>'
> 
> yet cassandra performs normally, then I run another ruby script to get how many columns
I have inserted:
> 
>     p cas_client.count_columns(TestColumnFamily,TestRow)
> 
> this script crashed again, same error message. And cassandra process remain in 100% cpu
usage.
> 
> 
>     AWS m1.xlarge type instance (15GB mem,800GB harddisk, 4cores cpu)
>     cassandra-1.1.2
>     ruby-1.9.3-p194
>     jdk-7u6-linux-x64
>     ruby-gems:
>         cassandra (0.15.0)
>         thrift (0.8.0)
>         thrift_client (0.8.1)
> 
> What is the problem?
> 
> 


Mime
View raw message