cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Shook <jsh...@gmail.com>
Subject Re: java.lang.RuntimeException: java.io.IOException: Value too large for defined data type
Date Tue, 15 Jun 2010 23:58:25 GMT
Actually, you shouldn't expect errors in the general case, unless you
are simply trying to use data that can't fit in available heap. There
are some practical limitations, as always.

If there aren't enough resources on the server side to service the
clients, the expectation should be that the servers have a graceful
performance degradation, or in the worst case throw an error specific
to resource exhaustion or explicit resource throttling. The fact that
Cassandra does some background processing complicates this a bit.
There are things which can cause errors after the fact, but these are
generally considered resource tuning issues and are somewhat clear
cut. There are specific changes in the works to bring background load
exceptions into view of a client session, where users normally expect
them.

@see https://issues.apache.org/jira/browse/CASSANDRA-685

But otherwise, users shouldn't be expecting that simply increasing
client load can blow up their Cassandra cluster. Any time this
happens, it should be considered a bug or a misfeature. Devs please
correct me here if I'm wrong.

Jonathan


On Tue, Jun 15, 2010 at 6:44 PM, Charles Butterfield
<charles.butterfield@nextcentury.com> wrote:
> Benjamin Black <b <at> b3k.us> writes:
>
>>
>> I am only saying something obvious: if you don't have sufficient
>> resources to handle the demand, you should reduce demand, increase
>> resources, or expect errors.  Doing lots of writes without much heap
>> space is such a situation (whether or not it is happening in this
>> instance), but there are many others.  This constraint it not specific
>> to Cassandra.  Hence, there is no free lunch.
>>
>> b
>
> I guess my point is that I have rarely run across database servers that die
> from either too many client connections, or too rapid client requests.  They
> generally stop accepting incoming connections when there are too many connection
> requests, and further they do not queue and acknowledge an unbounded number of
> client requests on any given connection.
>
> In the example at hand, Julie has 8 clients, each of which is in a loop that
> writes 100 rows at a time (via batch_mutate), waits for successful completion,
> then writes another bunch of 100, until it completes all of the rows it is
> supposed to write (typically 100,000).  So at any one time, each client should
> have about 10 MB of request (100 rows x 100 KB/row), times 8 clients, for a max
> pending request of no more than 80 MB.
>
> Further each request is running with a CL=ALL, so in theory, the request should
> not complete until each row has been handed off to the ultimate destination
> node, and perhaps written to the commit log (that part is not clear to me).
>
> It sounds like something else must be gobbling up either an unbounded amount
> of heap, or alternatively, a bounded, but large amount of heap.  In the former
> case it is unclear how to make the application robust.  In the later, it would
> be helpful to understand what the heap ussage upper bound is, and what
> parameters might have a significant effect on that value.
>
> To clarify the history here -- initially we were writing with CL=0 and had
> great performance but ended up killing the server.  It was pointed out that
> we were really asking the server to accept and acknowledge an unbounded
> number of requests without waiting for any final disposition of the rows.
> So we had a "doh!" moment.  That is why we went to the other extreme of
> CL=ALL, to let the server fully dispose of each request before acknowledging
> it and getting the next.
>
> TIA
> -- Charlie
>
>
>
>
>

Mime
View raw message