Well this is bad news... Why is this logged only as a WARN?  Seems quite SEVERE to me.

Fortunately for my app, I can go back to my source, recreate the data and reload it at a slower rate (and cross my fingers that this rate is 'slow enough')... unfortunately, it took me 20 hours to load the data (200M records) at a rate of 3K/sec.  It looks like I only dropped about 600K records out of 200M (.3%) ... but no way to tell which ones were dropped. :(

(I know, I know, more machines... they are on order... sigh.)

Is there a way to configure Cassandra so that it won't do this?  I'd rather have it run slow than drop anything on the floor.

Thanks,
Ron


On Fri, Aug 20, 2010 at 4:58 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
updates being dropped on the floor is exactly what that message means

(the batch_mutate you send are decomposed under the hood to individual
rows for replication, which is why the number is greater than your
3000)

On Fri, Aug 20, 2010 at 3:50 PM, Ronald Park
<ronald.park@cbsinteractive.com> wrote:
> How about this message:
>
>  WARN [DroppedMessagesLogger] 2010-08-20 08:02:27,668 MessagingService.java
> (lin
> e 512) Dropped 46469 messages in the last 1000ms
>
> For this application, I also have a 3-node cluster but I'm running a version
> of Cassandra built off the trunk (we wanted the 'time to live' feature
> before 0.7.beta was out).  My application runs a daily loading script which
> peaks at under 3000 batch_mutate requests per second, well below the 46K in
> this log.  Are these some sort of internal messages, like pings? Or are
> updates being dropped on the floor?
>
> Thanks,
> Ron
>
> On Wed, Aug 18, 2010 at 9:13 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> This is Cassandra trying to protect itself from the load spike.  You
>> probably need to balance your nodes, tune for performance, add
>> capacity, or some combination.
>>
>> On Wed, Aug 18, 2010 at 6:16 AM, Jianing Hu <jianing@gmail.com> wrote:
>> > We have a 3-node cluster of Cassandra 0.6.4, on one of them there's a
>> > ton of error message like the following one:
>> > WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-18 04:10:57,767
>> > MessageDeserializationTask.java (line 47) dropping message (1ms past
>> > timeout)
>> >
>> > The nod's load spikes and is super slow to read from. Any ideas?
>> >
>> > Thanks.
>> >
>> > - Jianing
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com