Well this is bad news... Why is this logged only as a WARN? Seems quite SEVERE to me.
Fortunately for my app, I can go back to my source, recreate the data and reload it at a slower rate (and cross my fingers that this rate is 'slow enough')... unfortunately, it took me 20 hours to load the data (200M records) at a rate of 3K/sec. It looks like I only dropped about 600K records out of 200M (.3%) ... but no way to tell which ones were dropped. :(
(I know, I know, more machines... they are on order... sigh.)
Is there a way to configure Cassandra so that it won't do this? I'd rather have it run slow than drop anything on the floor.
updates being dropped on the floor is exactly what that message means
(the batch_mutate you send are decomposed under the hood to individual
rows for replication, which is why the number is greater than your
On Fri, Aug 20, 2010 at 3:50 PM, Ronald Park
> How about this message:
> WARN [DroppedMessagesLogger] 2010-08-20 08:02:27,668 MessagingService.java
> e 512) Dropped 46469 messages in the last 1000ms
> For this application, I also have a 3-node cluster but I'm running a version
> of Cassandra built off the trunk (we wanted the 'time to live' feature
> before 0.7.beta was out). My application runs a daily loading script which
> peaks at under 3000 batch_mutate requests per second, well below the 46K in
> this log. Are these some sort of internal messages, like pings? Or are
> updates being dropped on the floor?
> On Wed, Aug 18, 2010 at 9:13 AM, Jonathan Ellis <firstname.lastname@example.org> wrote:
>> This is Cassandra trying to protect itself from the load spike. You
>> probably need to balance your nodes, tune for performance, add
>> capacity, or some combination.
>> On Wed, Aug 18, 2010 at 6:16 AM, Jianing Hu <email@example.com> wrote:
>> > We have a 3-node cluster of Cassandra 0.6.4, on one of them there's a
>> > ton of error message like the following one:
>> > WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-18 04:10:57,767
>> > MessageDeserializationTask.java (line 47) dropping message (1ms past
>> > timeout)
>> > The nod's load spikes and is super slow to read from. Any ideas?
>> > Thanks.
>> > - Jianing
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support