incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronald Park <ronald.p...@cbsinteractive.com>
Subject Re: "MessageDeserializationTask.java (line 47) dropping message" errors
Date Fri, 20 Aug 2010 22:02:49 GMT
Well this is bad news... Why is this logged only as a WARN?  Seems quite
SEVERE to me.

Fortunately for my app, I can go back to my source, recreate the data and
reload it at a slower rate (and cross my fingers that this rate is 'slow
enough')... unfortunately, it took me 20 hours to load the data (200M
records) at a rate of 3K/sec.  It looks like I only dropped about 600K
records out of 200M (.3%) ... but no way to tell which ones were dropped. :(

(I know, I know, more machines... they are on order... sigh.)

Is there a way to configure Cassandra so that it won't do this?  I'd rather
have it run slow than drop anything on the floor.

Thanks,
Ron


On Fri, Aug 20, 2010 at 4:58 PM, Jonathan Ellis <jbellis@gmail.com> wrote:

> updates being dropped on the floor is exactly what that message means
>
> (the batch_mutate you send are decomposed under the hood to individual
> rows for replication, which is why the number is greater than your
> 3000)
>
> On Fri, Aug 20, 2010 at 3:50 PM, Ronald Park
> <ronald.park@cbsinteractive.com> wrote:
> > How about this message:
> >
> >  WARN [DroppedMessagesLogger] 2010-08-20 08:02:27,668
> MessagingService.java
> > (lin
> > e 512) Dropped 46469 messages in the last 1000ms
> >
> > For this application, I also have a 3-node cluster but I'm running a
> version
> > of Cassandra built off the trunk (we wanted the 'time to live' feature
> > before 0.7.beta was out).  My application runs a daily loading script
> which
> > peaks at under 3000 batch_mutate requests per second, well below the 46K
> in
> > this log.  Are these some sort of internal messages, like pings? Or are
> > updates being dropped on the floor?
> >
> > Thanks,
> > Ron
> >
> > On Wed, Aug 18, 2010 at 9:13 AM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >> This is Cassandra trying to protect itself from the load spike.  You
> >> probably need to balance your nodes, tune for performance, add
> >> capacity, or some combination.
> >>
> >> On Wed, Aug 18, 2010 at 6:16 AM, Jianing Hu <jianing@gmail.com> wrote:
> >> > We have a 3-node cluster of Cassandra 0.6.4, on one of them there's a
> >> > ton of error message like the following one:
> >> > WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-08-18 04:10:57,767
> >> > MessageDeserializationTask.java (line 47) dropping message (1ms past
> >> > timeout)
> >> >
> >> > The nod's load spikes and is super slow to read from. Any ideas?
> >> >
> >> > Thanks.
> >> >
> >> > - Jianing
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of Riptano, the source for professional Cassandra support
> >> http://riptano.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message