cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lu Ming" <>
Subject Hintedhandoff will never complete when a BIG rowmutation
Date Mon, 28 Jun 2010 10:29:08 GMT
     These days I found my Cassandra is strange, much slower than before.  And I Spent much
time to figure it out and today I got the answer.

    Some bad buy keeps on writing many data day and night, then made a very big row mutation
which size is about 140M.
In this period I restarted some Cassandra nodes, and when the nodes is alive again, them got
some hintedhandoff messages.
HintedHandOffManager.sendMessage() will send the rowmutations to these nodes, but the rowmutation
is too big to finish transferring in
8 seconds (defined in DatabaseDescriptor.getRpcTimeout()), and sendMessage() return false
when got a TimeoutException.

Every one hour HintedHandOffManager will check hintedhandoff ColumnFamily then send out the
big rowmutations to alive nodes,
It fails again because of the TimeoutException, so the task will never finish and the big
rowmutation is sending again and again.

   In multi-datacenters,  a big rowmutation can not be transferred in several seconds. so
It is a potential risk when  a big rowmutation occurs.

View raw message