incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julie <julie.su...@nextcentury.com>
Subject Re: java.lang.RuntimeException: java.io.IOException: Value too large for defined data type
Date Thu, 17 Jun 2010 03:51:43 GMT
Jonathan Ellis <jbellis <at> gmail.com> writes:

> > Another thing that is odd is that even when the server nodes are quiescent
> > because compacting is complete, I am still seeing cpu usage stay at
> > about 40% .  Even after several hours, no reading or writing to the 
database
> > and all compactions complete, the cpu usage is staying around 40%.
> 
> Possibly this is Hinted Handoff scanning going on.  You can rm
> data/system/Hint* (while the node is shut down) if you want to take a
> shot in the dark.  Otherwise you'll want to follow
> http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?
topic=/com.ibm.java.doc.igaa/_1vg0001475cb4a-1190e2e0f74-8000_1007.html
> to figure out which thread is actually consuming the CPU.
> 

Thank you for all of the helpful advice.  We upgraded to 0.6.2 to see if any 
problems would resolve but we're still seeing the 40% cpu usage on certain 
nodes several hours after any reads or writes have taken place. We attached to 
one of these super busy cassandra nodes and have isolated it to a single 
thread “Thread-32” that seems to be buzzing in a tight loop.

The loop is in IncomingStreamReader.java, line 62, a 3-line while loop.  
bytesRead is not changing.  pendingFile.getExpectedBytes() returns 
7,161,538,639 but bytesRead is stuck at 2,147,483,647.  

The stack is:

Sun.nio.ch.Util.getTemporaryDirectBuffer() – line 50
Sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel() – line 556
Sun.nio.ch.FileChannelImpl.transferFrom() – line 603
Org.apache.cassandra.streaming.IncomingStreamReader.read() – line 62
Org.apache.cassandra.net.IncomingTcpConnection.run() – line 66

This same 3 line loop in IncomingStreamReader was also getting stuck last 
night with 0.6.1 so whatever it is is still happening in 0.6.2.
 
Thanks for your help,
Julie




Mime
View raw message