incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: java.lang.RuntimeException: java.io.IOException: Value too large for defined data type
Date Thu, 17 Jun 2010 05:33:50 GMT
That is consistent with the
https://issues.apache.org/jira/browse/CASSANDRA-1169 bug I mentioned,
that is fixed in the 0.6 svn branch.

On Wed, Jun 16, 2010 at 10:51 PM, Julie <julie.sugar@nextcentury.com> wrote:
> Jonathan Ellis <jbellis <at> gmail.com> writes:
>
>> > Another thing that is odd is that even when the server nodes are quiescent
>> > because compacting is complete, I am still seeing cpu usage stay at
>> > about 40% .  Even after several hours, no reading or writing to the
> database
>> > and all compactions complete, the cpu usage is staying around 40%.
>>
>> Possibly this is Hinted Handoff scanning going on.  You can rm
>> data/system/Hint* (while the node is shut down) if you want to take a
>> shot in the dark.  Otherwise you'll want to follow
>> http://publib.boulder.ibm.com/infocenter/javasdk/tools/index.jsp?
> topic=/com.ibm.java.doc.igaa/_1vg0001475cb4a-1190e2e0f74-8000_1007.html
>> to figure out which thread is actually consuming the CPU.
>>
>
> Thank you for all of the helpful advice.  We upgraded to 0.6.2 to see if any
> problems would resolve but we're still seeing the 40% cpu usage on certain
> nodes several hours after any reads or writes have taken place. We attached to
> one of these super busy cassandra nodes and have isolated it to a single
> thread “Thread-32” that seems to be buzzing in a tight loop.
>
> The loop is in IncomingStreamReader.java, line 62, a 3-line while loop.
> bytesRead is not changing.  pendingFile.getExpectedBytes() returns
> 7,161,538,639 but bytesRead is stuck at 2,147,483,647.
>
> The stack is:
>
> Sun.nio.ch.Util.getTemporaryDirectBuffer() – line 50
> Sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel() – line 556
> Sun.nio.ch.FileChannelImpl.transferFrom() – line 603
> Org.apache.cassandra.streaming.IncomingStreamReader.read() – line 62
> Org.apache.cassandra.net.IncomingTcpConnection.run() – line 66
>
> This same 3 line loop in IncomingStreamReader was also getting stuck last
> night with 0.6.1 so whatever it is is still happening in 0.6.2.
>
> Thanks for your help,
> Julie
>
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message