cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Horman (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1175) Client/Server deadlock in aggressive bulk upload (maybe a thrift bug?) (2010-06-07_12-31-16)
Date Tue, 05 Oct 2010 16:41:38 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918064#action_12918064
] 

Jason Horman commented on CASSANDRA-1175:
-----------------------------------------

We seem to be experiencing this same issue against cassandra 0.6.5.

Did you manage to find a work around Jesse?

> Client/Server deadlock in aggressive bulk upload (maybe a thrift bug?) (2010-06-07_12-31-16)
> --------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-1175
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1175
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: apache-cassandra-2010-06-07_12-31-16
>            Reporter: Jesse Hallio
>
> I was testing to see how long it takes to upload some 222M lines into Cassandra. Using
a single machine (4-core, 8G of mem, opensolaris, 1.6.0_10 64bit and 32bit) to run the server
and client.
> The client creates a single keyspace with a single column family. The inserted data is
8-byte key, with 26 NVs (3-5 ascii bytes per key, 8 bytes per value) for each line. Using
RackUnawareStrategy and replication factor 1. The server install is pretty much out-of-the-box,
with -d64 -server -Xmx6G for the server end. (3G for the 32bit VM). The client writes the
changes in batches of 1000 lines with batch_mutate, and outputs a logging line every 50k lines.
> The import hangs at random points - sometimes after 6900K mark (I think I saw even >10M
yesterday, but I lost the window and the backbuffer with it), sometimes only after 1750K.
kill -QUIT gives for the server:
> ---
> "pool-1-thread-1" prio=3 tid=0x0000000000b68800 nid=0x5c runnable [0xfffffd7e676f5000..0xfffffd7e676f5920]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>         - locked <0xfffffd7e7ac80fb8> (a java.io.BufferedInputStream)
>         at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>         at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>         at org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
>         at org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:12840)
>         at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:1743)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1317)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> ---
> and for the client:
> ---
> "main" prio=3 tid=0x08070000 nid=0x2 runnable [0xfe38e000..0xfe38ed38]
>    java.lang.Thread.State: RUNNABLE
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(SocketInputStream.java:129)
>         at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>         - locked <0xbad8a840> (a java.io.BufferedInputStream)
>         at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:126)
>         at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>         at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314)
>         at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262)
>         at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
>         at org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:745)
>         at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:729)
>         at mycode.MyClass.main(MyClass.java:169)
> ---
> It looks like both ends are trying to read simultaneously from each other, which kind
of looks like a thrift bug; but I don't have clear idea what happens in org.apache.cassandra.thrift.Cassandra.
> I tried using with thrift r948492, but it didn't help (I didn't recompile the interface
classes, I only switched the runtime jar).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message