cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kjellman (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-4813) Problem using BulkOutputFormat while streaming several SSTables simultaneously from a given node.
Date Fri, 09 Nov 2012 23:15:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494317#comment-13494317
] 

Michael Kjellman edited comment on CASSANDRA-4813 at 11/9/12 11:13 PM:
-----------------------------------------------------------------------

yes, actually the first time I tested I did forget to ensure both the Hadoop jar and Cassandra
nodes had the newest patched version. I've tested a few times now just to make sure i'm not
missing anything. Seems to die as soon as it tries to stream the first sstable to the nodes.
Never progresses past 0% on the streaming and then throws the exception.

applied patch to trunk when it was at commit f09a89f4cd13af2087fcc92f09f6cf1ee4785feb. i rebuilt
the entire cluster, and ensured my maven dependencies were all set. Still reproduced the problem
unfortunately (i actually thought it had been resolved but i just reproduced the java.io.IOException:
Broken pipe again).

MD5 (build/apache-cassandra-1.2.0-beta2-SNAPSHOT.jar) = 92d8ffacb3963116dd153a2c8c83fbe9
                
      was (Author: mkjellman):
    yes, actually the first time i didn't and then i realized i hadn't rebuilt the Cassandra
nodes and only replaced the compiled jar in my hadoop job. I've tested a few times now just
to make sure i'm not missing anything. Seems to die as soon as it tries to stream the first
sstable to the nodes. Never progresses past 0% on the streaming and then throws the exception.
                  
> Problem using BulkOutputFormat while streaming several SSTables simultaneously from a
given node.
> -------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-4813
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4813
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.0
>         Environment: I am using SLES 10 SP3, Java 6, 4 Cassandra + Hadoop nodes, 3 Hadoop
only nodes (datanodes/tasktrackers), 1 namenode/jobtracker. The machines used are Six-Core
AMD Opteron(tm) Processor 8431, 24 cores and 33 GB of RAM. I get the issue on both cassandra
1.1.3, 1.1.5 and I am using Hadoop 0.20.2.
>            Reporter: Ralph Romanos
>            Assignee: Yuki Morishita
>            Priority: Minor
>              Labels: Bulkoutputformat, Hadoop, SSTables
>             Fix For: 1.2.0 rc1
>
>         Attachments: 4813.txt
>
>
> The issue occurs when streaming simultaneously SSTables from the same node to a cassandra
cluster using SSTableloader. It seems to me that Cassandra cannot handle receiving simultaneously
SSTables from the same node. However, when it receives simultaneously SSTables from two different
nodes, everything works fine. As a consequence, when using BulkOutputFormat to generate SSTables
and stream them to a cassandra cluster, I cannot use more than one reducer per node otherwise
I get a java.io.EOFException in the tasktracker's logs and a java.io.IOException: Broken pipe
in the Cassandra logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message