Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 29 Oct 2013 20:47:26 +0000 (UTC)
From: "Brandon Williams (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12676474.1383076076396.1503.1383079646065@arcas>
In-Reply-To: <JIRA.12676474.1383076076396@arcas>
References: <JIRA.12676474.1383076076396@arcas>
Subject: [jira] [Commented] (CASSANDRA-6268) Poor performance of Hadoop if
 any DC is using VNodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CASSANDRA-6268?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
13808433#comment-13808433 ]=20

Brandon Williams commented on CASSANDRA-6268:
---------------------------------------------

We could make something like describe_local_ring where it would only take i=
nto account the node's local dc.

> Poor performance of Hadoop if any DC is using VNodes
> ----------------------------------------------------
>
>                 Key: CASSANDRA-6268
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6268
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Hadoop
>            Reporter: Piotr Ko=C5=82aczkowski
>            Assignee: Piotr Ko=C5=82aczkowski
>         Attachments: 0001-DSP-2572-Adds-ability-to-set-target-DCs-where-a=
-Hado.patch
>
>
> Some customers are complaining about huge number of splits in Hadoop caus=
ed by VNodes. Disabling vnodes only in Hadoop DC does not fix it. Splits ar=
e generated from the results of describe_ring, which returns a huge number =
of ranges anyways, and doesn't take into account that there will be huge nu=
mber of consecutive ranges residing on the nodes we'd like the M/R job to b=
e run.
> The proposed fix:
> 1. allows for specifying the DC(s) the Hadoop job should be run in (in DS=
E - defaults to all Hadoop DCs)
> 2. merges consecutive ranges before generating Hadoop splits, so we don't=
 have artificial range splitting caused by vnodes in the other DCs
> For non-DSE users this feature is turned off by default and doesn't chang=
e the old behaviour.


--
This message was sent by Atlassian JIRA
(v6.1#6144)