Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com
 designates 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=odizTaupzHLLo88iBqr1ujo5IyGGflxggpSq+UPbTxHtuBgwwTn/CEGu8ImI7XP3IW
         X7Bjl5r5c3YTp7W4THmb2BxLMlFaTbTCNTYqri+4wRDDAPQH7QrRcXuOw5L5JCgUn/VQ
         LUH+WzqJBysnzGonP0tiFKu5sE0yKimdIt0M8=
MIME-Version: 1.0
In-Reply-To: <FF03ACCA-0557-4C45-9E11-27DE73BF1626@doddington.net>
References: <FF03ACCA-0557-4C45-9E11-27DE73BF1626@doddington.net>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 17 Mar 2011 09:58:34 +0530
Message-ID: <AANLkTinYE1OQqzHhxktJVcBwDgMuGXBfA4pR50rrS2b=@mail.gmail.com>
Subject: Re: How does sqoop distribute it's data evenly across HDFS?
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

There's a balancer available to re-balance DNs across the HDFS cluster
in general. It is available in the $HADOOP_HOME/bin/ directory as
start-balancer.sh

But what I think sqoop implies is that your data is balanced due to
the map jobs it runs for imports (using a provided split factor
between maps), which should make it write chunks of data out to
different DataNodes.

I guess you could get more information on the Sqoop mailing list
sqoop-user@cloudera.org,
https://groups.google.com/a/cloudera.org/group/sqoop-user/topics

On Thu, Mar 17, 2011 at 5:04 AM, BeThere <andy@doddington.net> wrote:
> The sqoop documentation seems to imply that it uses the key information p=
rovided to it on the command line to ensure that the SQL data is distribute=
d evenly across the DFS. However I cannot see any mechanism for achieving t=
his explicitly other than relying on the implicit distribution provided by =
default by HDFS. Is this correct or are there methods on some API that allo=
w me to manage the distribution to ensure that it is balanced across all no=
des in my cluster?
>
> Thanks,
>
> =A0 =A0 =A0 =A0 Andy D
>
>


--=20
Harsh J
http://harshj.com