Return-Path: 
 <user-return-10441-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-user-archive@www.apache.org
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA27610B0C
	for <apmail-hadoop-common-user-archive@www.apache.org>;
 Thu, 29 Aug 2013 23:23:33 +0000 (UTC)
Received: (qmail 56814 invoked by uid 500); 29 Aug 2013 23:23:28 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 56594 invoked by uid 500); 29 Aug 2013 23:23:28 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 56587 invoked by uid 99); 29 Aug 2013 23:23:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 23:23:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of adeelmahmood@gmail.com
 designates 209.85.212.175 as permitted sender)
Received: from [209.85.212.175] (HELO mail-wi0-f175.google.com)
 (209.85.212.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Aug 2013 23:23:21 +0000
Received: by mail-wi0-f175.google.com with SMTP id hi5so3352628wib.2
        for <user@hadoop.apache.org>; Thu, 29 Aug 2013 16:23:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=x5iAu6wwMzsEnC9ftr2YIlCEPGNz+QHpVDkOlqcDf5w=;
        b=vYbhAF0ZJZS0usMVtBD4km650iWy5v4LoeWT6VDAdJqkPL+HA7DkO2vfz+ttvhnj8W
         bh2B5i8Cu0mHvhrUev1wYidesnHo3alNhvL2O001TrQ393Z5oL4WjdDhXPdE3UtMZc3o
         6A/+hck9QZNqXVabD0krjew+Kv+GQoqifBjhwNgOvwFmqHRXBPNABmCaKxsuT0+vUeK0
         RJ7UeiNW/WM6/HSO95J4pggXlQCHkbQto/zxuRtHxFxCN0u7UlHoO17+UDU9cq402Hgk
         K8OR5DLIgfGAi4SYjY3FBttpf/SNU13G59XgajGmc7Pj8PbrhlZ3Kdx/qx5HoxVYRLGW
         w7KQ==
MIME-Version: 1.0
X-Received: by 10.195.13.45 with SMTP id ev13mr8382645wjd.20.1377818581610;
 Thu, 29 Aug 2013 16:23:01 -0700 (PDT)
Received: by 10.180.72.101 with HTTP; Thu, 29 Aug 2013 16:23:01 -0700 (PDT)
Date: Thu, 29 Aug 2013 19:23:01 -0400
Message-ID: 
 <CAH1YrM43=6DKE+EERJbJ1wRKyUX9zXWJVKQ2sMd8zN0sTas7_A@mail.gmail.com>
Subject: secondary sort - number of reducers
From: Adeel Qureshi <adeelmahmood@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7bfced1e1f369104e51e6381
X-Virus-Checked: Checked by ClamAV on apache.org

--047d7bfced1e1f369104e51e6381
Content-Type: text/plain; charset=ISO-8859-1

I have implemented secondary sort in my MR job and for some reason if i
dont specify the number of reducers it uses 1 which doesnt seems right
because im working with 800M+ records and one reducer slows things down
significantly. Is this some kind of limitation with the secondary sort that
it has to use a single reducer .. that kind of would defeat the purpose of
having a scalable solution such as secondary sort. I would appreciate any
help.

Thanks
Adeel

--047d7bfced1e1f369104e51e6381
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I have implemented secondary sort in my MR job and for som=
e reason if i dont specify the number of reducers it uses 1 which doesnt se=
ems right because im working with 800M+ records and one reducer slows thing=
s down significantly. Is this some kind of limitation with the secondary so=
rt that it has to use a single reducer .. that kind of would defeat the pur=
pose of having a scalable solution such as secondary sort. I would apprecia=
te any help.<div>
<br></div><div>Thanks</div><div>Adeel</div></div>

--047d7bfced1e1f369104e51e6381--