Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6F2821097D for ; Fri, 7 Jun 2013 06:43:19 +0000 (UTC) Received: (qmail 40883 invoked by uid 500); 7 Jun 2013 06:43:17 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 40841 invoked by uid 500); 7 Jun 2013 06:43:16 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 40797 invoked by uid 99); 7 Jun 2013 06:43:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jun 2013 06:43:14 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [98.138.121.129] (HELO nm49-vm1.bullet.mail.ne1.yahoo.com) (98.138.121.129) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jun 2013 06:43:05 +0000 Received: from [98.138.90.53] by nm49.bullet.mail.ne1.yahoo.com with NNFMP; 07 Jun 2013 06:42:43 -0000 Received: from [98.138.89.197] by tm6.bullet.mail.ne1.yahoo.com with NNFMP; 07 Jun 2013 06:42:43 -0000 Received: from [127.0.0.1] by omp1055.mail.ne1.yahoo.com with NNFMP; 07 Jun 2013 06:42:43 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 888355.46514.bm@omp1055.mail.ne1.yahoo.com Received: (qmail 1944 invoked by uid 60001); 7 Jun 2013 06:42:43 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1370587363; bh=I/2NHkFQRNsciyG7k6F16vzT1SSws1sjyxB/V1ImTLE=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=vOGrIIlLEbKybx6+fqGHjg/l4xS0FlEQyBNybTipsfK39t9vkcarV+i/EaDLFbOJvTr1m/fYdHnYuy8xClCKtsV0G7XC6lrBWiubmdp37MUXGfjN8LfRTN7ZqTZ6GYXu1AGn6OS8bXYpZZtjRiySViUQR7buTDujNDCFKuj5TPI= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=JELho82bLnQBK9GGl5/V9auA/ZvLAU1GlvwvFNGmgrqKNblIrY3tkNQcfDfyrLMDY5oY80JybtUiPO0m4JXvPiVtSXsfh67yexCeQnc9BgILO7MveigMxat9+Dkn28hVzIuk9fo2FmytbP2bbyL/JCfPIv8342PMyBboZPa4uZA=; X-YMail-OSG: _Qy7xmAVM1lPvnltJDZF5GfdOFp59gicYx1W8Lr2iKLJpR6 heACqLQtVuh1h97r9zrXEsOjmd0a2vTbbC__YlWjTP2d0c52b_rW0uiBlK5R dxBVIuUJxZ9ms1J2Bqduvp_Zp1qrzL3Bmmxr.d0ehiiV0lgM1HMwQR3dl2AM 5qV1ijgfsSuT54edM8aIF8f2AxILbUQnQyinfTyXdv_ilSAsC8m1h_k0M090 1wsg5xrhfLahVj1XMuXSVbzEJ9NqZ_BQ03nPgxvIApcp1ep9c.v5UAnykpDj Bzaam1raQxUMMyhI_uCkVYRr3EynwdtgsvXlfYk5weAPXeU3N6zFl4Jgs0S4 cE5qZ5FcAa.3rjC7NY2wnp_U3QPS.WYI8FFXEOVrDwL1IL1AT9bWgppK7ekJ WhnNCXswuCN8wxS8kDs9HhVJK.Lp1wwl8CAAgVHt5qrmIshRXXZoYqQruOg8 _Je0SAjDRwHLihgpiwZaJScMd4GURS0hSM97C82_BZiIjX5zb8h2sSUyDGkD dRzTKRuz.l62GvpOqvjZFXBIHU78h3_XSe5V.mnrFcVZZg2sT.PebEg-- Received: from [198.241.217.15] by web121604.mail.ne1.yahoo.com via HTTP; Thu, 06 Jun 2013 23:42:43 PDT X-Rocket-MIMEInfo: 002.001,SGksCsKgCkkgYW0gaGl0IGJ5IHNrZXdlZCBKb2luLCBteSBsYXN0IHJlZHVjZXIgaXMgZ2V0dGluZyBzYW1lIG51bWJlciBvZiBSZWR1Y2UgaW5wdXQgZ3JvdXBzL3JlY29yZHMuClJlZHVjZSBpbnB1dCBncm91cHPCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgNDMyLDQ0Niw5NDIKUmVkdWNlIHNodWZmbGUgYnl0ZXPCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDEzLDAxMiw2MTMsMjc1ClJlZHVjZSBpbnB1dCByZWNvcmRzwqDCoMKgwqDCoMKgwqDCoMKgwqDCoAEwAQEBAQ-- X-Mailer: YahooMailWebService/0.8.145.547 References: , Message-ID: <1370587363.93771.YahooMailNeo@web121604.mail.ne1.yahoo.com> Date: Thu, 6 Jun 2013 23:42:43 -0700 (PDT) From: sumit ghosh Reply-To: sumit ghosh Subject: Skewed Join To: "user@hive.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="-36511695-197678492-1370587363=:93771" X-Virus-Checked: Checked by ClamAV on apache.org ---36511695-197678492-1370587363=:93771 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi,=0A=A0=0AI am hit by skewed Join, my last reducer is getting same number= of Reduce input groups/records.=0AReduce input groups=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 432,446,942=0AReduce shuffle byt= es=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 13,012,613,275=0ARedu= ce input records=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 432,446,942=A0 =0A=A0=0AWhy is this happening? I have turned on skew jo= in optimization:=0Ahive.optimize.skewjoin=3Dtrue;=0Ahive.skewjoin.key=3D100= 000;=0A=A0=0A=A0=0AThanks,=0ASumit=A0 ---36511695-197678492-1370587363=:93771 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
H= i,
 
I am hit by skewed Join, my last reducer is gett= ing same number of Reduce input groups/records.
Reduce input groups     = ;            &n= bsp;    432,446,942
Reduce shuffle bytes    &nbs= p;             = 13,012,613,275
Reduce input records    &nbs= p;            &= nbsp;   432,446,942 
 
Why is this happening? I have turned on skew joi= n optimization:
hive.optimize.skewjoin=3Dtrue;
hive.skewjoin.key=3D100000;
&nbs= p;
&nbs= p;
= Thanks,
Sumit
 
---36511695-197678492-1370587363=:93771--