Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9067DEA9B for ; Wed, 16 Jan 2013 20:27:00 +0000 (UTC) Received: (qmail 27321 invoked by uid 500); 16 Jan 2013 20:26:59 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 27240 invoked by uid 500); 16 Jan 2013 20:26:58 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 27231 invoked by uid 99); 16 Jan 2013 20:26:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jan 2013 20:26:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.223.181 as permitted sender) Received: from [209.85.223.181] (HELO mail-ie0-f181.google.com) (209.85.223.181) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jan 2013 20:26:53 +0000 Received: by mail-ie0-f181.google.com with SMTP id 16so3390788iea.12 for ; Wed, 16 Jan 2013 12:26:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=NOxzwR4RIhED7iSvio15CT0eSMx0G5zpHRKdJ9qv5AU=; b=VhAwGn7lje/YZDI8U9xehrZjtG1GlUerf6+pKHHBTLvGAr5Qh+mM4DdhzSbD6QoTvA OQ4rxcAyTd7FoANcfyFC9nPQTSTKPGOi0/ZaMBNlwyTnqGjUze9Gnv+4lceL2++VntMm qsTQeAReWJtc/og5DsOOuNhMQKtmMkU8qDvfl+AcsFV4Ub+c54VVsTgcNgOQxV42ajw7 jFfSQeUK/f4oWdpQ9ZTIAtKhmJYGblZ7ASFR9vU06K4B1mKymTCyPcapr7eLxjG3JHO+ Jb0+RnAtKdzFMHtFCxRf5PdZsWMswLjQ95NosflnoyFmI9s/KSzmdJYAYKUYwNvCa/RB Kn8Q== MIME-Version: 1.0 X-Received: by 10.42.249.80 with SMTP id mj16mr1501027icb.53.1358367993386; Wed, 16 Jan 2013 12:26:33 -0800 (PST) Received: by 10.64.32.166 with HTTP; Wed, 16 Jan 2013 12:26:33 -0800 (PST) Received: by 10.64.32.166 with HTTP; Wed, 16 Jan 2013 12:26:33 -0800 (PST) In-Reply-To: References: Date: Thu, 17 Jan 2013 01:56:33 +0530 Message-ID: Subject: RE: Limitation of key-value pairs for a particular key. From: Harsh J To: mapreduce-user Content-Type: multipart/alternative; boundary=20cf3011d8b1b8572d04d36db11b X-Gm-Message-State: ALoCoQmGnZf2g/ffehcwgl430N15qKE/5RDBkVMmBsqtQbY+ureUOQrD+WljYHEKfnlgQ0dJJPZx X-Virus-Checked: Checked by ClamAV on apache.org --20cf3011d8b1b8572d04d36db11b Content-Type: text/plain; charset=ISO-8859-1 We don't sort values (only keys) nor apply any manual limits in MR. Can your post a reproduceable test case to support your suspicion? On Jan 16, 2013 4:34 PM, "Utkarsh Gupta" wrote: > Hi,**** > > Thanks for the response. There was some issues with my code. I have > checked that in detail. **** > > All the values of map are present in reducer but not in sorted order. This > case happens if the number of values are too large for a key. **** > > ** ** > > Thanks**** > > Utkarsh**** > > ** ** > > *From:* Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com] > *Sent:* Thursday, January 10, 2013 11:00 PM > *To:* mapreduce-user@hadoop.apache.org > *Subject:* Re: Limitation of key-value pairs for a particular key.**** > > ** ** > > There isn't any limit like that. Can you reproduce this consistently? If > so, please file a ticket. > > It will definitely help if you can provide a test case which can reproduce > this issue. > > Thanks, > +Vinod**** > > ** ** > > On Thu, Jan 10, 2013 at 12:41 AM, Utkarsh Gupta > wrote:**** > > Hi,**** > > **** > > I am using Apache Hadoop 1.0.4 on a 10 node cluster of commodity machines > with Ubuntu 12.04 Server edition. I am having a issue with my map reduce > code. While debugging I found that the reducer can take 262145 values for a > particular key. If more values are there, they seem to be corrupted. I > checked the values while emitting from map and again checked in reducer.** > ** > > I am wondering is there any such kind of limitation in the Hadoop or is it > a configuration problem.**** > > **** > > **** > > Thanks and Regards**** > > Utkarsh Gupta**** > > **** > > **** > > **************** CAUTION - Disclaimer ********************* > > This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely **** > > for the use of the addressee(s). If you are not the intended recipient, please **** > > notify the sender by e-mail and delete the original message. Further, you are not **** > > to copy, disclose, or distribute this e-mail or its contents to any other person and **** > > any such actions are unlawful. This e-mail may contain viruses. Infosys has taken **** > > every reasonable precaution to minimize this risk, but is not liable for any damage **** > > you may sustain as a result of any virus in this e-mail. You should carry out your **** > > own virus checks before opening the e-mail or attachment. Infosys reserves the **** > > right to monitor and review the content of all messages sent to or from this e-mail **** > > address. Messages sent to or from this e-mail address may be stored on the **** > > Infosys e-mail system.**** > > ***INFOSYS******** End of Disclaimer ********INFOSYS******* > > > > > -- > +Vinod > Hortonworks Inc. > http://hortonworks.com/ ** ** > --20cf3011d8b1b8572d04d36db11b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

We don't sort values (only keys) nor apply any manual li= mits in MR. Can your post a reproduceable test case to support your suspici= on?

On Jan 16, 2013 4:34 PM, "Utkarsh Gupta&quo= t; <Utkarsh_Gupta@infosys.c= om> wrote:

Hi,<= /p>

Thanks for the response. = There was some issues with my code. I have checked that in detail.

All the values of map are= present in reducer but not in sorted order. This case happens if the numbe= r of values are too large for a key.

=A0<= /p>

Thanks

Utkarsh

=A0<= /p>

From: Vinod Ku= mar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Thursday, January 10, 2013 11:00 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Limitation of key-value pairs for a particular key.<= /u>

=A0

There isn't any limit like that. Can you reprodu= ce this consistently? If so, please file a ticket.

It will definitely help if you can provide a test case which can reproduce = this issue.

Thanks,
+Vinod

=A0

On Thu, Jan 10, 2013 at 12:41 AM, Utkarsh Gupta <= Utkarsh_Gupt= a@infosys.com> wrote:

Hi,

=A0

I am using Apache Hadoop 1.0.4 on a 10 node cluster = of commodity machines with Ubuntu 12.04 Server edition. I am having a issue= with my map reduce code. While debugging I found that the reducer can take 262145 values for a particular key. If more valu= es are there, they seem to be corrupted. I checked the values while emittin= g from map and again checked in reducer.

I am wondering is there any such kind of limitation = in the Hadoop or is it a configuration problem.

=A0

=A0

Thanks and Regards

Utkarsh Gupta

=A0

=A0

**************** CAUTION - Disclaimer *****************=

This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATI=
ON intended solely 
for the use of the addressee(s). If you are not the intend=
ed recipient, please 
notify the sender by e-mail and delete the original messag=
e. Further, you are not 
to copy, disclose, or distribute this e-mail or its conten=
ts to any other person and 
any such actions are unlawful. This e-mail may contain vir=
uses. Infosys has taken 
every reasonable precaution to minimize this risk, but is =
not liable for any damage 
you may sustain as a result of any virus in this e-mail. Y=
ou should carry out your 
own virus checks before opening the e-mail or attachment. =
Infosys reserves the 
right to monitor and review the content of all messages se=
nt to or from this e-mail 
address. Messages sent to or from this e-mail address may =
be stored on the 
Infosys e-mail system.
***INFOSYS******** End of Disclaimer ********INFOSYS***=




--
+Vinod
Hortonworks Inc.
http://hortonworks.co= m/

--20cf3011d8b1b8572d04d36db11b--