Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A8D37105F7 for ; Wed, 9 Oct 2013 12:46:24 +0000 (UTC) Received: (qmail 73129 invoked by uid 500); 9 Oct 2013 12:46:18 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 73072 invoked by uid 500); 9 Oct 2013 12:46:17 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 73033 invoked by uid 99); 9 Oct 2013 12:46:06 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Oct 2013 12:46:06 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nitinpawar432@gmail.com designates 209.85.216.43 as permitted sender) Received: from [209.85.216.43] (HELO mail-qa0-f43.google.com) (209.85.216.43) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Oct 2013 12:46:00 +0000 Received: by mail-qa0-f43.google.com with SMTP id i13so1446302qae.2 for ; Wed, 09 Oct 2013 05:45:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=PamdN8LbtzGuDjCQrgYEKjY/qwzukuqKvZptFtuOgw8=; b=a1FxAoxwQSzc7vwotzSCqWqJh4p7kGca5iPZ8U+l2Z7f7v2dM+WktKGAoUdEYT4hBR 4nGsABIpjMYmw943HnswWYFPvLxGrGC+A7n40Yate0VQPDezE42faoVZtaVBw5WxDfQ1 MY47qd67/GIjjdrd9tBeuYn0rUea8NUgowtC/hOB0Bb8jAohrFCMV/AKKTKl4REdTc52 n9foxhCuSrx1lLXz7AlkaNTEUsVZzv3gYZzXDzHC1wx5Y3HrKTS537CdznjoNFySmdcj 4ColgrYQ3d8D6qRTlHtOj0Se1JWW2tnLJ3/pm0sy1TUWv5+CHi+4vZpkiWUzfKa98O4l e5xw== MIME-Version: 1.0 X-Received: by 10.224.76.68 with SMTP id b4mr1424583qak.118.1381322739360; Wed, 09 Oct 2013 05:45:39 -0700 (PDT) Received: by 10.224.193.136 with HTTP; Wed, 9 Oct 2013 05:45:39 -0700 (PDT) In-Reply-To: References: <658CDA0F8466BC4FA160446B17B8C7B613A894A4@LTCFISWMSGMB12.FNFIS.com> Date: Wed, 9 Oct 2013 18:15:39 +0530 Message-ID: Subject: Re: Can we reduce/fix number of mappers in Hive From: Nitin Pawar To: "user@hive.apache.org" Content-Type: multipart/alternative; boundary=001a11c308c63328b404e84e4367 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c308c63328b404e84e4367 Content-Type: text/plain; charset=ISO-8859-1 We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1 -- Very old versions . You may want to move to hadoop 1.2.0 and hive-0.11 to have few more things. If you want to limit how many maps a particular job or particular user can occupy, take a look at hadoop's fair scheduler can capacity schedulers. If you want change the number of mapper and reducers for a hive query, you can change the input min and max split sizes as said by Rajesh. Here are few of the settings set mapred.min.split.size=1024000; set mapred.max.split.size=4096000; set hive.exec.reducers.max=2048; may be once you tell clearly what you want achieve we will be able to help better. On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan wrote: > Did you try adjusting fileinputformat. Min and max size parameters? > On Oct 9, 2013 5:51 PM, "Garg, Rinku" wrote: > >> Hi All**** >> >> We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1. We have >> the following query:**** >> >> ** ** >> >> Is there any option in Hive where mappers can be reduced/fixed?**** >> >> ** ** >> >> For example if there are 8 mapping tasks defined in Hadoop configuration, >> then is it possible to use 4 hive setups that uses 2 mappers each?**** >> >> ** ** >> >> Thanks & Regards,**** >> >> Rinku Garg >> >> >> **** >> >> ** ** >> >> ** ** >> >> ** ** >> _____________ >> The information contained in this message is proprietary and/or >> confidential. If you are not the intended recipient, please: (i) delete the >> message and all copies; (ii) do not disclose, distribute or use the message >> in any manner; and (iii) notify the sender immediately. In addition, please >> be aware that any message addressed to our domain is subject to archiving >> and review by persons other than the intended recipient. Thank you. >> > -- Nitin Pawar --001a11c308c63328b404e84e4367 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
We did a successful setup of hadoo= p-0.20.203.0 and hive-0.7.1 -- Very old versions .
You may want to move to hadoop 1.2.0 and hive-0.11 to have f= ew more things.=A0

= If you want to limit how many maps a particular job or particular user can = occupy, take a look at hadoop's fair scheduler can capacity schedulers.= =A0
If you want change the number of mapper and re= ducers for a hive query, you can change the input min and max split sizes a= s said by Rajesh.=A0

= Here are few of the settings
set mapred.min.split.size=3D1024000;
set mapred.max.split.size=3D4096000;
set hive.exec.reducers= .max=3D2048;

may be once you tell clearly what you want achieve we w= ill be able to help better.=A0



On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan &l= t;rajesh.ba= lamohan@gmail.com> wrote:

Did you try adjusting fileinputformat. Mi= n and max size parameters?

On Oct 9, 2013 5:51 PM, "Garg, Rinku" = <Rinku.Gar= g@fisglobal.com> wrote:

Hi All<= u>

We did = a successful setup of hadoop-0.20.203.0 and hive-0.7.1. =A0We have the foll= owing query:

= =A0

Is there any option in Hive where mappers can be redu= ced/fixed?

=A0

For example if there are 8 mapping tasks defined in H= adoop configuration, then is it possible to use 4 hive setups that uses 2 m= appers each?

= =A0

Thanks = & Regards,

Rinku Garg

=A0

=A0

=A0

_____________
The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you.



--
= Nitin Pawar
--001a11c308c63328b404e84e4367--