hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Garg, Rinku" <Rinku.G...@fisglobal.com>
Subject RE: Can we reduce/fix number of mappers in Hive
Date Thu, 10 Oct 2013 06:49:08 GMT
Hi Nitin/Rajesh,

Thanks for the reply...

I tried with both the ways as you guys mentioned in the mail trail... but using that way hive
does not fix the number of mappers for a hive query.

My orginal question is :

if there are 8 mapping tasks defined in Hadoop configuration, then is it possible to use 4
hive setups that uses 2 mappers each?

Please reply.

Thanks
Rinku Garg
________________________________
From: Nitin Pawar [nitinpawar432@gmail.com]
Sent: Wednesday, October 09, 2013 6:15 PM
To: user@hive.apache.org
Subject: Re: Can we reduce/fix number of mappers in Hive

We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1 -- Very old versions .
You may want to move to hadoop 1.2.0 and hive-0.11 to have few more things.

If you want to limit how many maps a particular job or particular user can occupy, take a
look at hadoop's fair scheduler can capacity schedulers.
If you want change the number of mapper and reducers for a hive query, you can change the
input min and max split sizes as said by Rajesh.

Here are few of the settings
set mapred.min.split.size=1024000;
set mapred.max.split.size=4096000;
set hive.exec.reducers.max=2048;

may be once you tell clearly what you want achieve we will be able to help better.



On Wed, Oct 9, 2013 at 6:09 PM, Rajesh Balamohan <rajesh.balamohan@gmail.com<mailto:rajesh.balamohan@gmail.com>>
wrote:

Did you try adjusting fileinputformat. Min and max size parameters?

On Oct 9, 2013 5:51 PM, "Garg, Rinku" <Rinku.Garg@fisglobal.com<mailto:Rinku.Garg@fisglobal.com>>
wrote:
Hi All
We did a successful setup of hadoop-0.20.203.0 and hive-0.7.1.  We have the following query:

Is there any option in Hive where mappers can be reduced/fixed?

For example if there are 8 mapping tasks defined in Hadoop configuration, then is it possible
to use 4 hive setups that uses 2 mappers each?

Thanks & Regards,
Rinku Garg





_____________
The information contained in this message is proprietary and/or confidential. If you are not
the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review
by persons other than the intended recipient. Thank you.



--
Nitin Pawar

_____________
The information contained in this message is proprietary and/or confidential. If you are not
the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review
by persons other than the intended recipient. Thank you.

Mime
View raw message