hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhijie Shen <zs...@hortonworks.com>
Subject Re: ulimit for Hive
Date Tue, 12 Aug 2014 17:33:53 GMT
+ Hive user mailing list

It should be a better place for your questions.

On Mon, Aug 11, 2014 at 3:17 PM, Ana Gillan <ana.gillan@gmail.com> wrote:

> Hi,
> I’ve been reading a lot of posts about needing to set a high ulimit for
> file descriptors in Hadoop and I think it’s probably the cause of a lot of
> the errors I’ve been having when trying to run queries on larger data sets
> in Hive. However, I’m really confused about how and where to set the limit,
> so I have a number of questions:
>    1. How high is it recommended to set the ulimit?
>    2. What is the difference between soft and hard limits? Which one
>    needs to be set to the value from question 1?
>    3. For which user(s) do I set the ulimit? If I am running the Hive
>    query with my login, do I set my own ulimit to the high value?
>    4. Do I need to set this limit for these users on all the machines in
>    the cluster? (we have one master node and 6 slave nodes)
>    5. Do I need to restart anything after configuring the ulimit?
> Thanks in advance,
> Ana

Zhijie Shen
Hortonworks Inc.

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message