hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris MacKenzie <stu...@chrismackenziephotography.co.uk>
Subject Re: ulimit for Hive
Date Tue, 12 Aug 2014 18:24:12 GMT
Hi Zhijie,

ulimit is common between hard and soft ulimit

The hard limit can only be set by a sys admin. It can be used for a fork
bomb dos attack.
The sys admin hard ulimit can be set per user i.e hadoop_user

A user can add a line to their .profile file setting a soft -ulimit up to
the hard limit. You can google how to do that

You can check the ulimits like so:

ulimit -H -a // hard limit
ulimit -S -a // soft limit

The max value for the hard limit is -unlimited. I currently have mine set
to this as I was running out of processes (nproc)

I don¹t know about restarting, I think so.
I don¹t know about hive.

Warm regards.


telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk

From:  Zhijie Shen <zshen@hortonworks.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Tuesday, 12 August 2014 18:33
To:  <user@hadoop.apache.org>, <user@hive.apache.org>
Subject:  Re: ulimit for Hive

+ Hive user mailing list
It should be a better place for your questions.

On Mon, Aug 11, 2014 at 3:17 PM, Ana Gillan <ana.gillan@gmail.com> wrote:


I¹ve been reading a lot of posts about needing to set a high ulimit for
file descriptors in Hadoop and I think it¹s probably the cause of a lot of
the errors I¹ve been having when trying to run queries on larger data sets
in Hive. However, I¹m really confused about how and where to set the
limit, so I have a number of questions:

1. How high is it recommended to set the ulimit?
2. What is the difference between soft and hard limits? Which one needs to
be set to the value from question 1?
3. For which user(s) do I set the ulimit? If I am running the Hive query
with my login, do I set my own ulimit to the high value?
4. Do I need to set this limit for these users on all the machines in the
cluster? (we have one master node and 6 slave nodes)
5. Do I need to restart anything after configuring the ulimit?

Thanks in advance,

Zhijie ShenHortonworks Inc.

NOTICE: This message is intended for the use of the individual or entity
to which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately and delete it from your system. Thank You.

View raw message