hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ana Gillan <ana.gil...@gmail.com>
Subject ulimit for Hive
Date Mon, 11 Aug 2014 22:17:18 GMT

I¹ve been reading a lot of posts about needing to set a high ulimit for file
descriptors in Hadoop and I think it¹s probably the cause of a lot of the
errors I¹ve been having when trying to run queries on larger data sets in
Hive. However, I¹m really confused about how and where to set the limit, so
I have a number of questions:
1. How high is it recommended to set the ulimit?
2. What is the difference between soft and hard limits? Which one needs to
be set to the value from question 1?
3. For which user(s) do I set the ulimit? If I am running the Hive query
with my login, do I set my own ulimit to the high value?
4. Do I need to set this limit for these users on all the machines in the
cluster? (we have one master node and 6 slave nodes)
5. Do I need to restart anything after configuring the ulimit?
Thanks in advance,

View raw message