hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Birender Saini <bsa...@hortonworks.com>
Subject Re: One job blocks all queues.
Date Thu, 28 May 2015 16:00:48 GMT
Julien -

Sounds like you are using default Capacity Scheduler settings which has  minimum-user-limit-percent
= 100, meaning the minimum guaranteed resources for a single user is 100%

Read more about this property here - http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_system-admin-guide/content/setting_user_limits.html

If you want to read more about Capacity Scheduler and key properties that can help you fine
tune multi tenancy, see this -
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.4/bk_yarn_resource_mgt/content/ref-25c07006-4490-4e57-b04c-7582c6ee16b8.1.html


Heres another article explaining how to tune Hive for Interactive and Batch Queries -
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.4/bk_performance_tuning/content/ch_performance_interactive_queue_chapter.html


Biren Saini  Solutions Engineer, Hortonworks
Mobile: 704-526-8148
Email: bsaini@hortonworks.com<mailto:masha@hortonworks.com>
Website: http://www.hortonworks.com/



[cid:1744F2FE-770C-4EF7-ADE9-446FFD932CFB]

On May 28, 2015, at 11:28 AM, Julien Carme <julien.carme@gmail.com<mailto:julien.carme@gmail.com>>
wrote:

Hello,

I am experimenting the use of multi-tenancy in Hadoop.

I have a Hive queries which does never give a result and whose containers seem to freeze forever.
It is basically a join where all key values of both input tables are the same.

I understand there can be bugs in Hive and they will be corrected at some point, and twisted
queries like this one might crash Hive.

However, one this query is submitted, all the cluster is frozen including other queues. The
entire cluster is useless until you have manually killed the faulty application. If you want
to use a single hadoop cluster for several customers, this is a major issue.

Is it the expected behavior? Once Yarn has assigned all its containers, is the only thing
it can do is wait until they have finished there job? What could be a solution to this problem?

Thanks for your answers.


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message