hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kiran Kumar Reddy Govind <kgov...@ega.ae>
Subject RE: Problems in running spark on Yarn
Date Wed, 02 Sep 2015 03:33:15 GMT

How does Hive work with MAPREDUCE , can someone clarify unable to understand this concept.


From: Matteo Luzzi [mailto:matteo.luzzi@gmail.com]
Sent: Wednesday, September 02, 2015 2:26 AM
To: user@hadoop.apache.org
Subject: Problems in running spark on Yarn

Hi all!
I'm developing a system where I need to run spark jobs over yarn. I'm using a two node cluster
(one master and one slave) for testing and I'm submitting the application through oozie, but
after the first application starts running (the oozie container) the other one remains in
accepted stated. I am new to yarn so probably I am missing some concepts about how containers
are requested and assigned to the applications. It seems that I can execute only one container
at the time, even though there are still free resources. When I kill the first running application,
the other one passes to running state. I'm also using the Fair Scheduler as according the
documentation, it should avoid any starvation problems.
I don't know if it is a problem of spark or yarn. Please come with suggestion if you have

[Immagine incorporata 1]

Kiran Govind
Senior Superintendent - Maint. Planning, Power & Desalination Maintenance
UAE Operations
E  kgovind@ega.ae
D +97148021153
M +971555430816

Emirates Global Aluminium
P.O. Box 3627, Dubai
United Arab Emirates



This is an e-mail from Emirates Global Aluminium PJSC. Its contents are confidential to the
intended recipient. If you are not the intended recipient be advised that you have received
this email in error and that any use, dissemination, forwarding, printing or copying of this
e-mail is strictly prohibited. It may not be disclosed to or used by anyone other than its
intended recipient, nor may it be copied in any way. If received in error please e-mail a
reply to the sender and delete it from your system. Although this e-mail has been scanned
for viruses, Emirates Global Aluminium cannot ultimately accept any responsibility for viruses
and it is your responsibility to scan attachments (if any).
View raw message