Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A7A75DA24 for ; Thu, 15 Nov 2012 15:29:37 +0000 (UTC) Received: (qmail 84379 invoked by uid 500); 15 Nov 2012 15:29:32 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 84283 invoked by uid 500); 15 Nov 2012 15:29:32 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 84254 invoked by uid 99); 15 Nov 2012 15:29:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Nov 2012 15:29:31 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [206.47.135.205] (HELO Spam1.prd.mpac.ca) (206.47.135.205) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Nov 2012 15:29:26 +0000 Received: from Spam1.prd.mpac.ca (unknown [127.0.0.1]) by IMSVA80 (Postfix) with ESMTP id E59711D8045; Thu, 15 Nov 2012 10:29:02 -0500 (EST) Received: from SMAIL1.prd.mpac.ca (unknown [172.29.2.53]) by Spam1.prd.mpac.ca (Postfix) with ESMTP id 93A671D8040; Thu, 15 Nov 2012 10:29:02 -0500 (EST) Received: from SMAIL1.prd.mpac.ca ([fe80::d548:4221:967c:4cfb]) by SMAIL1.prd.mpac.ca ([fe80::18cb:8648:b77f:2b55%11]) with mapi id 14.02.0318.004; Thu, 15 Nov 2012 10:29:02 -0500 From: "Kartashov, Andy" To: "user@hadoop.apache.org" , "cdh-user@cloudera.org" Subject: RE: Oozie apparent concurrency deadlocking Thread-Topic: Oozie apparent concurrency deadlocking Thread-Index: Ac3DP79WHwgoOYTmTJaXQnkJOFBLWgABb8xw Date: Thu, 15 Nov 2012 15:29:02 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.29.60.102] Content-Type: multipart/alternative; boundary="_000_BD42F346AE90F544A731516A805D1B8AD869F1SMAIL1prdmpacca_" MIME-Version: 1.0 X-TM-AS-Product-Ver: IMSVA-8.0.0.1304-6.5.0.1024-19370.000 X-TM-AS-Result: No--32.190-5.0-31-10 X-imss-scan-details: No--32.190-5.0-31-10 X-TM-AS-Result-Xfilter: Match text exemption rules:No X-TMASE-MatchedRID: I31hiQfYWUNJtzxaLxhtWwPZZctd3P4BLoYOuiLW+uXagsZM0qVv18XG pgh6xlCIeNxt5D1jxfVuQ+UlLQf4mVkmrXvRXTvgvnSVh24OCE54wk2zaLy2UYAg7Qouyf/N3H7 nuH0iA07DkuFF6sqe1avPa/p9kAcdo28kkGKrXVqxo9yzdPhMvYyzstdwoG+Pqr3CBdU3C2APAW f/fn5/CY6pqMpCoFQ5z9NLSGa+BaKGbBkDXwNZV7MjW/sniEQKD6C/6wX1l2KxW4YXFsTUDvN2h vGZaoQBI8qZxoWaGPVuxKsZC71H8nz5lEEBuvacaJiWzzHYz3SulZCKpQthQLkiJ/BgvX6rxhH6 LU2YFCWWpScHVmQird+suYAg9ZWJF+VTXvolHkUYUxiUNTPCvUCrr/LkAQ46QU3XqMtEfQ8r3ft oV0XlcwZrwOwmgRBE/s1MQqnJmA5kHLr/RyESlx8GcZrN9c2Oq93da5kXQoDjB19nvhZ7jS2fSf cXA1chYQLJEEG8qEAzMjB5R/FEZp+QdJZ5NPK0wbRQ2BpmlirZTRt/OdIjdh6mOpQCY7tdqGiKz +vA3vXQ23n+P5pIMRQjizXtCXfTWYqLLUX2mAu4jAucHcCqnXcF/0kiqyh4DxjBugJBzzygQwd/ PXlwGPaF4QnP1HwCjgPzQeXnnVqleNL7tmzh/yI9MxSOQ6CSLVEjAx3HjK6k7BPGf466/uygQWd kAVMsmFPzz2KKYMLobRqsEtFzY9J24xzpGcj5BxsweNg3EaGFkCkkB0UMNq9dKZJ2VxiaLnuFYf aWSqYPx/GB/Kxojh+dfU447V7XGRPaEfy9vsDoKx9oW7Unv7/KWCrPQtBHnE+cq/iumoh+3Bndf XUhXQ== X-Virus-Checked: Checked by ClamAV on apache.org --_000_BD42F346AE90F544A731516A805D1B8AD869F1SMAIL1prdmpacca_ Content-Type: text/plain; charset="us-ascii" Guys, Further to my post below and while searching the web , I believe I found hints to my problem solution but have no idea how to implement it: Qte ... - are you using FairScheduler? If so and since you mention that the Sqoop import command is successful, you could be hitting your per user job limit. Whenever Oozie launches a job, it requires two job submissions (if not more) - one being the monitor+launcher, and the subsequent ones being the ones that do the real logic work. The launcher job is something that will launch the remaining jobs, and hence sticks around until they have all ended - taking up one running job slot for the whole lifetime of the Oozie job. For example, with a per user job limit of 3, if you were to run 3 Oozie jobs, the 3 slots would be filled with launchers first. These would submit their real jobs next, and those would end up being in a queue - thereby forming a resource deadlock. The solution is to channel Oozie launcher hadoop jobs into a dedicated launcher pool. This pool can have a running job limit too but won't cause a deadlock because the pools are now separated. To do this, you need to pass the config property: "oozie.launcher." via WF elements or files to point to the separate pool. Unqte And also Qte Harsh J wrote: > >> In a FairScheduler environment, especially where max-running-job >> limits are configured, it is recommended to override the Oozie >> launcher job's pool to be different than the actual required working >> pool (for actions that launch other MR jobs). >> >> If your scheduler is configured to pick ${user.name} up automatically, >> then your Oozie launcher config must use the super-override pool name >> config: >> >> oozie.launcher.mapred.fairscheduler.pool=launcherpoolname >> >> Your target pool for launchers can still carry limitations, but it >> should no longer deadlock your actual MR execution (after which the >> launcher dies away anyway). Unqte Please help. Thanks, Ak-47 From: Kartashov, Andy Sent: Thursday, November 15, 2012 9:45 AM To: user@hadoop.apache.org; 'cdh-user@cloudera.org' Subject: Oozie apparent concurrency deadlocking Guys, Have struggled for the last four days with this and still cannot find an answer even after hours of searching the web. I tried oozie workflow to execute my consecutive sqoop jobs in parallel. I use forking that executes 9 sqoop-action-nodes. I had no problem executing the job on a pseudo-distributed cluster but with an added DN/TT node I ran into (what seems like) deadlocking. Oozie web interface displays those jobs as "Running" indefinitely until I eventually kill the workflow. What I did noticed wasd that if I was to reduce the number of sqoop-action-nodes to 3, all works fine. I found somewhere about oozie.service.CallableQueueService.callable.concurrency property to be set by default to 3 and it hinted me that this must be it them. I tried to over-ride this property by increasing this number to 5 in oozie-site.xml and restart oozie server and then run 4 sqoop-action-nodes in fork but the result is the same. 2 out of 4 nodes execute successfully (not in the same order every time) but the other 2 get hung in indefinite "Running...". There were some suggestion about changing queue name from default but nothing was clear as to what it change it to and where. In case someone found a solution to this please do share. I will greatly appreciate it. Thanks, AK47 NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel --_000_BD42F346AE90F544A731516A805D1B8AD869F1SMAIL1prdmpacca_ Content-Type: text/html; charset="us-ascii"

Guys,

 

Further to my post below and while searching the web , I believe I found hints to my problem solution but have no idea how to implement it:

 

Qte

… - are you using FairScheduler? If so and since you mention that
the Sqoop import command is successful, you could be hitting your per
user job limit.

Whenever Oozie launches a job, it requires two job submissions (if not
more) - one being the monitor+launcher, and the subsequent ones being
the ones that do the real logic work. The launcher job is something
that will launch the remaining jobs, and hence sticks around until
they have all ended - taking up one running job slot for the whole
lifetime of the Oozie job.

For example, with a per user job limit of 3, if you were to run 3
Oozie jobs, the 3 slots would be filled with launchers first. These
would submit their real jobs next, and those would end up being in a
queue - thereby forming a resource deadlock.

The solution is to channel Oozie launcher hadoop jobs into a dedicated
launcher pool. This pool can have a running job limit too but won't
cause a deadlock because the pools are now separated.

To do this, you need to pass the config property:
"oozie.launcher.<property that specifies your pool>" via WF
<configuration> elements or <job-xml> files to point to the separate
pool.

Unqte

 

And also

 

Qte

Harsh J <harsh@cloudera.com> wrote:

> 

>> In a FairScheduler environment, especially where max-running-job

>> limits are configured, it is recommended to override the Oozie

>> launcher job's pool to be different than the actual required working

>> pool (for actions that launch other MR jobs).

>> 

>> If your scheduler is configured to pick ${user.name} up automatically,

>> then your Oozie launcher config must use the super-override pool name

>> config:

>> 

>> oozie.launcher.mapred.fairscheduler.pool=launcherpoolname

>> 

>> Your target pool for launchers can still carry limitations, but it

>> should no longer deadlock your actual MR execution (after which the

>> launcher dies away anyway).

 

Unqte

 

Please help.

 

Thanks,

Ak-47

 

 

From: Kartashov, Andy
Sent: Thursday, November 15, 2012 9:45 AM
To: user@hadoop.apache.org; 'cdh-user@cloudera.org'
Subject: Oozie apparent concurrency deadlocking

 

Guys,

 

Have struggled for the last four days with this and still cannot find an answer even after hours of searching the web.

 

I tried oozie workflow to execute my consecutive sqoop jobs in parallel.  I use forking that executes 9 sqoop-action-nodes.

 

I had no problem executing the job on a pseudo-distributed cluster but with an added DN/TT node I ran into (what seems like) deadlocking.  Oozie web interface displays those jobs as “Running” indefinitely until I eventually kill the workflow.

 

What I did noticed wasd that if I was to reduce the number of sqoop-action-nodes to 3, all works fine.

 

I found somewhere about oozie.service.CallableQueueService.callable.concurrency property to be set by default to 3 and it hinted me that this must be it them. I tried to over-ride this property by increasing this number to 5 in oozie-site.xml and restart oozie server and then run 4 sqoop-action-nodes in fork but the result is the same. 2 out of 4 nodes execute successfully (not in the same order every time) but the other 2 get hung in indefinite “Running…”.    

 

There were some suggestion about changing queue name from default but nothing was clear as to what it change it to and where.

 

In case someone found a solution to this please do share. I will greatly appreciate it.

 

Thanks,

AK47

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel --_000_BD42F346AE90F544A731516A805D1B8AD869F1SMAIL1prdmpacca_--