Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DC0C0D20A for ; Fri, 14 Dec 2012 04:39:53 +0000 (UTC) Received: (qmail 71749 invoked by uid 500); 14 Dec 2012 04:39:49 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 70741 invoked by uid 500); 14 Dec 2012 04:39:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 70706 invoked by uid 99); 14 Dec 2012 04:39:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Dec 2012 04:39:46 +0000 X-ASF-Spam-Status: No, hits=0.2 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,RCVD_IN_DNSWL_NONE,SPF_PASS,UNPARSEABLE_RELAY X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [98.139.213.151] (HELO nm7-vm0.bullet.mail.bf1.yahoo.com) (98.139.213.151) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Dec 2012 04:39:37 +0000 Received: from [98.139.212.150] by nm7.bullet.mail.bf1.yahoo.com with NNFMP; 14 Dec 2012 04:39:16 -0000 Received: from [98.139.213.2] by tm7.bullet.mail.bf1.yahoo.com with NNFMP; 14 Dec 2012 04:39:16 -0000 Received: from [127.0.0.1] by smtp102.mail.bf1.yahoo.com with NNFMP; 14 Dec 2012 04:39:16 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1355459956; bh=hHJTOH5YmP19N5FMePA+Y0Km70N0fmShMsfRYYHlkao=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Mailer:Thread-Index:Content-Language; b=qj2Hraf1ku6RSWrstN2795daEenxYTOIeqhrnqhMUv5cCOQH5wTeMhlu2DyDqNeMvCIi4PTag8P6GwrVyGyjCsamGLSVc4nt3bTcfLuoRP45HTkp5nILznzbalUh9hTwR5ROxVr/sxpS139+2meA82cgkGFfS5E7rMC6NZ/qiJw= X-Yahoo-Newman-Id: 577159.87029.bm@smtp102.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: rrX0LZQVM1m.sAw8y1Jn5EgRp0ETxDOAiEIP0ooYsMaQ6T. imyRhB20vJuVFuEE1B3DErWgfOyiMWVdfTNUaqJEOpXgENTeNOeYaH1fNb7q CPUJtkCNKCuzhuMUQEIWQmEQ0IKHgOcoAIrrY_qQKomZqY3Y__dBlNUQb_87 XgBGuzgYuNibF9AWkII44FfA8rU3BRmoDxuGpFPvvQU7vXVp26Og6c5M3nlZ HZaBpe0xz.ye60HzcZxKILQnGXZbyBcXNn.uJcKwZLeiTFeLzESdYElOPvqE 12llxh6GwZBcSv9Jb1gphtgHIHaVhtcsU3Yhl0yD4NPd0wnGzUwnTimNK.9L DCbPeEfTDMLo0jUWgp4Lg7x5t556wr1e8ybSIaUsNKOz6vCw4hEmSuhkCv1B oRIwCpGJ4sUxJQv_.9YyvZVSkIyd5rJSULJTs4AghjRslf52LN7pGJOf5mFX pUjWlyeGCXL0uWCkH X-Yahoo-SMTP: k2gD1GeswBAV_JFpZm8dmpTCwr4ufTKOyA-- Received: from sattelite (davidparks21@113.161.75.108 with login) by smtp102.mail.bf1.yahoo.com with SMTP; 13 Dec 2012 20:39:16 -0800 PST From: "David Parks" To: Subject: How to submit Tool jobs programatically in parallel? Date: Fri, 14 Dec 2012 11:39:05 +0700 Message-ID: <085401cdd9b4$f48a7df0$dd9f79d0$@yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac3ZtPKWZuYuNHdeTIyT/n7HUQmsBg== Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org I'm submitting unrelated jobs programmatically (using AWS EMR) so they run in parallel. I'd like to run an s3distcp job in parallel as well, but the interface to that job is a Tool, e.g. ToolRunner.run(...). ToolRunner blocks until the job completes though, so presumably I'd need to create a thread pool to run these jobs in parallel. But creating multiple threads to submit concurrent jobs via ToolRunner, blocking on the jobs completion, just feels improper. Is there an alternative?