Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1ADC44D4E for ; Sat, 9 Jul 2011 13:57:47 +0000 (UTC) Received: (qmail 70745 invoked by uid 500); 9 Jul 2011 13:57:44 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 70579 invoked by uid 500); 9 Jul 2011 13:57:42 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 70571 invoked by uid 99); 9 Jul 2011 13:57:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jul 2011 13:57:42 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of chenggn@gmail.com designates 74.125.83.176 as permitted sender) Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jul 2011 13:57:34 +0000 Received: by pve37 with SMTP id 37so2555166pve.35 for ; Sat, 09 Jul 2011 06:57:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=qmBi4utNTkmIsjzsM45OnYXy0a6ngTNX5srCf30to84=; b=bQeiDthXMfrJUpDKotR1/7pP4B5Dy2fBOI0OHHdJrSOxx7cIKw8Q/In11LWqL+690q BVYI1tgodBR/J23ABLocNBKgDr6aQSik3Int8tXaAULlQoYxnBcrfdNZrSLPfxFoHlG9 +6teHoxpeVHeFID9d57yRFDlZI6F9oBzdBYH8= Received: by 10.68.13.228 with SMTP id k4mr4574570pbc.40.1310219833083; Sat, 09 Jul 2011 06:57:13 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.47.202 with HTTP; Sat, 9 Jul 2011 06:56:33 -0700 (PDT) In-Reply-To: References: From: Guang-Nan Cheng Date: Sat, 9 Jul 2011 21:56:33 +0800 Message-ID: Subject: Re: How to optimize the package job jar process? To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=bcaec5216315ba870104a7a35301 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec5216315ba870104a7a35301 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sorry, wrong question. I found it's not caused by packageJobJar. The slowness happens while putting those small files to HDFS. On Tue, Jul 5, 2011 at 2:18 PM, Guang-Nan Cheng wrote: > I'm passing the whole ruby home to Hadoop, which contains thousands of > small files. The packaging process takes few minutes, any tips to speed > this up? > > > -files ruby-1.9.2-p180 > -D > mapred.child.env=3DPATH=3Druby-1.9.2-p180/bin:'$PATH',GEM_HOME=3Druby-1.9= .2-p180,LD_LIBRARY_PATH=3Druby-1.9.2-p180/lib,GEM_PATH=3Druby-1.9.2-p180,RU= BYLIB=3Druby-1.9.2-p180/lib/ruby/site_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/s= ite_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/site_ruby:ruby-1.9.2-p= 180/lib/ruby/vendor_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x= 86_64-linux:ruby-1.9.2-p180/lib/ruby/vendor_ruby:ruby-1.9.2-p180/lib/ruby/1= .9.1:ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-linux > \ > > > --bcaec5216315ba870104a7a35301--