hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guang-Nan Cheng <chen...@gmail.com>
Subject Re: How to optimize the package job jar process?
Date Sat, 09 Jul 2011 13:56:33 GMT
Sorry, wrong question. I found it's not caused by packageJobJar. The
slowness happens while putting those small files to HDFS.

On Tue, Jul 5, 2011 at 2:18 PM, Guang-Nan Cheng <chenggn@gmail.com> wrote:

> I'm passing the whole ruby home to Hadoop, which contains thousands of
> small files.  The packaging process takes few minutes, any tips to speed
> this up?
>
>
>     -files ruby-1.9.2-p180
>     -D
> mapred.child.env=PATH=ruby-1.9.2-p180/bin:'$PATH',GEM_HOME=ruby-1.9.2-p180,LD_LIBRARY_PATH=ruby-1.9.2-p180/lib,GEM_PATH=ruby-1.9.2-p180,RUBYLIB=ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/site_ruby:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/vendor_ruby/1.9.1/x86_64-linux:ruby-1.9.2-p180/lib/ruby/vendor_ruby:ruby-1.9.2-p180/lib/ruby/1.9.1:ruby-1.9.2-p180/lib/ruby/1.9.1/x86_64-linux
> \
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message