hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject run arbitrary job (non-MR) on YARN ?
Date Mon, 27 Oct 2014 06:19:45 GMT
I happened to run into this interesting scenario:

I had some mahout seq2sparse jobs, originally i run them in parallel using
the distributed mode. but because the input files are so small, running
them locally actually is much faster. so I truned them to local mode.

but I run 10 of these jobs in parallel, so when 10 mahout jobs are run
together, everyone became very slow.

is there an existing code that takes a desired shell script, and possibly
some archive files (could contain the jar file, or C++ --generated
executable code). I understand that I could use yarn API to code such a
thing, but it would be nice if I could just take it and run in shell..


View raw message