hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Chacko <peterchack...@gmail.com>
Subject Re: doubt regarding multiprocessing in hadoop
Date Wed, 20 May 2009 14:50:25 GMT
If your process is not inherently designed for concurrency with
multiple logical components as different threads of execution,
dispatchable to multiple cores in parallel, you cannot make it
mufti-core aware by splitting it as different threads. (if these
threads are sequential by nature, you get the same compute  latency )

If  your process is computation intensive, you can try using intel
CMP-compilers, or  if it is IO intensive, there is nothing much you
can do unless its a NUMA system or (any asymmetric cluster, where you
can leverage node-locality benefits).

I am just curious to know what process is this, that takes 1-2 weeks
to execute, in   CPUs. currently available in the market..


On Wed, May 20, 2009 at 4:52 PM, sree deepya <mdeepya@gmail.com> wrote:
> Hi,
>   I have a small doubt regarding possibility of multiprocessing  in
> hadoop.Is it possible ????
>  To be more clear,I have some processes that take around a week or two to
> execute.Is it possible to use hadoop to
>  reduce this execution time by dividing  these processes into a number of
> processes and executing simultaneously on
>  different nodes???Or is there any other way in which my objective can be
> achieved using hadoop????
>   If yes, Can you please throw some light on this issue????
> Thanks in advance,
> regards,
> SreeDeepya.

View raw message