hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tim robertson <timrobertson...@gmail.com>
Subject Re: hadoop jobs take long time to setup
Date Sun, 28 Jun 2009 19:54:15 GMT
How long does it take to start the code locally in a single thread?

Can you reuse the JVM so it only starts once per node per job?
conf.setNumTasksToExecutePerJvm(-1)

Cheers,
Tim



On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<marcus.herou@tailsweep.com> wrote:
> Hi.
>
> Wonder how one should improve the startup times of a hadoop job. Some of my
> jobs which have a lot of dependencies in terms of many jar files take a long
> time to start in hadoop up to 2 minutes some times.
> The data input amounts in these cases are neglible so it seems that Hadoop
> have a really high setup cost, which I can live with but this seems to much.
>
> Let's say a job takes 10 minutes to complete then it is bad if it takes 2
> mins to set it up... 20-30 sec max would be a lot more reasonable.
>
> Hints ?
>
> //Marcus
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
>

Mime
View raw message