crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkwhita...@gmail.com>
Subject Re: Hadoop Configuration from DoFn
Date Tue, 13 Oct 2015 19:55:30 GMT
Luke,
  Generally that configuration should be set on the Configuration object
passed to Pipeline vs on the individual DoFns.  The configure(...) method
is called when re-instantiating the DoFn on the Map/Reduce task and at that
point those memory settings wouldn't be honored.

On Tue, Oct 13, 2015 at 2:52 PM, Luke Hansen <luke@wealthfront.com> wrote:

> Does anyone know if this is the right way to configure Hadoop from a
> Crunch DoFn?  This didn't seem to affect anything.
>
> Thanks!
>
> @Override
> public void configure(Configuration conf) {
>   conf.set("mapreduce.map.java.opts", "-Xmx3900m");
>   conf.set("mapreduce.reduce.java.opts", "-Xmx3900m");
>
>   conf.set("mapreduce.map.memory.mb", "4096");
>   conf.set("mapreduce.reduce.memory.mb", "4096");
> }
>
>

Mime
View raw message