crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ortiz <dor...@videologygroup.com>
Subject RE: Hadoop Configuration from DoFn
Date Tue, 13 Oct 2015 19:54:04 GMT
That is a correct way to do it, but it can also be affected by settings in other DoFns that
may be declared if they run in the same mapper or reducer.

From: Luke Hansen [mailto:luke@wealthfront.com]
Sent: Tuesday, October 13, 2015 3:52 PM
To: user@crunch.apache.org
Subject: Hadoop Configuration from DoFn

Does anyone know if this is the right way to configure Hadoop from a Crunch DoFn?  This didn't
seem to affect anything.

Thanks!

@Override
public void configure(Configuration conf) {
  conf.set("mapreduce.map.java.opts", "-Xmx3900m");
  conf.set("mapreduce.reduce.java.opts", "-Xmx3900m");

  conf.set("mapreduce.map.memory.mb", "4096");
  conf.set("mapreduce.reduce.memory.mb", "4096");
}

This email is intended only for the use of the individual(s) to whom it is addressed. If you
have received this communication in error, please immediately notify the sender and delete
the original email.
Mime
View raw message