crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luke Hansen <l...@wealthfront.com>
Subject Hadoop Configuration from DoFn
Date Tue, 13 Oct 2015 19:52:22 GMT
Does anyone know if this is the right way to configure Hadoop from a Crunch
DoFn?  This didn't seem to affect anything.

Thanks!

@Override
public void configure(Configuration conf) {
  conf.set("mapreduce.map.java.opts", "-Xmx3900m");
  conf.set("mapreduce.reduce.java.opts", "-Xmx3900m");

  conf.set("mapreduce.map.memory.mb", "4096");
  conf.set("mapreduce.reduce.memory.mb", "4096");
}

Mime
View raw message