Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9286718B08 for ; Tue, 13 Oct 2015 19:55:40 +0000 (UTC) Received: (qmail 21431 invoked by uid 500); 13 Oct 2015 19:55:39 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 21271 invoked by uid 500); 13 Oct 2015 19:55:39 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 21184 invoked by uid 99); 13 Oct 2015 19:55:39 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2015 19:55:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id E0D7A1A215A for ; Tue, 13 Oct 2015 19:55:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Ud1jZoi56nZs for ; Tue, 13 Oct 2015 19:55:32 +0000 (UTC) Received: from mail-ob0-f170.google.com (mail-ob0-f170.google.com [209.85.214.170]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id B24A620A92 for ; Tue, 13 Oct 2015 19:55:31 +0000 (UTC) Received: by obbda8 with SMTP id da8so22970969obb.1 for ; Tue, 13 Oct 2015 12:55:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=naHkEGcyGD+KXX1d5cjj2OYECRzHDa6qa7NnD2SGpa4=; b=dUf6lePzBcWOpLefDogMBqjlP3hLHxXrLdGcpNNrFCUwEkIeyHFAZ40rVUvH1uRsjD nrTagu0ig1W+qCt4Bb0MY835ZbFNiIAEQ77+O2Uqfbgskf598gMieHgBborOtooyBK0A oBkQvq5rTpO6J6eYHEpNmYJY+GSttDP1Z9FMlqEVb6AsQDE3ECEYyQ9hW4dglm8T5Tvj S9TNxls608SuSXBfY1wDYcf4Qnd4nhxNtKPYEIZ4gVIoTuD7tyQ7trqbW7yHcU5T/u0D OwbHqFQcGUbmDDUcLvcCciQ2n9TZK6VCMiQipZNDPJyAiPebXT73+CkdsrYHPfhT7CIX ptgg== MIME-Version: 1.0 X-Received: by 10.60.131.198 with SMTP id oo6mr20040038oeb.40.1444766130548; Tue, 13 Oct 2015 12:55:30 -0700 (PDT) Received: by 10.202.75.145 with HTTP; Tue, 13 Oct 2015 12:55:30 -0700 (PDT) In-Reply-To: References: Date: Tue, 13 Oct 2015 14:55:30 -0500 Message-ID: Subject: Re: Hadoop Configuration from DoFn From: Micah Whitacre To: user@crunch.apache.org Content-Type: multipart/alternative; boundary=047d7b471d48feb7b3052201d347 --047d7b471d48feb7b3052201d347 Content-Type: text/plain; charset=UTF-8 Luke, Generally that configuration should be set on the Configuration object passed to Pipeline vs on the individual DoFns. The configure(...) method is called when re-instantiating the DoFn on the Map/Reduce task and at that point those memory settings wouldn't be honored. On Tue, Oct 13, 2015 at 2:52 PM, Luke Hansen wrote: > Does anyone know if this is the right way to configure Hadoop from a > Crunch DoFn? This didn't seem to affect anything. > > Thanks! > > @Override > public void configure(Configuration conf) { > conf.set("mapreduce.map.java.opts", "-Xmx3900m"); > conf.set("mapreduce.reduce.java.opts", "-Xmx3900m"); > > conf.set("mapreduce.map.memory.mb", "4096"); > conf.set("mapreduce.reduce.memory.mb", "4096"); > } > > --047d7b471d48feb7b3052201d347 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Luke,
=C2=A0 Generally that configuration should be se= t on the Configuration object passed to Pipeline vs on the individual DoFns= .=C2=A0 The configure(...) method is called when re-instantiating the DoFn = on the Map/Reduce task and at that point those memory settings wouldn't= be honored.

On Tue, Oct 13, 2015 at 2:52 PM, Luke Hansen <<= a href=3D"mailto:luke@wealthfront.com" target=3D"_blank">luke@wealthfront.c= om> wrote:
Does anyone know if this is the right way= to configure Hadoop from a Crunch DoFn?=C2=A0 This didn't seem to affe= ct anything.

Thanks!
@Override
public void configure(Configuration conf) {
 =
 conf.set("mapredu=
ce.map.java.opts", "-Xmx3900m");
  conf.set("mapreduce.reduce.java.opts"=
, "-Xmx3900m"=
);

  conf.set("mapreduce.map.memory.mb", "4096");
  conf.set("mapreduce.reduce.memory.mb=
", "40=
96");
}

--047d7b471d48feb7b3052201d347--