Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 07AE6118B2 for ; Thu, 3 Jul 2014 07:56:59 +0000 (UTC) Received: (qmail 3176 invoked by uid 500); 3 Jul 2014 07:56:54 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 3064 invoked by uid 500); 3 Jul 2014 07:56:54 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 3054 invoked by uid 99); 3 Jul 2014 07:56:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 07:56:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dechouxb@gmail.com designates 209.85.215.45 as permitted sender) Received: from [209.85.215.45] (HELO mail-la0-f45.google.com) (209.85.215.45) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 07:56:52 +0000 Received: by mail-la0-f45.google.com with SMTP id hr17so7774072lab.4 for ; Thu, 03 Jul 2014 00:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9WFMDQjX3Td9Ev+M6DLDjiLU4NOhbpIuoUik/N9ZJIc=; b=YoO8XhsuwR+wrmavEYdsSnQAe65BkpytiFvSzADxZz+003koxpaRpXSFddeFPDJHj0 Y5D4n6Ismrz+SwKsvbPtwRpR0suzd0H2Dm3MBAcUaJq6J9NyfLo7Ae0S25z1McBPcBqc eoxkFlNJ2XkfEGOeWL49wGObDQFqEX3EqiQct92A6YLLaSFd7i3sZ3H2xfnwVQLdgIqj dmJfg+9ejxETEOOYBmaADPN/2jgjOfDICFGWan0YikBY8qrrmEV8O7TeA+04q9roKgJJ H8IHjfGDBpPIn8Gvq4cQweUtiGSEpLLp/AQF5g0J49AGddf2Mp+5lCaMhXeIN8YOs3+v le4g== MIME-Version: 1.0 X-Received: by 10.112.202.39 with SMTP id kf7mr2225176lbc.37.1404374187591; Thu, 03 Jul 2014 00:56:27 -0700 (PDT) Received: by 10.112.228.199 with HTTP; Thu, 3 Jul 2014 00:56:27 -0700 (PDT) In-Reply-To: References: Date: Thu, 3 Jul 2014 09:56:27 +0200 Message-ID: Subject: Re: What is the correct way to get a string back from a mapper or reducer From: Bertrand Dechoux To: Chris MacKenzie Cc: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c37746954ec904fd455825 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c37746954ec904fd455825 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The stackoverflow question doesn't add any useful information. Like I said you can emit the string inside a record. Or if you really want to handle lots of complexity, write it yourself within a file or a datastore from the reducer. But you will then have to consider performance issues and be able to handle to lifecycle of the task, its potential multiple attempts and the global lifecyle of the job itself. So it's not necessary obvious, it would depend on the context. The concept of "global variable" in distributed computing should be well understood. By essence, its not possible to have a distributed, always-available, always-consistent variable (see CAP). Bertrand Dechoux On Thu, Jul 3, 2014 at 7:51 AM, Chris MacKenzie < studio@chrismackenziephotography.co.uk> wrote: > Hi Bertrand, > > Thank you for your quick response, I simply need a string returned from > the reducer. > > I have all over the place looking for a solution. I keep coming back to: > http://stackoverflow.com/questions/16222205/use-global-variable-in-reudce= r- > class > > Chris MacKenzie > > > > > > From: Bertrand Dechoux > Reply-To: > Date: Thursday, 3 July 2014 06:43 > To: "user@hadoop.apache.org" > Subject: Re: What is the correct way to get a string back from a mapper > or reducer > > > Configuration is from an architecture point of view immutable once the jo= b > is started even though the API does not reflect that explicitly.I would > say in a record. But the question is : what do you want to achieve? > > Bertrand Dechoux > > > On Thu, Jul 3, 2014 at 7:37 AM, Chris MacKenzie > wrote: > > Hi, > > I have the following code and am using hadoop 2.4: > > In my driver: > Configuration conf =3D new Configuration(); > conf.set("sub", "help"); > =C5=A0.. > String s =3D conf.get("sub=C2=B2); > > In my reducer: > Configuration conf =3D context.getConfiguration(); > conf.set("sub", "Test=C2=B2); > > When I test the value in the driver, it isn=C2=B9t updated following the = reduce > > > Best, > > Chris MacKenzie > > > > > > > > > > --001a11c37746954ec904fd455825 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The stackoverflow question doesn't add any useful in= formation.

Like I said you can emit the string inside a record. Or if you rea= lly want to handle lots of complexity, write it yourself within a file or a= datastore from the reducer. But you will then have to consider performance= issues and be able to handle to lifecycle of the task, its potential multi= ple attempts and the global lifecyle of the job itself. So it's not nec= essary obvious, it would depend on the context.

The concept of "global variable" in distrib= uted computing should be well understood. By essence, its not possible to h= ave a distributed, always-available, always-consistent variable (see CAP).<= /div>

Bertrand Dechoux


On Thu, Jul 3, 2014 at 7:51 AM, Chris Ma= cKenzie <studio@chrismackenziephotography.co.uk&g= t; wrote:
Hi Bertrand,

Thank you for your quick response, I simply need a string returned from
the reducer.

I have all over the place looking for a solution. I keep coming back to: http://stackoverflow.com/questions/16222205/= use-global-variable-in-reudcer-
class

Chris MacKenzie
<http://www.chrismackenziephotography.co.uk/>




From: =C2=A0Bertrand Dechoux <dech= ouxb@gmail.com>
Reply-To: =C2=A0<user@hadoop.a= pache.org>
Date: =C2=A0Thursday, 3 July 2014 06:43
To: =C2=A0"user@hadoop.apach= e.org" <user@hadoop.a= pache.org>
Subject: =C2=A0Re: What is the correct way to get a string back from a mapp= er
or reducer


Configuration is from an architecture point of view immutable once the job<= br> is started even though the API does not reflect that explicitly.I would
say in a record. But the question is : what do you want to achieve?

Bertrand Dechoux


On Thu, Jul 3, 2014 at 7:37 AM, Chris MacKenzie
<studio@chrism= ackenziephotography.co.uk> wrote:

Hi,

I have the following code and am using hadoop 2.4:

In my driver:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Configuration conf =3D new Configuration();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf.set("sub", "help"); =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C5=A0..
=C2=A0 =C2=A0 =C2=A0 =C2=A0 String s =3D conf.get("sub=C2=B2);

In my reducer:
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Configuration conf =3D context.getConfiguration= ();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 conf.set("sub", "Test=C2=B2);
When I test the value in the driver, it isn=C2=B9t updated following the re= duce


Best,

Chris MacKenzie
=C2=A0<http://www.chrismackenziephotography.co.uk/>









--001a11c37746954ec904fd455825--