Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com
 designates 209.85.160.50 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DXx70hBN9fMZMZdgYadHWW+Vf6aCVMtna0WfvmtWogGkNdJ/F1FNj4/S1jZqrsT21J
         g1y3suxuHXKMYUaYt/vwlMmdQrt+L1lXSbHOZOrG+Cj/laVJ1QzJvUIMXdzC4Wyxf52H
         UqSmcHvCxpMkqRF9gIWglUkVg1bwhQgfBxJxg=
MIME-Version: 1.0
In-Reply-To: <651686.3536.qm@web19001.mail.hk2.yahoo.com>
References: <651686.3536.qm@web19001.mail.hk2.yahoo.com>
Date: Wed, 23 Dec 2009 06:55:53 -0800
Message-ID: <314098690912230655j4d26433aj6a3f23cbcb8a9e19@mail.gmail.com>
Subject: Re: sharing variables across chained jobs
From: Jason Venner <jason.hadoop@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636920849e4e9fa047b668419

--001636920849e4e9fa047b668419
Content-Type: text/plain; charset=ISO-8859-1

If your jobs are launched by separate jvm instances, the only real
persistence framework you have is hdfs.

You have to basic choices:

   1. Write a summary data to a persistent store, an hdfs file being a
   simple case, that your next job reads
   2. Write the data you need as a job counter, via the Reporter object, and
   have the next job read the counters from the previous job via the
   JobClient.getJob(jobid) interface.


Case 2 requires that the counters still exist, they are usually discarded
with in 24 hours, and that you can determine the jobid of the job you need
to interrogate.

On Tue, Dec 22, 2009 at 11:51 PM, Himanshu <ll_oz_ll@yahoo.com.hk> wrote:

> Hi everyone,
> I run multiple map/reduce jobs which are
> chained together. The output of one map/reduce is the input of another.
> There are also some integer valued variables which are outputted from
> one map/reduce job and used as input in the consequent one. These
> variables are got by summing up integer valued data having a certain
> key in the reduce step.
>
> My question was - what would be the
> best way to share these several integer valued variables across
> multiply map/reduce iterations? I don't want to write each of them to a
> separate file in the reduce step and consequently read those files in
> the next iteration of map/reduce.
>
> Looking forward to all the suggestions
>
> H
>
> Send instant messages to your online friends http://uk.messenger.yahoo.com
>


-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--001636920849e4e9fa047b668419--