Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of markkerzner@gmail.com
 designates 209.85.214.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=I0rmUH2sTeFnIxHPQnLbJFISye72B+Nh8KHbrzm9/oWEU6v8shYQDissko0ESTqW8R
         B0pbbTJdEFMVxR16xZWke3daTxYQ+4mg2p+QObi6r9xYE3kiI8kY68dFzfPmC72/sn85
         5UNQEnrQ6jg4STwEULxy3iKeroQbTlPLvVKt4=
MIME-Version: 1.0
In-Reply-To: <BANLkTi=a9+1x4ouJu2wLDtFYk-zo0Mb1FA@mail.gmail.com>
References: <BANLkTikzRhEKhcYRo7xYf6gRo1YhggGi+w@mail.gmail.com>
	<BANLkTi=a9+1x4ouJu2wLDtFYk-zo0Mb1FA@mail.gmail.com>
Date: Fri, 20 May 2011 12:17:26 -0500
Message-ID: <BANLkTi=sg9eQ_QT7CXSgCYMdwAFzayKKPw@mail.gmail.com>
Subject: Re: Can I number output results with a Counter?
From: Mark Kerzner <markkerzner@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325554fe2b2173c04a3b84be2

--000325554fe2b2173c04a3b84be2
Content-Type: text/plain; charset=ISO-8859-1

Joey,

You understood me perfectly well. I see your first advice, but I am not
allowed to have gaps. A central service is something I may consider if
single reducer becomes a worse bottleneck than it.

But what are counters for? They seem to be exactly that.

Mark

On Fri, May 20, 2011 at 12:01 PM, Joey Echeverria <joey@cloudera.com> wrote:

> To make sure I understand you correctly, you need a globally unique
> one up counter for each output record?
>
> If you had an upper bound on the number of records a single reducer
> could output and you can afford to have gaps, you could just use the
> task id and multiply that by the max number of records and then one up
> from there.
>
> If that doesn't work for you, then you'll need to use some kind of
> central service for allocating numbers which could become a
> bottleneck.
>
> -Joey
>
> On Fri, May 20, 2011 at 9:55 AM, Mark Kerzner <markkerzner@gmail.com>
> wrote:
> > Hi, can I use a Counter to give each record in all reducers a consecutive
> > number? Currently I am using a single Reducer, but it is an anti-pattern.
> > But I need to assign consecutive numbers to all output records in all
> > reducers, and it does not matter how, as long as each gets its own
> number.
> >
> > If it IS possible, then how are multiple processes accessing those
> counters
> > without creating race conditions.
> >
> > Thank you,
> >
> > Mark
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>

--000325554fe2b2173c04a3b84be2--