hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Cluster wide atomic operations
Date Mon, 29 Oct 2012 05:58:22 GMT
On Sun, Oct 28, 2012 at 9:15 PM, David Parks <davidparks21@yahoo.com> wrote:

> I need a unique & permanent ID assigned to new item encountered, which has
> a constraint that it is in the range of, let’s say for simple discussion,
> one to one million.
>

Having such a limited range may require that you have a central service to
generate ID's.  The use of a central service can be disastrous for
throughput.


> ****
>
> ** I suppose I could assign a range of usable IDs to each reduce task
> (where ID’s are assigned) and keep those organized somehow at the end of
> the job, but this seems clunky too.
>
> **
>

Yes.  Much better.


>  Since this is on AWS, zookeeper is not a good option. I thought it was
> part of the hadoop cluster (and thus easy to access), but guess I was wrong
> there.
>

No.  This is specifically not part of Hadoop for performance reasons.


> ** I would think that such a service would run most logically on the
> taskmaster server. I’m surprised this isn’t a common issue. I guess I could
> launch a separate job that runs such a sequence service perhaps. But that’s
> non trivial its self with failure concerns.
>

The problem is that a serial number service is a major loss of performance
in a parallel system.  Unless you relax the idea considerably (by allowing
blocks, or having lots of bits like Snowflake), then you wind up with a
round-trip per id and you have a critical section on the ID generator.
 This is bad.

Look up Amdahl's Law.


> ** Perhaps there’s just a better way of thinking of this?
>

Yes.  Use lots of bits and be satisfied with uniqueness rather than perfect
ordering and limited range.

As the other respondent said, look up Snowflake.

Mime
View raw message