Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of viral.bajaria@gmail.com
 designates 209.85.128.47 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <0FCBF5E7-49BD-46A8-9B49-24549312EC2E@yahoo.com>
References: <-5717258686639839845@unknownmsgid>
	<0FCBF5E7-49BD-46A8-9B49-24549312EC2E@yahoo.com>
Date: Thu, 14 Feb 2013 12:48:30 -0800
Message-ID: 
 <CALckxSMmpQ6mL7ETPkRFsU28Sg7OoRQ27T3KjWHW3wdMre1Msw@mail.gmail.com>
Subject: Re: Using HBase for Deduping
From: Viral Bajaria <viral.bajaria@gmail.com>
To: Rahul Ravindran <rahulrv@yahoo.com>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=20cf300fb4b3a2a8cf04d5b5616a

--20cf300fb4b3a2a8cf04d5b5616a
Content-Type: text/plain; charset=ISO-8859-1

You could do with a 2-pronged approach here i.e. some MR and some HBase
lookups. I don't think this is the best solution either given the # of
events you will get.

FWIW, the solution below again relies on the assumption that if a event is
duped in the same hour it won't have a dupe outside of that hour boundary.
If it can have then you are better of with running a MR job with the
current hour + another 3 hours of data or an MR job with the current hour +
the HBase table as input to the job too (i.e. no HBase lookups, just read
the HFile directly) ?

- Run a MR job which de-dupes events for the current hour i.e. only runs on
1 hour worth of data.
- Mark records which you were not able to de-dupe in the current run
- For the records that you were not able to de-dupe, check against HBase
whether you saw that event in the past. If you did, you can drop the
current event or update the event to the new value (based on your business
logic)
- Save all the de-duped events (via HBase bulk upload)

Sorry if I just rambled along, but without knowing the whole problem it's
very tough to come up with a probable solution. So correct my assumptions
and we could drill down more.

Thanks,
Viral

On Thu, Feb 14, 2013 at 12:29 PM, Rahul Ravindran <rahulrv@yahoo.com> wrote:

> Most will be in the same hour. Some will be across 3-6 hours.
>
> Sent from my phone.Excuse the terseness.
>
> On Feb 14, 2013, at 12:19 PM, Viral Bajaria <viral.bajaria@gmail.com>
> wrote:
>
> > Are all these dupe events expected to be within the same hour or they
> > can happen over multiple hours ?
> >
> > Viral
> > From: Rahul Ravindran
> > Sent: 2/14/2013 11:41 AM
> > To: user@hbase.apache.org
> > Subject: Using HBase for Deduping
> > Hi,
> >    We have events which are delivered into our HDFS cluster which may
> > be duplicated. Each event has a UUID and we were hoping to leverage
> > HBase to dedupe them. We run a MapReduce job which would perform a
> > lookup for each UUID on HBase and then emit the event only if the UUID
> > was absent and would also insert into the HBase table(This is
> > simplistic, I am missing out details to make this more resilient to
> > failures). My concern is that doing a Read+Write for every event in MR
> > would be slow (We expect around 1 Billion events every hour). Does
> > anyone use Hbase for a similar use case or is there a different
> > approach to achieving the same end result. Any information, comments
> > would be great.
> >
> > Thanks,
> > ~Rahul.
>

--20cf300fb4b3a2a8cf04d5b5616a--