Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of m.vijayaraghava@gmail.com
 designates 209.85.214.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=F0mr9LXGQK1FUvdyJbjVoHCsYPumdQZNblH5os3aP/4mU9BS0MdAuOIIq4rBmQfgT+
         1xw3h9J3H9MHyYtN7k8ToZea2AN014MsspapvFdRa1ISXVNHPyrKXK+UxNBoPzXiMMDY
         5l+huzTLZpUm71VTvnCPGhK5fyXUDhCyU/SAc=
MIME-Version: 1.0
From: Raghava Mutharaju <m.vijayaraghava@gmail.com>
Date: Tue, 22 Jun 2010 03:49:56 -0400
Message-ID: <AANLkTinG2yVyUvE-YdSV3b8Org_4ITC4Bc28C2dFLkx3@mail.gmail.com>
Subject: multiple reads from a Map - optimization question
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=0016e649c74c0a7c52048999ac42

--0016e649c74c0a7c52048999ac42
Content-Type: text/plain; charset=ISO-8859-1

Hello all,

      In the data, I have to check for multiple conditions and then work
with the data that satisfies all the conditions. I am doing this as an MR
job with no reduce and the conditions are translated to a set of filters.
Among the multiple conditions (2 or 3 max), data that satisfies one of them
would come as input to the Map (initial filter is set in the scan to the
mappers). Now, from among the dataset that comes through to each map, I
would check for other conditions (1 or 2 remaining conditions). Since map()
is called for each row of data, it would mean 1 or 2 read calls (with
filter) to HBase tables. This setup, even for small data (data would fit in
a region and so only 1 map is taking in all the data) is very slow.

Here, note that, I shouldn't be filtering the incoming data to map but based
on that data, next set of filtering conditions would be formed.

Can this be improved? Would constructing secondary indexes help (would need
a dramatic improvement actually)? Or is this type of problem not suitable
for HBase?

Thank you.

Regards,
Raghava.

--0016e649c74c0a7c52048999ac42--