Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jlist@streamy.com designates
 72.34.249.3 as permitted sender)
Message-ID: <4A64C14D.7040705@streamy.com>
Date: Mon, 20 Jul 2009 12:11:09 -0700
From: Jonathan Gray <jlist@streamy.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: hbase-user@hadoop.apache.org
Subject: Re: A question about MapReduce job extracts recent data from
 	HBase/Bigtable
References: <fa03480d0907201108l59877c11la7df0dc3b727ba76@mail.gmail.com>
In-Reply-To: <fa03480d0907201108l59877c11la7df0dc3b727ba76@mail.gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit

The row key is (website,stamp) so the table is GROUP BY website and then 
ORDER BY stamp.  If you'd want to just get recent data, you'd do some 
kind of row filter server-side so you were only returned clicks from the 
range you specified for that particular MR job.

Does that make sense?  Do you think it's more complex than that?  They 
are grouping by the website, not strictly ordering by the stamp, so 
there's no way to prevent a full table scan (server-side), you can use 
filters to prevent all the unnecessary data from moving back to the 
client/job.

JG

Schubert Zhang wrote:
> Hi all,
> 
> I have a periodically scheduled MapReduce job need to extract recent data
> from a HBase table for analysis, and avoid scanning/reading the analyzed
> data. Do you have any idea?
> 
> In the Google paper <Bigtable: A Distributed Storage System for Structured
> Data>
> Section: 8.1 Google Analytics
> 
> The raw click table (�200 TB) maintains a row for each end-user session. The
> row name is a tuple containing the website's name and the time at which the
> session was created. This schema ensures that sessions that visit the same
> web site are contiguous, and that they are sorted chronologically. This
> table compresses to 14% of its original size.
> 
> The summary table (~20 TB) contains various predefined summaries for each
> website. This table is generated from the raw click table by periodically
> scheduled MapReduce jobs. Each MapReduce job extracts recent session data
> from the raw click table. The overall system's throughput is limited by the
> throughput of GFS. This table compresses to 29% of its original size.
> 
> Can anybody share your ideas about how "Each MapReduce job extracts recent
> session data from the raw click table."?
> 
> Thanks!
> Schubert
>