Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of gcjhhu-hbase-user@m.gmane.org
 designates 80.91.229.12 as permitted sender)
To: user@hbase.apache.org
From: Luke Forehand <luke.forehand@networkedinsights.com>
Subject: Re: Secondary Index versus Full Table Scan
Date: Tue, 3 Aug 2010 16:36:50 +0000 (UTC)
Lines: 28
Message-ID: <loom.20100803T181808-232@post.gmane.org>
References: <loom.20100803T163448-875@post.gmane.org>
 <AANLkTinRWeJz9B0y9OX8Th-DwwY8Z86uU=xehNqt2u=m@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
User-Agent: Loom/3.14 (http://gmane.org/)

Edward Capriolo <edlinuxguru@...> writes:

> Generally speaking: If you are doing full range scans of a table
> indexes will not help. Adding indexes will make the performance worse,
> it will take longer to load your data and now fetching the data will
> involve two lookups instead of one.
> 
> If you are doing full range scans adding more nodes should result in
> linear scale up.
> 
> 

Edward,

Can you clarify what "full range scan" means?  I am not doing "full" range
scans, but I am doing relatively large range scans (3 million records), so I
think what you are saying applies.  Thanks for the insight.  

We initially implemented the secondary index out of a need to have our main data
sorted by multiple dimensions for various use cases.  Now I'm thinking it may be
better to have multiple copies of our main data, sorted in multiple ways, to
avoid the two lookups.  So I'm faced with two options right now; multiple copies
of the data sorted in multiple ways to do range scans, or buy a lot more servers
and do full scans.  Given these two choices, do people have general
recommendations on which makes the most sense?

Thanks!
-Luke