hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Region Server shutdown (Replay HLOg required)
Date Sun, 16 Jan 2011 01:36:22 GMT
On Fri, Jan 14, 2011 at 11:36 AM, charan kumar <charan.kumar@gmail.com> wrote:
> Could this be an effect of hotspotting? Since I am persisting millions of
> keys through MR (which are URLS) and they are already sorted.
>

I don't think so.  HBase usually buckles first.  For sure you've upped
ulimit and xceivers as per HBase requirements?  Confirm HBase is
seeing your upped ulimit setting.  Its first thing logged when HBase
starts up.  See these two sections:
http://people.apache.org/~stack/hbase-0.90.0-candidate-3/docs/notsoquick.html#ulimit


> Should I consider some normalization on the Key (hashing, prefix, reverse
> the url chars) ? I
>

If your inserts are pre-sorted, then your load is probably not well
distributed over the cluster (you can check your Master UI -- see
where the requests are going... I'd guess all are being fielded by one
server, then another, and so on rather than all taking even load).
Are your keys sorted because they came out of a MR reduce?  If so,
could you back up and in your Map insert into HBase?  Then the keys
would not be sorted and load better likely better distributed (There
is a little on this topic in this section on HBase+MapReduce if you've
not seen it already:
http://people.apache.org/~stack/hbase-0.90.0-candidate-3/docs/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#package_description).

St.Ack


St.Ack

Mime
View raw message