hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: gauging cost of region movement
Date Mon, 21 Mar 2011 23:32:31 GMT
it would make sense to avoid moving regions, so therefore the more
recently a region was moved, the less likely we should move it.

you could imagine a hypothetical perfect 'region move cost' function
that might look like:

F(r) = timeSinceMoved(r) + size(r) + loadAvg(r)

The functions should probably be normalized to [0,1], so the range of
F would be [0,3] with 3 == 'dont move' and 0 == 'move first'.

The goal is to minimize all the F(r[i]) in the moves.

-ryan

On Mon, Mar 21, 2011 at 4:26 PM, Jonathan Gray <jgray@fb.com> wrote:
> Also, using more stable measures of request count will help, such as 30 minute rolling
averages.
>
>> -----Original Message-----
>> From: Jonathan Gray [mailto:jgray@fb.com]
>> Sent: Monday, March 21, 2011 4:23 PM
>> To: dev@hbase.apache.org
>> Subject: RE: gauging cost of region movement
>>
>> This is an interesting direction, and definitely file a JIRA as this could be an
>> additional metric in the future, but it's not exactly what I had in mind.
>>
>> One of the hardest parts of load balancing based on request count and other
>> dynamic/transient measures is that you can get some pretty pathological
>> conditions where you are always moving stuff around.
>>
>> To guard against it, I think we'll need to move to more of a cost-based
>> algorithm that is taking not just the difference in request counts into account
>> but also a baseline "cost" of moving a region.  The cost difference in load
>> between two unbalanced servers would have to outweigh the cost
>> associated with moving a region.  As you say, looking at the number of live
>> operations to a given region could contribute to the cost of moving that
>> region, but the best measure for that is probably just looking at request
>> count (it's all requests that incur a cost, not just active scanners).
>>
>> JG
>>
>> > -----Original Message-----
>> > From: Ted Yu [mailto:yuzhihong@gmail.com]
>> > Sent: Monday, March 21, 2011 3:44 PM
>> > To: dev@hbase.apache.org
>> > Subject: gauging cost of region movement
>> >
>> > Can we add a counter for the number of InternalScanner's to HRegion ?
>> > We decrement this counter when close() is called.
>> >
>> > Such counter can be used to gauge the cost of moving the underlying
>> region.
>> >
>> > Cheers
>

Mime
View raw message