hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2636) [hbase] Make cache flush triggering less simplistic
Date Sun, 27 Jan 2008 19:27:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563012#action_12563012
] 

Billy Pearson commented on HADOOP-2636:
---------------------------------------

works ok until the first flush then they flush back to back.
I got my flush size set to 64MB and you see below the flush is on the columns are in the low
KB's and they are flushed several times a min
{code}
2008-01-27 13:23:12,428 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/anchor/3002468228224142295
with 2198 entries, sequence id 2169440, and size 250.1k for 673880687/anchor
2008-01-27 13:23:12,428 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/anchor in 503ms, sequenceid=2169440
2008-01-27 13:23:12,429 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/in_rank
2008-01-27 13:23:13,009 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/in_rank/8574408391914465288
with 1428 entries, sequence id 2171549, and size 158.0k for 673880687/in_rank
2008-01-27 13:23:13,010 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/in_rank in 581ms, sequenceid=2171549
2008-01-27 13:23:13,011 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/last_seen
2008-01-27 13:23:13,603 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/last_seen/1870135185038371518
with 2736 entries, sequence id 2173487, and size 295.4k for 673880687/last_seen
2008-01-27 13:23:13,603 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/last_seen in 592ms, sequenceid=2173487
2008-01-27 13:23:13,609 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/anchor
2008-01-27 13:23:14,179 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/anchor/580140819100995513
with 1989 entries, sequence id 2175620, and size 216.4k for 673880687/anchor
2008-01-27 13:23:14,180 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/anchor in 572ms, sequenceid=2175620
2008-01-27 13:23:14,181 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/in_rank
2008-01-27 13:23:14,693 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/in_rank/5202798596377303202
with 1896 entries, sequence id 2177448, and size 213.7k for 673880687/in_rank
2008-01-27 13:23:14,693 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/in_rank in 512ms, sequenceid=2177448
2008-01-27 13:23:14,696 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/last_seen
2008-01-27 13:23:15,260 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/last_seen/8482135742832457522
with 1940 entries, sequence id 2179528, and size 206.2k for 673880687/last_seen
2008-01-27 13:23:15,260 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/last_seen in 564ms, sequenceid=2179528
2008-01-27 13:23:15,276 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/anchor
2008-01-27 13:23:15,839 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/anchor/7980708509699803761
with 1991 entries, sequence id 2181788, and size 204.6k for 673880687/anchor
2008-01-27 13:23:15,839 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/anchor in 564ms, sequenceid=2181788
2008-01-27 13:23:15,840 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/in_rank
2008-01-27 13:23:16,423 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/in_rank/1305487102801331533
with 2150 entries, sequence id 2184079, and size 221.0k for 673880687/in_rank
2008-01-27 13:23:16,423 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/in_rank in 583ms, sequenceid=2184079
2008-01-27 13:23:16,425 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/last_seen
2008-01-27 13:23:17,138 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/last_seen/1135861004256783910
with 2258 entries, sequence id 2186492, and size 221.2k for 673880687/last_seen
2008-01-27 13:23:17,138 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/last_seen in 713ms, sequenceid=2186492
2008-01-27 13:23:17,139 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region webdata,,1201461077996 store 673880687/anchor
2008-01-27 13:23:17,366 INFO org.apache.hadoop.hbase.HRegionServer: Rolling hlog. Number of
entries: 30003
2008-01-27 13:23:17,738 DEBUG org.apache.hadoop.hbase.HStore: Added 673880687/anchor/5010915044698800730
with 2232 entries, sequence id 2188716, and size 236.5k for 673880687/anchor
2008-01-27 13:23:17,738 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
store 673880687/anchor in 599ms, sequenceid=2188716
{code}

> [hbase] Make cache flush triggering less simplistic
> ---------------------------------------------------
>
>                 Key: HADOOP-2636
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2636
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/hbase
>    Affects Versions: 0.16.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.17.0
>
>         Attachments: patch.txt, patch.txt, patch.txt
>
>
> When flusher runs -- its triggered when the sum of all Stores in a Region > a configurable
max size -- we flush all Stores though a Store memcache might have but a few bytes.
> I would think Stores should only dump their memcache disk if they have some substance.
> The problem becomes more acute, the more families you have in a Region.
> Possible behaviors would be to dump the biggest Store only, or only those Stores >
50% of max memcache size.  Behavior would vary dependent on the prompt that provoked the flush.
 Would also log why the flush is running: optional or > max size.
> This issue comes out of HADOOP-2621.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message