hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Kellerman (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-2674) memcache size reported in debug log doesn ot add up
Date Tue, 22 Jan 2008 08:16:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561241#action_12561241
] 

jimk edited comment on HADOOP-2674 at 1/22/08 12:15 AM:
-----------------------------------------------------------------

I think the reason it doesn't add up is that there is either an optional cache flush going
on or the region is being closed during a split. When a region is closed, what gets flushed
is what ever is in the cache at the time.  

In the current implementation, the caches are flushed when the total for all columns reaches
64M. The only other callers of cache flush are the optional timed cache flush and when a region
is closed.

Optional cache flushing and zero length flushes will be removed in HADOOP-2636. Further, since
cache flushing will be on a per column basis, cache flushing will only occur when the cache
for that column reaches 64M.

      was (Author: jimk):
    I think the reason it doesn't add up is that there is either an optional cache flush going
on or the region is being closed during a split. When a region is closed, what gets flushed
is what ever is in the cache at the time.  

Since the caching is done on a per column basis, each column can have up to 64M cached.

Optional cache flushing and zero length flushes will be removed in HADOOP-2636.
  
> memcache size reported in debug log doesn ot add up
> ---------------------------------------------------
>
>                 Key: HADOOP-2674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2674
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>         Environment: hadoop svn 613782
>            Reporter: Billy Pearson
>            Priority: Minor
>             Fix For: 0.16.0
>
>
> below shows a memcache size of 64M but when you add up the regions I get
> 149.3K =  152883.2B
> 153B = 153B
> 153B = 153B
> 3.8M = 3984588.8B
> 2.6M = 2726297.6B
> 153B = 153B
> 129K = 132096B
> 2.6M = 2726297.6B
> 153B = 153B
> total  = 9569891.4B = 9.12M not 64M
> {code}
> 2008-01-21 23:51:30,812 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush
for region webdata,,1200970130265. Size 64.0m
> 2008-01-21 23:51:31,694 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/size/2337100008775916241
with 11004 entries, sequence id 32535827, and size 149.3k for 540402979/size
> 2008-01-21 23:51:31,956 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/rank_total/6511354828803004572
with 0 entries, sequence id 32535827, and size 153.0 for 540402979/rank_total
> 2008-01-21 23:51:32,210 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/first_seen/116626538562002170
with 0 entries, sequence id 32535827, and size 153.0 for 540402979/first_seen
> 2008-01-21 23:51:40,084 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/anchor/5635668875929221096
with 189726 entries, sequence id 32535827, and size 3.8m for 540402979/anchor
> 2008-01-21 23:51:48,134 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/last_seen/5152520466951569025
with 189726 entries, sequence id 32535827, and size 2.6m for 540402979/last_seen
> 2008-01-21 23:51:48,400 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/source/7366562139108336733
with 0 entries, sequence id 32535827, and size 153.0 for 540402979/source
> 2008-01-21 23:51:49,075 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/stime/5227996581170475543
with 11004 entries, sequence id 32535827, and size 129.6k for 540402979/stime
> 2008-01-21 23:51:55,944 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/in_rank/7235419568042410782
with 189726 entries, sequence id 32535827, and size 2.6m for 540402979/in_rank
> 2008-01-21 23:51:56,217 DEBUG org.apache.hadoop.hbase.HStore: Added 540402979/rank_visual/5975562391477267376
with 0 entries, sequence id 32535827, and size 153.0 for 540402979/rank_visual
> 2008-01-21 23:51:56,217 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush
for region webdata,,1200970130265 in 25405ms, sequenceid=32535827
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message