hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-674) memcache size unreliable
Date Fri, 04 Jul 2008 08:29:42 GMT

    [ https://issues.apache.org/jira/browse/HBASE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610482#action_12610482
] 

Billy Pearson commented on HBASE-674:
-------------------------------------

I changed my flush size to 16MB from my default 128MB and run a large job here is some lines
from the logs
also I added back StringUtils.humanReadableInt(this.memcacheSize.get()) on the end so I could
see if the size was growing after every flush

{code}
2008-07-04 03:09:46,684 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region webdata,,1214968800601 in 5595ms, sequence id=237700684, 12.9m, 351.2k
2008-07-04 03:15:02,169 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region webdata,,1214968800601 in 5741ms, sequence id=239128833, 13.4m, 188.1k
2008-07-04 03:15:04,145 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region webdata,,1214968800601 in 1975ms, sequence id=239155758, 167.6k, 222.8k
{code}

so the above looks good now the last number is the memcacheSize.get() and its moveing down
and up so thats good to see I thank this patch solved my problem of the flushes.
I run the job for quite a while and flushes seams to happen normal in place of less time between
flushes.

The only other thing I see on this issue is the size before flush of the memcache is still
off like the logs stack posted above

{code}
2008-07-04 03:09:41,089 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region webdata,,1214968800601. Current region memcache size 16.0m
2008-07-04 03:09:42,587 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/webdata/1748955538/anchor/mapfiles/5595070426400799233
with 29142 entries, sequence id 237700684, data size 3.7m, file size 507.2k
2008-07-04 03:09:43,719 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/webdata/1748955538/stime/mapfiles/4943615301545296809
with 2872 entries, sequence id 237700684, data size 154.9k, file size 24.3k
2008-07-04 03:09:45,197 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/webdata/1748955538/in_rank/mapfiles/367761225010760821
with 36409 entries, sequence id 237700684, data size 4.5m, file size 415.8k
2008-07-04 03:09:45,470 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/webdata/1748955538/size/mapfiles/6451240725630689572
with 2872 entries, sequence id 237700684, data size 135.9k, file size 28.4k
2008-07-04 03:09:46,683 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/webdata/1748955538/last_seen/mapfiles/3472123077784461203
with 36410 entries, sequence id 237700684, data size 4.4m, file size 409.3k
2008-07-04 03:09:46,684 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memcache
flush for region webdata,,1214968800601 in 5595ms, sequence id=237700684, 12.9m, 351.2k
{code}

16m is whats reported as the size before the flush and the total data flushed was 12.9m

> memcache size unreliable
> ------------------------
>
>                 Key: HBASE-674
>                 URL: https://issues.apache.org/jira/browse/HBASE-674
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.1.2
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.2.0
>
>         Attachments: 674-v2.patch, 674.patch, patch.txt
>
>
> Multiple updates against same row/column/ts will be seen as increments to cache size
on insert but when we then play the memcache at flush time, we'll only see the most recent
entry and decrement the memcache size by whatever its size; memcache will be off.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message