hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3421) Very wide rows -- 30M plus -- cause us OOME
Date Fri, 27 May 2011 06:34:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040082#comment-13040082
] 

Todd Lipcon commented on HBASE-3421:
------------------------------------

I think I'm seeing this on one workload as well - I added a "stats" command for the HFile
tool, then ran it on a problematic 1.5GB gzip-compressed HFile that won't compact due to OOME.

Key length: count: 272358037    min: 59 max: 2259       mean: 59.620136732003246
Val length: count: 272358037    min: 0  max: 0  mean: 0.0
Row size (bytes): count: 1287745        min: 67 max: 15615127736        mean: 14301.657317248368
Row size (columns): count: 1287745      min: 1  max: 233061608  mean: 211.49997631518661

so in this case there's actually a row with 233 million columns which takes up 15GB uncompressed!

> Very wide rows -- 30M plus -- cause us OOME
> -------------------------------------------
>
>                 Key: HBASE-3421
>                 URL: https://issues.apache.org/jira/browse/HBASE-3421
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.90.0
>            Reporter: stack
>
> From the list, see 'jvm oom' in http://mail-archives.apache.org/mod_mbox/hbase-user/201101.mbox/browser,
it looks like wide rows -- 30M or so -- causes OOME during compaction.  We should check it
out. Can the scanner used during compactions use the 'limit' when nexting?  If so, this should
save our OOME'ing (or, we need to add to the next a max size rather than count of KVs).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message