hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "karthikp (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-15309) GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
Date Thu, 25 Feb 2016 20:19:18 GMT

     [ https://issues.apache.org/jira/browse/HBASE-15309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

karthikp updated HBASE-15309:
-----------------------------
    Description: 
I'd memory leak issue in regionserver process, VM and RSS memory continiously increasing 64MB.
I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
I've huge write load and frequent minor compaction, We've used GZip hfile compression.
Max java regionserver heap size is 32GB.


{noformat}
top - 14:28:30 up 201 days, 21:06,  3 users,  load average: 5.67, 3.72, 3.31
Tasks: 803 total,   1 running, 802 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.3%us,  2.1%sy,  0.0%ni, 85.9%id,  3.5%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  65932340k total, 63961912k used,  1970428k free,  2394528k buffers
Swap: 29659132k total,    63532k used, 29595600k free,  1095268k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 57335 hbase     20   0 46.4g  44g 9296 S 98.2 70.9  13319:10 java
{noformat}

{noformat}
 [hbase@xxxxxx-hslave ~]$  pmap -x 57335 | sort -k 3 -nr | more
total kB        48695984 46765792 46756512
00007ff312460000 33171072 33169464 33169464 rwx--    [ anon ]
000000004010a000 1448552 1448552 1448552 rwx--    [ anon ]
00007ff2d1810000  612120  603124  603124 rwx--    [ anon ]
00007ff2fadff000  383364  383364  383364 rwx--    [ anon ]
00007ff0e8000000  131072  131072  131072 rwx--    [ anon ]
00007ff218000000  131048  131048  131048 rwx--    [ anon ]
00007ff128000000  131068  131048  131048 rwx--    [ anon ]
00007ff230000000  131044  131044  131044 rwx--    [ anon ]
00007ff000000000  131036  131036  131036 rwx--    [ anon ]
00007fefe0000000  131060  131036  131036 rwx--    [ anon ]
00007ff23c000000   65536   65536   65536 rwx--    [ anon ]
00007ff0a4000000   65536   65536   65536 rwx--    [ anon ]
00007ff054000000   65536   65536   65536 rwx--    [ anon ]
00007ff01c000000   65536   65536   65536 rwx--    [ anon ]
00007fefb4000000   65536   65536   65536 rwx--    [ anon ]
00007ff22c000000   65532   65532   65532 rwx--    [ anon ]
00007ff110000000   65532   65532   65532 rwx--    [ anon ]
00007ff10c000000   65532   65532   65532 rwx--    [ anon ]
00007ff0b8000000   65532   65532   65532 rwx--    [ anon ]
00007ff09c000000   65532   65532   65532 rwx--    [ anon ]
00007feff8000000   65532   65532   65532 rwx--    [ anon ]
00007ff250000000   65528   65528   65528 rwx--    [ anon ]
--More--
{noformat}

{noformat}
$pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print $i}'
| wc -l
146
{noformat}

Regionserver process has many anon pages, size varying from 64000 to 65536 KB that shown above.

  was:
I'd memory leak issue in regionserver process, VM and RSS memory continiously increasing 64MB.
I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
I've huge write load and frequent minor compaction, We've used GZip hfile compression.
Max java regionserver heap size is 32GB.


{noformat}
top - 14:28:30 up 201 days, 21:06,  3 users,  load average: 5.67, 3.72, 3.31
Tasks: 803 total,   1 running, 802 sleeping,   0 stopped,   0 zombie
Cpu(s):  8.3%us,  2.1%sy,  0.0%ni, 85.9%id,  3.5%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  65932340k total, 63961912k used,  1970428k free,  2394528k buffers
Swap: 29659132k total,    63532k used, 29595600k free,  1095268k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 57335 hbase     20   0 46.4g  44g 9296 S 98.2 70.9  13319:10 java
{noformat}

{noformat}
 [hbase@xxxxxx-hslave ~]$  pmap -x 57335 | sort -k 3 -nr | more
total kB        48695984 46765792 46756512
00007ff312460000 33171072 33169464 33169464 rwx--    [ anon ]
000000004010a000 1448552 1448552 1448552 rwx--    [ anon ]
00007ff2d1810000  612120  603124  603124 rwx--    [ anon ]
00007ff2fadff000  383364  383364  383364 rwx--    [ anon ]
00007ff0e8000000  131072  131072  131072 rwx--    [ anon ]
00007ff218000000  131048  131048  131048 rwx--    [ anon ]
00007ff128000000  131068  131048  131048 rwx--    [ anon ]
00007ff230000000  131044  131044  131044 rwx--    [ anon ]
00007ff000000000  131036  131036  131036 rwx--    [ anon ]
00007fefe0000000  131060  131036  131036 rwx--    [ anon ]
00007ff23c000000   65536   65536   65536 rwx--    [ anon ]
00007ff0a4000000   65536   65536   65536 rwx--    [ anon ]
00007ff054000000   65536   65536   65536 rwx--    [ anon ]
00007ff01c000000   65536   65536   65536 rwx--    [ anon ]
00007fefb4000000   65536   65536   65536 rwx--    [ anon ]
00007ff22c000000   65532   65532   65532 rwx--    [ anon ]
00007ff110000000   65532   65532   65532 rwx--    [ anon ]
00007ff10c000000   65532   65532   65532 rwx--    [ anon ]
00007ff0b8000000   65532   65532   65532 rwx--    [ anon ]
00007ff09c000000   65532   65532   65532 rwx--    [ anon ]
00007feff8000000   65532   65532   65532 rwx--    [ anon ]
00007ff250000000   65528   65528   65528 rwx--    [ anon ]
--More--
{noformat}

{noformat}
$pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print $i}'
| wc -l
146
{noformat}

Will MALLOC_AREANA have exactly 65536KB? 
Regionserver process has many anon pages, size varying from 64000 to 65536 KB that shown above.


> GZIP HFile Compression & Decompression Memory Leak, Similar Issue HBASE-5516
> ----------------------------------------------------------------------------
>
>                 Key: HBASE-15309
>                 URL: https://issues.apache.org/jira/browse/HBASE-15309
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile
>    Affects Versions: 0.94.6
>            Reporter: karthikp
>            Assignee: Todd Lipcon
>            Priority: Blocker
>
> I'd memory leak issue in regionserver process, VM and RSS memory continiously increasing
64MB.
> I'm using hbase-0.94.6 , MALLOC_ARENA_MAX=4 was set in hbase-env.sh
> I've huge write load and frequent minor compaction, We've used GZip hfile compression.
> Max java regionserver heap size is 32GB.
> {noformat}
> top - 14:28:30 up 201 days, 21:06,  3 users,  load average: 5.67, 3.72, 3.31
> Tasks: 803 total,   1 running, 802 sleeping,   0 stopped,   0 zombie
> Cpu(s):  8.3%us,  2.1%sy,  0.0%ni, 85.9%id,  3.5%wa,  0.0%hi,  0.1%si,  0.0%st
> Mem:  65932340k total, 63961912k used,  1970428k free,  2394528k buffers
> Swap: 29659132k total,    63532k used, 29595600k free,  1095268k cached
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  57335 hbase     20   0 46.4g  44g 9296 S 98.2 70.9  13319:10 java
> {noformat}
> {noformat}
>  [hbase@xxxxxx-hslave ~]$  pmap -x 57335 | sort -k 3 -nr | more
> total kB        48695984 46765792 46756512
> 00007ff312460000 33171072 33169464 33169464 rwx--    [ anon ]
> 000000004010a000 1448552 1448552 1448552 rwx--    [ anon ]
> 00007ff2d1810000  612120  603124  603124 rwx--    [ anon ]
> 00007ff2fadff000  383364  383364  383364 rwx--    [ anon ]
> 00007ff0e8000000  131072  131072  131072 rwx--    [ anon ]
> 00007ff218000000  131048  131048  131048 rwx--    [ anon ]
> 00007ff128000000  131068  131048  131048 rwx--    [ anon ]
> 00007ff230000000  131044  131044  131044 rwx--    [ anon ]
> 00007ff000000000  131036  131036  131036 rwx--    [ anon ]
> 00007fefe0000000  131060  131036  131036 rwx--    [ anon ]
> 00007ff23c000000   65536   65536   65536 rwx--    [ anon ]
> 00007ff0a4000000   65536   65536   65536 rwx--    [ anon ]
> 00007ff054000000   65536   65536   65536 rwx--    [ anon ]
> 00007ff01c000000   65536   65536   65536 rwx--    [ anon ]
> 00007fefb4000000   65536   65536   65536 rwx--    [ anon ]
> 00007ff22c000000   65532   65532   65532 rwx--    [ anon ]
> 00007ff110000000   65532   65532   65532 rwx--    [ anon ]
> 00007ff10c000000   65532   65532   65532 rwx--    [ anon ]
> 00007ff0b8000000   65532   65532   65532 rwx--    [ anon ]
> 00007ff09c000000   65532   65532   65532 rwx--    [ anon ]
> 00007feff8000000   65532   65532   65532 rwx--    [ anon ]
> 00007ff250000000   65528   65528   65528 rwx--    [ anon ]
> --More--
> {noformat}
> {noformat}
> $pmap -x 57335 | awk '{print $3}' | awk '{ if($i<65536 && $i>64000) print
$i}' | wc -l
> 146
> {noformat}
> Regionserver process has many anon pages, size varying from 64000 to 65536 KB that shown
above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message