hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Stepachev <oct...@gmail.com>
Subject Re: Java Commited Virtual Memory significally larged then Heap Memory
Date Wed, 12 Jan 2011 11:05:43 GMT
No, I use only malloc env var, and I set it (as suggested before) into
hbase-env.sh, and it looks like it eats more less memory (in my case 4.7G vs
3.3G with 2Gheap)

2011/1/12 Friso van Vollenhoven <fvanvollenhoven@xebia.com>

> Thanks.
>
> I went back to hbase 0.89 with 0.1 LZO, which works fine and does not show
> this issue.
>
> I tried with a newer Hbase and LZO version, also with the MALLOC... setting
> but without max direct memory set, so I was wondering whether you need a
> combination of the two to fix things (apparently not).
>
> Now i am wondering whether I did something wrong setting the env var. It
> should just be picked up when it's in hbase-env.sh, right?
>
>
> Friso
>
>
>
> On 12 jan 2011, at 10:59, Andrey Stepachev wrote:
>
> > with MALLOC_ARENA_MAX=2
> >
> > I check -XX:MaxDirectMemorySize=256m, before, but it doesn't affect
> anything
> > (even no OOM
> > exceptions or so on).
> >
> > But it looks like i have exactly the same issue (it looks like). I have
> many
> > 64Mb anon memory blocks.
> > (sometimes they 132MB). And on heavy load i have rapidly growing rss size
> of
> > jvm process.
> >
> > 2011/1/12 Friso van Vollenhoven <fvanvollenhoven@xebia.com>
> >
> >> Just to clarify: you fixed it by setting the MALLOC_MAX_ARENA=? in
> >> hbase-env.sh?
> >>
> >> Did you also use the -XX:MaxDirectMemorySize=256m ?
> >>
> >> It would be nice to check that this is a different than the leakage with
> >> LZO...
> >>
> >>
> >> Thanks,
> >> Friso
> >>
> >>
> >> On 12 jan 2011, at 07:46, Andrey Stepachev wrote:
> >>
> >>> My bad. All things work. Thanks for  Todd Lipcon :)
> >>>
> >>> 2011/1/11 Andrey Stepachev <octo47@gmail.com>
> >>>
> >>>> I tried to set MALLOC_ARENA_MAX=2. But still the same issue like in
> LZO
> >>>> problem thread. All those 65M blocks here. And JVM continues to eat
> >> memory
> >>>> on heavy write load. And yes, I use "improved" kernel
> >>>> Linux 2.6.34.7-0.5.
> >>>>
> >>>> 2011/1/11 Xavier Stevens <xstevens@mozilla.com>
> >>>>
> >>>> Are you using a newer linux kernel with the new and "improved" memory
> >>>>> allocator?
> >>>>>
> >>>>> If so try setting this in hadoop-env.sh:
> >>>>>
> >>>>> export MALLOC_ARENA_MAX=<number of cores you want to use>
> >>>>>
> >>>>> Maybe start by setting it to 4.  You can thank Todd Lipcon if this
> >> works
> >>>>> for you.
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>>
> >>>>> -Xavier
> >>>>>
> >>>>> On 1/11/11 7:24 AM, Andrey Stepachev wrote:
> >>>>>> No. I don't use LZO. I tried even remove any native support
(i.e.
> all
> >>>>> .so
> >>>>>> from class path)
> >>>>>> and use java gzip. But nothing.
> >>>>>>
> >>>>>>
> >>>>>> 2011/1/11 Friso van Vollenhoven <fvanvollenhoven@xebia.com>
> >>>>>>
> >>>>>>> Are you using LZO by any chance? If so, which version?
> >>>>>>>
> >>>>>>> Friso
> >>>>>>>
> >>>>>>>
> >>>>>>> On 11 jan 2011, at 15:57, Andrey Stepachev wrote:
> >>>>>>>
> >>>>>>>> After starting the hbase in jroĐükit found the same
memory leakage.
> >>>>>>>>
> >>>>>>>> After the launch
> >>>>>>>>
> >>>>>>>> Every 2,0 s: date & & ps - sort =- rss-eopid,
rss, vsz, pcpu |
> head
> >>>>>>>> Tue Jan 11 16:49:31 2011
> >>>>>>>>
> >>>>>>>> 11 16:49:31 MSK 2011
> >>>>>>>> PID RSS VSZ% CPU
> >>>>>>>> 7863 2547760 5576744 78.7
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> JR dumps:
> >>>>>>>>
> >>>>>>>> Total mapped 5576740KB (reserved = 2676404KB) - Java
heap
> 2048000KB
> >>>>>>>> (reserved = 1472176KB) - GC tables 68512KB - Thread
stacks 37236KB
> >> (#
> >>>>>>>> threads = 111) - Compiled code 1048576KB (used = 2599KB)
-
> Internal
> >>>>>>>> 1224KB - OS 549688KB - Other 1800976KB - Classblocks
1280KB
> >> (malloced
> >>>>>>>> = 1110KB # 3285) - Java class data 20224KB (malloced
= 20002KB #
> >> 15134
> >>>>>>>> in 3285 classes) - Native memory tracking 1024KB (malloced
= 325KB
> >> +10
> >>>>>>>> KB # 20)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> After running the mr which make high write load (~1hour)
> >>>>>>>>
> >>>>>>>> Every 2,0 s: date & & ps - sort =- rss-eopid,
rss, vsz, pcpu |
> head
> >>>>>>>> Tue Jan 11 17:08:56 2011
> >>>>>>>>
> >>>>>>>> 11 17:08:56 MSK 2011
> >>>>>>>> PID RSS VSZ% CPU
> >>>>>>>> 7863 4072396 5459572 100
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> JR said not important below specify why)
> >>>>>>>>
> >>>>>>>> http://paste.ubuntu.com/552820/
> >>>>>>>> <http://paste.ubuntu.com/552820/>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 7863:
> >>>>>>>> Total mapped                  5742628KB +165888KB
> >> (reserved=1144000KB
> >>>>>>>> -1532404KB)
> >>>>>>>> -              Java heap      2048000KB           (reserved=0KB
> >>>>>>> -1472176KB)
> >>>>>>>> -              GC tables        68512KB
> >>>>>>>> -          Thread stacks        38028KB    +792KB (#threads=114
> +3)
> >>>>>>>> -          Compiled code      1048576KB           (used=3376KB
> >> +776KB)
> >>>>>>>> -               Internal         1480KB    +256KB
> >>>>>>>> -                     OS       517944KB  -31744KB
> >>>>>>>> -                  Other      1996792KB +195816KB
> >>>>>>>> -            Classblocks         1280KB           (malloced=1156KB
> >>>>>>>> +45KB #3421 +136)
> >>>>>>>> -        Java class data        20992KB    +768KB
> (malloced=20843KB
> >>>>>>>> +840KB #15774 +640 in 3421 classes)
> >>>>>>>> - Native memory tracking         1024KB           (malloced=325KB
> >>>>> +10KB
> >>>>>>> #20)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>>>>>  OS                          *java    r x 0x0000000000400000.(
> >>>>>>> 76KB)
> >>>>>>>>  OS                          *java    rw  0x0000000000612000
(
> >>>>>>> 4KB)
> >>>>>>>>  OS                        *[heap]    rw  0x0000000000613000.(
> >>>>>>> 478712KB)
> >>>>>>>> INT                           Poll    r   0x000000007fffe000
(
> >>>>>>> 4KB)
> >>>>>>>> INT                         Membar    rw  0x000000007ffff000.(
> >>>>>>> 4KB)
> >>>>>>>> MSP              Classblocks (1/2)    rw  0x0000000082ec0000
(
> >>>>>>> 768KB)
> >>>>>>>> MSP              Classblocks (2/2)    rw  0x0000000082f80000
(
> >>>>>>> 512KB)
> >>>>>>>> HEAP                      Java heap    rw
> >>>>>>> 0x0000000083000000.(2048000KB)
> >>>>>>>>                                       rw  0x00007f2574000000
(
> >>>>>>> 65500KB)
> >>>>>>>>                                           0x00007f2577ff7000.(
> >>>>>>> 36KB)
> >>>>>>>>                                       rw  0x00007f2584000000
(
> >>>>>>> 65492KB)
> >>>>>>>>                                           0x00007f2587ff5000.(
> >>>>>>> 44KB)
> >>>>>>>>                                       rw  0x00007f258c000000
(
> >>>>>>> 65500KB)
> >>>>>>>>                                           0x00007f258fff7000
(
> >>>>>>> 36KB)
> >>>>>>>>                                       rw  0x00007f2590000000
(
> >>>>>>> 65500KB)
> >>>>>>>>                                           0x00007f2593ff7000
(
> >>>>>>> 36KB)
> >>>>>>>>                                       rw  0x00007f2594000000
(
> >>>>>>> 65500KB)
> >>>>>>>>                                           0x00007f2597ff7000
(
> >>>>>>> 36KB)
> >>>>>>>>                                       rw  0x00007f2598000000
(
> >>>>>>> 131036KB)
> >>>>>>>>                                           0x00007f259fff7000
(
> >>>>>>> 36KB)
> >>>>>>>>                                       rw  0x00007f25a0000000
(
> >>>>>>> 65528KB)
> >>>>>>>>                                           0x00007f25a3ffe000
(
> >>>>>>> 8KB)
> >>>>>>>>                                       rw  0x00007f25a4000000
(
> >>>>>>> 65496KB)
> >>>>>>>>                                           0x00007f25a7ff6000
(
> >>>>>>> 40KB)
> >>>>>>>>                                       rw  0x00007f25a8000000
(
> >>>>>>> 65496KB)
> >>>>>>>>                                           0x00007f25abff6000
(
> >>>>>>> 40KB)
> >>>>>>>>                                       rw  0x00007f25ac000000
(
> >>>>>>> 65504KB)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> So, the difference was in the pieces of memory like
this:
> >>>>>>>>
> >>>>>>>> rw 0x00007f2590000000 (65500KB)
> >>>>>>>>   0x00007f2593ff7000 (36KB)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Looks like HLog allocates memory (looks like HLog, becase
it is
> very
> >>>>>>> similar
> >>>>>>>> size)
> >>>>>>>>
> >>>>>>>> If we count this blocks we get amount of lost memory:
> >>>>>>>>
> >>>>>>>> 65M * 32 + 132M = 2212M
> >>>>>>>>
> >>>>>>>> So, it looks like HLog allcates to many memory, and
question is:
> how
> >>>>> to
> >>>>>>>> restrict it?
> >>>>>>>>
> >>>>>>>> 2010/12/30 Andrey Stepachev <octo47@gmail.com>
> >>>>>>>>
> >>>>>>>>> Hi All.
> >>>>>>>>>
> >>>>>>>>> After heavy load into hbase (single node, nondistributed
test
> >> system)
> >>>>> I
> >>>>>>> got
> >>>>>>>>> 4Gb process size of my HBase java process.
> >>>>>>>>> On 6GB machine there was no room for anything else
(disk cache
> and
> >> so
> >>>>>>> on).
> >>>>>>>>> Does anybody knows, what is going on, and how you
solve this.
> What
> >>>>> heap
> >>>>>>>>> memory is set on you hosts
> >>>>>>>>> and how much of RSS hbase process actually use.
> >>>>>>>>>
> >>>>>>>>> I don't see such things before, all tomcat and other
java apps
> >> don't
> >>>>>>> eats
> >>>>>>>>> significally more memory then -Xmx.
> >>>>>>>>>
> >>>>>>>>> Connection name:   pid: 23476
> >> org.apache.hadoop.hbase.master.HMaster
> >>>>>>>>> start   Virtual Machine:   Java HotSpot(TM) 64-Bit
Server VM
> >> version
> >>>>>>>>> 17.1-b03   Vendor:   Sun Microsystems Inc.   Name:
  23476@mars
> >>>>>>> Uptime:   12
> >>>>>>>>> hours 4 minutes   Process CPU time:   5 hours 45
minutes   JIT
> >>>>> compiler:
> >>>>>>> HotSpot
> >>>>>>>>> 64-Bit Server Compiler   Total compile time:   19,223
seconds
> >>>>>>>>> ------------------------------
> >>>>>>>>>  Current heap size:     703 903 kbytes   Maximum
heap size:   2
> >> 030
> >>>>>>> 976kbytes    Committed memory:
> >>>>>>>>> 2 030 976 kbytes   Pending finalization:   0 objects
     Garbage
> >>>>>>>>> collector:   Name = 'ParNew', Collections = 9 990,
Total time
> spent
> >> =
> >>>>> 5
> >>>>>>>>> minutes   Garbage collector:   Name = 'ConcurrentMarkSweep',
> >>>>> Collections
> >>>>>>> =
> >>>>>>>>> 20, Total time spent = 35,754 seconds
> >>>>>>>>> ------------------------------
> >>>>>>>>>  Operating System:   Linux 2.6.34.7-0.5-xen   Architecture:
> >> amd64
> >>>>>>> Number of processors:
> >>>>>>>>> 8   Committed virtual memory:   4 403 512 kbytes
    Total
> physical
> >>>>>>>>> memory:   6 815 744 kbytes   Free physical memory:
     82 720
> >> kbytes
> >>>>>>> Total swap space:
> >>>>>>>>> 8 393 924 kbytes   Free swap space:   8 050 880
kbytes
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message