Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of wav100@gmail.com designates
 209.85.161.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=k8l5z2awfffEL26aE0UDopeMqDxB8HEHI06yPq9aspJpEjEJlzssefiUQPfmwd8/Zv
         5vOREAC1RqGA3l6wFXc/7DJAcFYwpFpOGPSOoTnTy9sgLijgBdsio5Rzzq4nqNT3YZ7v
         txMpOx5PG6Wwc5NwGFtgfYX59WJ/4JUIsPYVg=
MIME-Version: 1.0
Date: Wed, 5 Jan 2011 11:10:18 -0500
Message-ID: <AANLkTin7deyjq1F4TQu5COfpckRZMCbz0-r17rFk9jZ9@mail.gmail.com>
Subject: JVM OOM
From: Wayne <wav100@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a62d11bd9604991b9f1b

--20cf3054a62d11bd9604991b9f1b
Content-Type: text/plain; charset=ISO-8859-1

I am still struggling with the JVM. We just had a hard OOM crash of a region
server after only running for 36 hours. Any help would be greatly
appreciated. Do we need to restart nodes every 24 hours under load?  GC
Pauses are something we are trying to plan for, but full out OOM crashes are
a new problem.

The message below seems to be where it starts going bad. It is followed by
no less than 63 Concurrent Mode Failure errors over a 16 minute period.

*GC locker: Trying a full collection because scavenge failed*

Lastly here is the end (after the 63 CMF errors).

Heap
 par new generation   total 1887488K, used 303212K [0x00000005fae00000,
0x000000067ae00000, 0x000000067ae00000)
  eden space 1677824K,  18% used [0x00000005fae00000, 0x000000060d61b078,
0x0000000661480000)
  from space 209664K,   0% used [0x000000066e140000, 0x000000066e140000,
0x000000067ae00000)
  to   space 209664K,   0% used [0x0000000661480000, 0x0000000661480000,
0x000000066e140000)
 concurrent mark-sweep generation total 6291456K, used 2440155K
[0x000000067ae00000, 0x00000007fae00000, 0x00000007fae00000)
 concurrent-mark-sweep perm gen total 31704K, used 18999K
[0x00000007fae00000, 0x00000007fccf6000, 0x0000000800000000)

Here again are our custom settings in case there are some suggestions out
there. Are we making it worse with these settings? What should we try next?

        -XX:+UseCMSInitiatingOccupancyOnly
        -XX:CMSInitiatingOccupancyFraction=60
        -XX:+CMSParallelRemarkEnabled
        -XX:SurvivorRatio=8
        -XX:NewRatio=3
        -XX:MaxTenuringThreshold=1


Thanks!

--20cf3054a62d11bd9604991b9f1b--