Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (herse.apache.org: domain of alexey.a.petrenko@gmail.com
 designates 209.85.132.244 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=gX94wMHaI70CfiAIbMpvcIIqzZfwG7Fokw4YZ8TJBfHGJ9uEcDcYmaS/3PByOk6OCREtCVsnb+nRS5yzx/pWvocjQZnNAgnAG7sEBdqifcUyV2B3D4kcnbqMxMSiwKsdFwKO5yaw31L9PZIUd1E6agE5wLW99JOlQH7mKZK2VrM=
Message-ID: <c3755b3a0705250448h4b7a9ffp5b673417e83e2d60@mail.gmail.com>
Date: Fri, 25 May 2007 15:48:14 +0400
From: "Alexey Petrenko" <alexey.a.petrenko@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [classlib] H-3148 (OOM when using native memory which is out of
 GC-control) again
In-Reply-To: <51d555c70705242212v6bcc4ebepd1800ad2e146384c@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <51abf0750705140922r41c0307dw56d79cb0706010aa@mail.gmail.com>
	 <4d0b24970705170724n5e9ce43ev3522f812072dadb8@mail.gmail.com>
	 <51abf0750705180808h6ba2e92at9a2e23767b2bf050@mail.gmail.com>
	 <2c9597b90705230356h435ead85x354ed5a73bef3016@mail.gmail.com>
	 <c3755b3a0705230408o4de85c5co681c3e2fb54d4b42@mail.gmail.com>
	 <51abf0750705230534k3e3c958an9a983c92d931efa2@mail.gmail.com>
	 <c3755b3a0705230604r5ca131e5oc5bf39cbd3fa5e4c@mail.gmail.com>
	 <51abf0750705230619t8434366s3e7e043ea0be44da@mail.gmail.com>
	 <c3755b3a0705230922u5881d606ic5f79d4e931e93b0@mail.gmail.com>
	 <51d555c70705242212v6bcc4ebepd1800ad2e146384c@mail.gmail.com>

2007/5/25, Rana Dasgupta <rdasgupt@gmail.com>:
> Yes GlobalMemoryStatusEx() checks memory pressure on the entire system
> and not for the current process. That's the information needed to
> predict if the next large allocation request could fail.
But this information has no relation to our process and it depends on
the environment only. Mostly... :)
So on highly loaded system this function will always return that GC is
needed but if the system is almost free then it will report that GC is
not needed. And both this results will not be connected to the process
state.

> How about removing absolute memory values from the threshold
> calculation ( since abslute value varies from machine to machine ),
> set loadThreshold to a conservative low value like 65-70%, and just
> use stat.dwMemoryLoad > loadThreshold to indicate low memory
> availability?
65-70% treshold will move the problem from let's say 4G to 6G and nothing else.

SY, Alexey

> On 5/23/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
> > Guys,
> >
> > unfortunately the suggested patch does not work on Windows.
> >
> > Here is the memory check code
> > === cut ===
> > #define MEMORYLOADTHRESHOLD 95 // 95%
> > #define MEMORYFREETHRESHOLD 0x4000000 //64M
> >
> > hysysinfo_is_system_physical_memory_low (struct HyPortLibrary
> > *portLibrary, UDATA load_threshold, UDATA free_threshold)
> > {
> >    MEMORYSTATUSEX stat;
> >    stat.dwLength = sizeof (stat);
> >
> >    if(GlobalMemoryStatusEx(&stat))
> >     {
> >        if(stat.dwMemoryLoad >= load_threshold || stat.ullAvailPhys <=
> > load_threshold){
> >            return TRUE;
> >    }
> > === cut ===
> >
> > GlobalMemoryStatusEx reports global memory status but not the status
> > for the current process. So we are waiting while full system memory
> > load will be higher then 95% or available physical memory will be less
> > the 95 bytes. I guess that this is usual situation on 512M or even 1G
> > system. But this is not true for my 4G server :) So this fuction
> > always return false for me. And OOM happens after 100 iterations while
> > running the reproducer.
> >
> > I undestand that free_threshold and load_threshold parameters are
> > mixed up here. But I believe that putting them in place will not help.
> >
> > So the issue needs additional investigation.
> >
> > SY, Alexey
> >
> > 2007/5/23, Mikhail Markov <mikhail.a.markov@gmail.com>:
> > > +1 for this approach.
> > > As for removing OSResourcesMonitor - not agree: IMO we should have this
> > > facility for another code not using OSMemory.malloc() method which wants to
> > > check the conditions for calling GC.
> > >
> > > Thanks,
> > > Mikhail
> > >
> > >
> > > On 5/23/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
> > > >
> > > > I suggest to call nativeMalloc anyway. nativeMalloc will check the
> > > > native heap and return null or something if gc is needed whithout real
> > > > allocation.
> > > > Java code will check the return value and call gc and nativeMalloc if
> > > > needed.
> > > >
> > > > So the code will look like the following
> > > >
> > > > === cut ===
> > > > public long malloc(long length) throws OutOfMemoryError {
> > > >    int res = mallocNative(length, true);
> > > >    if (res == 0) {
> > > >        System.gc();
> > > >        res = mallocNative(length, false);
> > > >    }
> > > >    return res;
> > > > }
> > > > === cut ===
> > > >
> > > > === cut ===
> > > > JNIEXPORT jlong JNICALL
> > > > Java_org_apache_harmony_luni_platform_OSMemory_mallocNative
> > > > (JNIEnv * env, jobject thiz, jlong size, jboolean checkHeap)
> > > > {
> > > > PORT_ACCESS_FROM_ENV (env);
> > > >
> > > > if (checkHeap) {
> > > >    if (hysysinfo_is_system_physical_memory_low_default()) {
> > > >      return 0;
> > > >    }
> > > > }
> > > >
> > > > void *address = hymem_allocate_memory ((UDATA) size);
> > > >
> > > > if (address == NULL)
> > > >    {
> > > >      throwNewOutOfMemoryError(env, "Insufficient memory available.");
> > > >    }
> > > >
> > > > return (jlong) ((IDATA) address);
> > > > }
> > > > === cut ===
> > > >
> > > > I think that this approach is better because:
> > > > 1. In the good case we got up to 2 times less number of native calls.
> > > > In the worst case we got the same number of native calls.
> > > > 2. OSResourcesMonitor class becomes unneeded.
> > > >
> > > > SY, Alexey
> > > >
> > > >
> > > > 2007/5/23, Mikhail Markov <mikhail.a.markov@gmail.com>:
> > > > > If the method returns negative flag (should run GC) then after GC
> > > > invocation
> > > > > the malloc should be allocated, so the native method could 1) call GC
> > > > from
> > > > > native code (come back to java) and then alloc memory or 2) (this way is
> > > > > implemented by the patch) return to java and java-code will call malloc
> > > > > after GC.
> > > > > Or you suggest to malloc before calling GC in any case?
> > > > >
> > > > > Thanks,
> > > > > Mikhail
> > > > >
> > > > >
> > > > > On 5/23/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
> > > > > >
> > > > > > 2007/5/23, Alexei Zakharov <alexei.zakharov@gmail.com>:
> > > > > > > As far as I know DRLVM does not run on kernels older than 2.6 - see
> > > > > > > old "building on Debian" thread for details [1]. So it looks like in
> > > > > > > order to enable Harmony on pre2.6 (and on pre2.3) kernels we will
> > > > need
> > > > > > > to fix our code in several places anyway. And the
> > > > > > > hysysinfo_is_system_physical_memory_low() function is not the most
> > > > > > > complicated one.
> > > > > > >
> > > > > > > This way, IMHO current patches for HARMONY-3148 look quite good. And
> > > > > > > since the issue itself is rather critical IMO it is not a bad idea
> > > > to
> > > > > > > finally have it fixed and commit patches in their current state. If
> > > > > > > later we decide to use other strategy for native memory deallocation
> > > > > > > then we can always create another JIRA & patch and fix this.
> > > > > > >
> > > > > > > BTW, Tim, if you don't have enough time for this (or do not
> > > > interested
> > > > > > > anymore)  I could take care of it.
> > > > > > I've took it already... :)
> > > > > >
> > > > > > I had a quick look into the patch and got one question...
> > > > > > The patch creates a new class OSResourcesMonitor with the
> > > > > > isSystemPhysicalMemoryLowDefault native method.
> > > > > > OSMemory.malloc calls this method in the very begining and then call
> > > > > > another native method to allocate memory.
> > > > > >
> > > > > > Why do we need two native calls here if we can check the native heap
> > > > > > size in the native method which allocates it? This method can return a
> > > > > > flag to run GC...
> > > > > >
> > > > > > Did I miss something?
> > > > > >
> > > > > > SY, Alexey
> > > > > >
> > > > >
> > > >
> > >
> >
>