Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (herse.apache.org: domain of liyilei1979@gmail.com
 designates 66.249.92.174 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=WzV3jJ8qEGgWrbEBPxFcLPhLuGLVaicBqq+gP2JupGbuzHaRKbQWj6FA8+jWiqNku+nCuB03vIJ1Q5aBrugxGvWcfD5j55rUWyQWa+vG5DOV0jvePEGRB7NYOxEylzwhnqdLGuPIBvudwjHIHrd2tEgQx/vcUogn6Hx6BW10vLA=
Message-ID: <e66844de0705250529t32b7f689x27b9647c3338f3c8@mail.gmail.com>
Date: Fri, 25 May 2007 20:29:35 +0800
From: "Leo Li" <liyilei1979@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [classlib] H-3148 (OOM when using native memory which is out of
 GC-control) again
In-Reply-To: <c3755b3a0705250448h4b7a9ffp5b673417e83e2d60@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_108903_6096273.1180096175469"
References: <51abf0750705140922r41c0307dw56d79cb0706010aa@mail.gmail.com>
	 <51abf0750705180808h6ba2e92at9a2e23767b2bf050@mail.gmail.com>
	 <2c9597b90705230356h435ead85x354ed5a73bef3016@mail.gmail.com>
	 <c3755b3a0705230408o4de85c5co681c3e2fb54d4b42@mail.gmail.com>
	 <51abf0750705230534k3e3c958an9a983c92d931efa2@mail.gmail.com>
	 <c3755b3a0705230604r5ca131e5oc5bf39cbd3fa5e4c@mail.gmail.com>
	 <51abf0750705230619t8434366s3e7e043ea0be44da@mail.gmail.com>
	 <c3755b3a0705230922u5881d606ic5f79d4e931e93b0@mail.gmail.com>
	 <51d555c70705242212v6bcc4ebepd1800ad2e146384c@mail.gmail.com>
	 <c3755b3a0705250448h4b7a9ffp5b673417e83e2d60@mail.gmail.com>

------=_Part_108903_6096273.1180096175469
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

   +1 to use loosen threshold.
   It is a strategy about when it is necessary to fire gc. The best value is
really an combination of os, vm and the system configuration.
Besides, according to the profiling of RI, it seems to behave quite
aggressively to fire gc and the memory it used seems to remain at a quite
low level when we are always allocating direct byte buffer. Thus RI can,to a
degree, assure not to encounter out-of-memory exception in the scenario.
  Although to aggressively call gc is also a burden on the whole system,
I prefer to use a loosen threshold, such as 65-70% to ensure the validity of
the function before we have a better solution.
  Furthermore, I believe, to balance preformance and reliability, the
default strategy cannot work so good. Maybe we can let user to set
the boundary of native memory usage, just like the -Xmx option. Of course,
this is another story.:)


On 5/25/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
>
> 2007/5/25, Rana Dasgupta <rdasgupt@gmail.com>:
> > Yes GlobalMemoryStatusEx() checks memory pressure on the entire system
> > and not for the current process. That's the information needed to
> > predict if the next large allocation request could fail.
> But this information has no relation to our process and it depends on
> the environment only. Mostly... :)
> So on highly loaded system this function will always return that GC is
> needed but if the system is almost free then it will report that GC is
> not needed. And both this results will not be connected to the process
> state.
>
> > How about removing absolute memory values from the threshold
> > calculation ( since abslute value varies from machine to machine ),
> > set loadThreshold to a conservative low value like 65-70%, and just
> > use stat.dwMemoryLoad > loadThreshold to indicate low memory
> > availability?
> 65-70% treshold will move the problem from let's say 4G to 6G and nothing
> else.
>
> SY, Alexey
>
> > On 5/23/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
> > > Guys,
> > >
> > > unfortunately the suggested patch does not work on Windows.
> > >
> > > Here is the memory check code
> > > === cut ===
> > > #define MEMORYLOADTHRESHOLD 95 // 95%
> > > #define MEMORYFREETHRESHOLD 0x4000000 //64M
> > >
> > > hysysinfo_is_system_physical_memory_low (struct HyPortLibrary
> > > *portLibrary, UDATA load_threshold, UDATA free_threshold)
> > > {
> > >    MEMORYSTATUSEX stat;
> > >    stat.dwLength = sizeof (stat);
> > >
> > >    if(GlobalMemoryStatusEx(&stat))
> > >     {
> > >        if(stat.dwMemoryLoad >= load_threshold || stat.ullAvailPhys <=
> > > load_threshold){
> > >            return TRUE;
> > >    }
> > > === cut ===
> > >
> > > GlobalMemoryStatusEx reports global memory status but not the status
> > > for the current process. So we are waiting while full system memory
> > > load will be higher then 95% or available physical memory will be less
> > > the 95 bytes. I guess that this is usual situation on 512M or even 1G
> > > system. But this is not true for my 4G server :) So this fuction
> > > always return false for me. And OOM happens after 100 iterations while
> > > running the reproducer.
> > >
> > > I undestand that free_threshold and load_threshold parameters are
> > > mixed up here. But I believe that putting them in place will not help.
> > >
> > > So the issue needs additional investigation.
> > >
> > > SY, Alexey
> > >
> > > 2007/5/23, Mikhail Markov <mikhail.a.markov@gmail.com>:
> > > > +1 for this approach.
> > > > As for removing OSResourcesMonitor - not agree: IMO we should have
> this
> > > > facility for another code not using OSMemory.malloc() method which
> wants to
> > > > check the conditions for calling GC.
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > >
> > > > On 5/23/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
> > > > >
> > > > > I suggest to call nativeMalloc anyway. nativeMalloc will check the
> > > > > native heap and return null or something if gc is needed whithout
> real
> > > > > allocation.
> > > > > Java code will check the return value and call gc and nativeMalloc
> if
> > > > > needed.
> > > > >
> > > > > So the code will look like the following
> > > > >
> > > > > === cut ===
> > > > > public long malloc(long length) throws OutOfMemoryError {
> > > > >    int res = mallocNative(length, true);
> > > > >    if (res == 0) {
> > > > >        System.gc();
> > > > >        res = mallocNative(length, false);
> > > > >    }
> > > > >    return res;
> > > > > }
> > > > > === cut ===
> > > > >
> > > > > === cut ===
> > > > > JNIEXPORT jlong JNICALL
> > > > > Java_org_apache_harmony_luni_platform_OSMemory_mallocNative
> > > > > (JNIEnv * env, jobject thiz, jlong size, jboolean checkHeap)
> > > > > {
> > > > > PORT_ACCESS_FROM_ENV (env);
> > > > >
> > > > > if (checkHeap) {
> > > > >    if (hysysinfo_is_system_physical_memory_low_default()) {
> > > > >      return 0;
> > > > >    }
> > > > > }
> > > > >
> > > > > void *address = hymem_allocate_memory ((UDATA) size);
> > > > >
> > > > > if (address == NULL)
> > > > >    {
> > > > >      throwNewOutOfMemoryError(env, "Insufficient memory
> available.");
> > > > >    }
> > > > >
> > > > > return (jlong) ((IDATA) address);
> > > > > }
> > > > > === cut ===
> > > > >
> > > > > I think that this approach is better because:
> > > > > 1. In the good case we got up to 2 times less number of native
> calls.
> > > > > In the worst case we got the same number of native calls.
> > > > > 2. OSResourcesMonitor class becomes unneeded.
> > > > >
> > > > > SY, Alexey
> > > > >
> > > > >
> > > > > 2007/5/23, Mikhail Markov <mikhail.a.markov@gmail.com>:
> > > > > > If the method returns negative flag (should run GC) then after
> GC
> > > > > invocation
> > > > > > the malloc should be allocated, so the native method could 1)
> call GC
> > > > > from
> > > > > > native code (come back to java) and then alloc memory or 2)
> (this way is
> > > > > > implemented by the patch) return to java and java-code will call
> malloc
> > > > > > after GC.
> > > > > > Or you suggest to malloc before calling GC in any case?
> > > > > >
> > > > > > Thanks,
> > > > > > Mikhail
> > > > > >
> > > > > >
> > > > > > On 5/23/07, Alexey Petrenko <alexey.a.petrenko@gmail.com> wrote:
> > > > > > >
> > > > > > > 2007/5/23, Alexei Zakharov <alexei.zakharov@gmail.com>:
> > > > > > > > As far as I know DRLVM does not run on kernels older than
> 2.6 - see
> > > > > > > > old "building on Debian" thread for details [1]. So it looks
> like in
> > > > > > > > order to enable Harmony on pre2.6 (and on pre2.3) kernels we
> will
> > > > > need
> > > > > > > > to fix our code in several places anyway. And the
> > > > > > > > hysysinfo_is_system_physical_memory_low() function is not
> the most
> > > > > > > > complicated one.
> > > > > > > >
> > > > > > > > This way, IMHO current patches for HARMONY-3148 look quite
> good. And
> > > > > > > > since the issue itself is rather critical IMO it is not a
> bad idea
> > > > > to
> > > > > > > > finally have it fixed and commit patches in their current
> state. If
> > > > > > > > later we decide to use other strategy for native memory
> deallocation
> > > > > > > > then we can always create another JIRA & patch and fix this.
> > > > > > > >
> > > > > > > > BTW, Tim, if you don't have enough time for this (or do not
> > > > > interested
> > > > > > > > anymore)  I could take care of it.
> > > > > > > I've took it already... :)
> > > > > > >
> > > > > > > I had a quick look into the patch and got one question...
> > > > > > > The patch creates a new class OSResourcesMonitor with the
> > > > > > > isSystemPhysicalMemoryLowDefault native method.
> > > > > > > OSMemory.malloc calls this method in the very begining and
> then call
> > > > > > > another native method to allocate memory.
> > > > > > >
> > > > > > > Why do we need two native calls here if we can check the
> native heap
> > > > > > > size in the native method which allocates it? This method can
> return a
> > > > > > > flag to run GC...
> > > > > > >
> > > > > > > Did I miss something?
> > > > > > >
> > > > > > > SY, Alexey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Leo Li
China Software Development Lab, IBM

------=_Part_108903_6096273.1180096175469--