harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LvJimmy´╝îJing" <firep...@gmail.com>
Subject Re: [VM]How to trigue GC while free native memory is low.
Date Sat, 03 Feb 2007 03:02:54 GMT
2007/2/2, Xiao-Feng Li <xiaofeng.li@gmail.com>:
> Thank you, Jimmy and Leo, for the clear explanations.
>
> I think temporarily you can put the System.gc() there to workaround
> this issue. Yes, this is only a workaround since we cannot rely on it
> without explicit guarantee in its semantics.
>
> To really solve this problem, we might need a careful design. Your
> proposed approach looks like a possible solution. Have you thought to
> solve the problem by using GC to manage the raw memory blocks? Without
> deep understanding of the problem, I have some wild idea below. Please
> see if it is reasonable or stupid. I will read the spec of ByteBuffer
> to get more understanding.
>
> Since what the ByeBuffer needs are starting address and capacity, it
> doesn't really care if this piece of memory is in Java heap or in
> native runtime memory ( I assume.) We probably can provide some
> special kind of virtual Java object that serves as raw memory block to
> nio. In this works, we need not monitor the native runtime memory
> usage.
>

Ah, this solution sounds interesting to me, yes, if we malloc memory
in heap then we no longer need to monitor the native memory. But we
may need a kit of methods offer this heap-memory service:
1) a function of malloc_in_heap()
2) a function of free_heap_block()
3) heap monitor should record this malloc/free action
4) ...
However I do think this may be a good solution, and it still depends
on how easy can it be implemented. Note not only direct ByteBuffer
require this native memory usage.

> This approach need certain contract between Java classes and GC about
> the special kind of Java object. Probably we can write a layer of Java
> class wrapper for raw memory allocation, which hides the contract from
> other common classes.
>

As we have a portlib, this can be implemented in such hymem_malloc or so.

> At the same time, I am thinking of your proposed approach, and will
> get back later.
>

Thanks, waiting for you the GC guru's appoach towards heap memory allocation :)

> Thanks,
> xiaofeng
>
> On 2/2/07, Leo Li <liyilei1979@gmail.com> wrote:
> > Hi, Xiao-Feng:
> >      Excuse me if I am confusing you.
> >      The direct byte buffer holds a block of native memory space used in nio
> > operation. The native byte buffer will be freed through the byte buffer is
> > gc collected and added to ReferenceQueue monitored by MemorySpy series
> > classes.
> >      But the gc will not be triggered if the java heap is still empty. Here
> > is an example: I allocate a large block of native memory for a byte buffer
> > while the byte buffer itself is quite small. Thus quickly native heap is
> > depleted while VM still does not think gc is needed since java heap has a
> > lot of free memory.
> >
> >
> > import java.nio.*;
> > public class Test {
> >     public static void main(String[] args) throws Exception {
> >         for(int i = 0;i<1000;i++)
> >         {
> >             ByteBuffer byteBuffer = ByteBuffer.allocateDirect(10240000);
> >             //System.gc();
> >         }
> >    }
> > }
> > RI runs well but Harmony will soon throw OutOfMemory exception.
> > But if remove the comment before System.gc(), Harmony will become ok.
> > So my point is that we need a mechanism to notify VM to start gc if we have
> > no more native memory because sometimes problem will be solved if gc is
> > fired. Of cause we cannot avoid a  user always malloc space but never free
> > them.:)
> >
> > Good luck!
> >
> > On 2/2/07, Xiao-Feng Li <xiaofeng.li@gmail.com> wrote:
> > >
> > > Leo, I can't fully understand your problem and proposal. I am working
> > > in DRLVM GC component, I want to understand how GC can tackle the
> > > issue you meet.
> > >
> > > You said the native memory is allocated (probably with malloc) for
> > > byte buffer. I wonder how the process goes. I assume it's in native
> > > code. Then you use malloc to allocate a block of memory, and put some
> > > Java object (byte buffer) into it? Is this what you do? I wonder how
> > > do you manipulate Java object placement in native code; and if the
> > > native memory block is out of Java heap, do you expect GC to manage
> > > it?
> > >
> > > You said when GC reclaims those byte buffer object, the native memory
> > > block can be freed in native code. How do you know if an object is
> > > reclaimed by GC or not?
> > >
> > > Thanks,
> > > xiaofeng
> > >
> > > On 2/2/07, Leo Li < liyilei1979@gmail.com> wrote:
> > > > Hi, all:
> > > >     After applying patch3073, I added the support for direct byte buffer
> > > in
> > > > nio module as spec requires. But now here exists a problem: The native
> > > > memory allocated to the byte buffer can be released when gc releases the
> > > > byte buffer object, however, if the native heap is full while java heap
> > > > still has space, gc will not be triggered.
> > > >     It seems that RI will start gc before native memory heap is depleted
> > > and
> > > > thus prevents out-of-memory error.
> > > >
> > > >    Then our work focuses on:
> > > >    1. When gc is required.
> > > >    2. Trigger a gc.
> > > >    The first one requires that we get support from operating system,
> > > since
> > > > the memory allocated in the native code, for example by malloc, is out
> > > of
> > > > the control of java VM.( I have ever thought of counting the used memory
> > > in
> > > > hymemory_allocate, but the plan fails since hymemory_free will not
> > > report
> > > > how much space is released.)
> > > >    The second one needs the help from VM. System.gc() is not so reliable
> > > as
> > > > spec says, so it is necessary to have a internal channel to notify VM
to
> > > > start gc.
> > > >
> > > >     One solution, I think, is to let a monitor thread in VM to check
> > > whether
> > > > OS physical memory is low. For example, the
> > > QueryMemoryResourceNotification
> > > > of win32 API is a candidate. Although the interrupt model is more
> > > effective:
> > > > win32 SDK provides a CreateMemoryResourceNotification to get a handler
> > > on
> > > > which the monitor thread can wait, maybe on other platforms, OS does not
> > > > supply such a convenience. So the monitor thread in the VM might have
to
> > >
> > > > check the OS resource once for a while and if necessary the monitor
> > > thread
> > > > will call a GC.
> > > >
> > > >    My suggestion is first to add some function to monitor memory in
> > > portlib,
> > > > since it is highly related to platforms and in portlib there has been
> > > > some useful tools.(Thanks Mark to point out that.) On the other hand,
we
> > > can
> > > > negotiate a channel to trigger gc in VM. Actually I am not an expert on
> > > VM
> > > > since I am not sure whether there has been some monitor thread and the
> > > load
> > > > on performance if such a monitor thread is added to the VM...
> > > >
> > > >
> > > > --
> > > > Leo Li
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Leo Li
> > China Software Development Lab, IBM
> >
> >
>


-- 

Best Regards!

Jimmy, Jing Lv
China Software Development Lab, IBM

Mime
View raw message