Leo Li wrote: > Thanks for all the advices. > Can we conclude that: > > 1. The problem is related to a series of problems, as Ivan summarized. > It is > about the pattern to release native resources in java. > 2. It is related to gc, if we do not intend to design another mechanism to > track object reference. Only when the object cannot be referenced, have we > the right to release the native resource of it. Actually the problem to > release native resources can occur in any object besides direct byte buffer > and zip related objects. > > But now we have some design problems: > 1. When to trigger gc? > It might be too late when native memory allocation fails. It will cost a > period of time for VM to complete gc and invoke phantom reference, not to > mention other resource releases written in finalize method, as Gregory > mentions. > 2. The decision will be made on which level? > System wide, per process, or per VM? > I think it is the easiest to monitor memory resource on system wide > level: Normally os will report its free physical memory and total physical > memory, which we can rely on to deduce a tension in memory allocation. > Per process is a little difficult:how much memory it used is regarded > to exist memory shortage? > Per VM is the most difficult: besides the problem from per process, we > can hardly discriminate the usage of each vm in one process. The normal > c-library just tells me the size we allocate in malloc by no message of the > memory it releases in free. > > 3. Is it wise to allocate memory in a managed heap? There is some > difference > between Xiao Feng and Jimmy's advice: > Xiao Feng considers a way to allocate direct byte buffer on java heap. > It will be great if it works: Then we will have no problem in releasing > native resources. > Jimmy proposes we can give each VM a memory booking. It also works, I > believe. But is it a little too onerous for us to implement a malloc/free > function ourselves? > > Here are my own opinion: > Although system wide memory monitor can be influnced by other process, I > prefer it. Base on per process or per vm memory allocation, we can get just > the information about virtual memory allocation. Hardly can we deduce that > memory shortage occurs only from the virtual memory one process or vm > uses : > by page exchanging, os is able to provide more physical memory for one > process or another.Native memory allocation failure is the direct result > from shortage of system physical memory : os cannot commit enough physial > memory for the required virtual pages. > > Since I am not an expert in the area, excuse me if I miss something.:) > On 2/3/07, LvJimmy，Jing wrote: >> >> 2007/2/2, Xiao-Feng Li : >> > Thank you, Jimmy and Leo, for the clear explanations. >> > >> > I think temporarily you can put the System.gc() there to workaround >> > this issue. Yes, this is only a workaround since we cannot rely on it >> > without explicit guarantee in its semantics. >> > >> > To really solve this problem, we might need a careful design. Your >> > proposed approach looks like a possible solution. Have you thought to >> > solve the problem by using GC to manage the raw memory blocks? Without >> > deep understanding of the problem, I have some wild idea below. Please >> > see if it is reasonable or stupid. I will read the spec of ByteBuffer >> > to get more understanding. >> > >> > Since what the ByeBuffer needs are starting address and capacity, it >> > doesn't really care if this piece of memory is in Java heap or in >> > native runtime memory ( I assume.) We probably can provide some >> > special kind of virtual Java object that serves as raw memory block to >> > nio. In this works, we need not monitor the native runtime memory >> > usage. >> > >> >> Ah, this solution sounds interesting to me, yes, if we malloc memory >> in heap then we no longer need to monitor the native memory. But we >> may need a kit of methods offer this heap-memory service: >> 1) a function of malloc_in_heap() >> 2) a function of free_heap_block() >> 3) heap monitor should record this malloc/free action >> 4) ... >> However I do think this may be a good solution, and it still depends >> on how easy can it be implemented. Note not only direct ByteBuffer >> require this native memory usage. >> >> > This approach need certain contract between Java classes and GC about >> > the special kind of Java object. Probably we can write a layer of Java >> > class wrapper for raw memory allocation, which hides the contract from >> > other common classes. >> > >> >> As we have a portlib, this can be implemented in such hymem_malloc or so. >> >> > At the same time, I am thinking of your proposed approach, and will >> > get back later. >> > >> >> Thanks, waiting for you the GC guru's appoach towards heap memory >> allocation :) >> >> > Thanks, >> > xiaofeng >> > >> > On 2/2/07, Leo Li wrote: >> > > Hi, Xiao-Feng: >> > > Excuse me if I am confusing you. >> > > The direct byte buffer holds a block of native memory space used >> in nio >> > > operation. The native byte buffer will be freed through the byte >> buffer is >> > > gc collected and added to ReferenceQueue monitored by MemorySpy >> series >> > > classes. >> > > But the gc will not be triggered if the java heap is still >> empty. >> Here >> > > is an example: I allocate a large block of native memory for a byte >> buffer >> > > while the byte buffer itself is quite small. Thus quickly native heap >> is >> > > depleted while VM still does not think gc is needed since java heap >> has a >> > > lot of free memory. >> > > >> > > >> > > import java.nio.*; >> > > public class Test { >> > > public static void main(String[] args) throws Exception { >> > > for(int i = 0;i<1000;i++) >> > > { >> > > ByteBuffer byteBuffer = ByteBuffer.allocateDirect >> (10240000); >> > > //System.gc(); >> > > } >> > > } >> > > } >> > > RI runs well but Harmony will soon throw OutOfMemory exception. >> > > But if remove the comment before System.gc(), Harmony will become ok. >> > > So my point is that we need a mechanism to notify VM to start gc >> if we >> have >> > > no more native memory because sometimes problem will be solved if gc >> is >> > > fired. Of cause we cannot avoid a user always malloc space but never >> free >> > > them.:) >> > > >> > > Good luck! >> > > >> > > On 2/2/07, Xiao-Feng Li wrote: >> > > > >> > > > Leo, I can't fully understand your problem and proposal. I am >> working >> > > > in DRLVM GC component, I want to understand how GC can tackle the >> > > > issue you meet. >> > > > >> > > > You said the native memory is allocated (probably with malloc) for >> > > > byte buffer. I wonder how the process goes. I assume it's in native >> > > > code. Then you use malloc to allocate a block of memory, and put >> some >> > > > Java object (byte buffer) into it? Is this what you do? I wonder >> how >> > > > do you manipulate Java object placement in native code; and if the >> > > > native memory block is out of Java heap, do you expect GC to manage >> > > > it? >> > > > >> > > > You said when GC reclaims those byte buffer object, the native >> memory >> > > > block can be freed in native code. How do you know if an object is >> > > > reclaimed by GC or not? >> > > > >> > > > Thanks, >> > > > xiaofeng >> > > > >> > > > On 2/2/07, Leo Li < liyilei1979@gmail.com> wrote: >> > > > > Hi, all: >> > > > > After applying patch3073, I added the support for direct byte >> buffer >> > > > in >> > > > > nio module as spec requires. But now here exists a problem: The >> native >> > > > > memory allocated to the byte buffer can be released when gc >> releases the >> > > > > byte buffer object, however, if the native heap is full while >> java >> heap >> > > > > still has space, gc will not be triggered. >> > > > > It seems that RI will start gc before native memory heap is >> depleted >> > > > and >> > > > > thus prevents out-of-memory error. >> > > > > >> > > > > Then our work focuses on: >> > > > > 1. When gc is required. >> > > > > 2. Trigger a gc. >> > > > > The first one requires that we get support from operating >> system, >> > > > since >> > > > > the memory allocated in the native code, for example by >> malloc, is >> out >> > > > of >> > > > > the control of java VM.( I have ever thought of counting the used >> memory >> > > > in >> > > > > hymemory_allocate, but the plan fails since hymemory_free will >> not >> > > > report >> > > > > how much space is released.) >> > > > > The second one needs the help from VM. System.gc() is not so >> reliable >> > > > as >> > > > > spec says, so it is necessary to have a internal channel to >> notify >> VM to >> > > > > start gc. >> > > > > >> > > > > One solution, I think, is to let a monitor thread in VM to >> check >> > > > whether >> > > > > OS physical memory is low. For example, the >> > > > QueryMemoryResourceNotification >> > > > > of win32 API is a candidate. Although the interrupt model is more >> > > > effective: >> > > > > win32 SDK provides a CreateMemoryResourceNotification to get a >> handler >> > > > on >> > > > > which the monitor thread can wait, maybe on other platforms, OS >> does not >> > > > > supply such a convenience. So the monitor thread in the VM might >> have to >> > > > >> > > > > check the OS resource once for a while and if necessary the >> monitor >> > > > thread >> > > > > will call a GC. >> > > > > >> > > > > My suggestion is first to add some function to monitor memory >> in >> > > > portlib, >> > > > > since it is highly related to platforms and in portlib there has >> been >> > > > > some useful tools.(Thanks Mark to point out that.) On the other >> hand, we >> > > > can >> > > > > negotiate a channel to trigger gc in VM. Actually I am not an >> expert on >> > > > VM >> > > > > since I am not sure whether there has been some monitor thread >> and >> the >> > > > load >> > > > > on performance if such a monitor thread is added to the VM... >> > > > > >> > > > > >> > > > > -- >> > > > > Leo Li >> > > > > China Software Development Lab, IBM >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > Leo Li >> > > China Software Development Lab, IBM >> > > >> > > >> > >> >> >> -- >> >> Best Regards! >> >> Jimmy, Jing Lv >> China Software Development Lab, IBM >> > > > -- Robin Garner Dept. of Computer Science Australian National University http://cs.anu.edu.au/people/Robin.Garner/