(I'm struggling to see your comment in here Robin ... ?) Robin Garner wrote: > Leo Li wrote: >> Thanks for all the advices. >> Can we conclude that: >> >> 1. The problem is related to a series of problems, as Ivan summarized. >> It is >> about the pattern to release native resources in java. >> 2. It is related to gc, if we do not intend to design another mechanism to >> track object reference. Only when the object cannot be referenced, have we >> the right to release the native resource of it. Actually the problem to >> release native resources can occur in any object besides direct byte buffer >> and zip related objects. >> >> But now we have some design problems: >> 1. When to trigger gc? >> It might be too late when native memory allocation fails. It will cost a >> period of time for VM to complete gc and invoke phantom reference, not to >> mention other resource releases written in finalize method, as Gregory >> mentions. >> 2. The decision will be made on which level? >> System wide, per process, or per VM? >> I think it is the easiest to monitor memory resource on system wide >> level: Normally os will report its free physical memory and total physical >> memory, which we can rely on to deduce a tension in memory allocation. >> Per process is a little difficult:how much memory it used is regarded >> to exist memory shortage? >> Per VM is the most difficult: besides the problem from per process, we >> can hardly discriminate the usage of each vm in one process. The normal >> c-library just tells me the size we allocate in malloc by no message of the >> memory it releases in free. >> >> 3. Is it wise to allocate memory in a managed heap? There is some >> difference >> between Xiao Feng and Jimmy's advice: >> Xiao Feng considers a way to allocate direct byte buffer on java heap. >> It will be great if it works: Then we will have no problem in releasing >> native resources. >> Jimmy proposes we can give each VM a memory booking. It also works, I >> believe. But is it a little too onerous for us to implement a malloc/free >> function ourselves? >> >> Here are my own opinion: >> Although system wide memory monitor can be influnced by other process, I >> prefer it. Base on per process or per vm memory allocation, we can get just >> the information about virtual memory allocation. Hardly can we deduce that >> memory shortage occurs only from the virtual memory one process or vm >> uses : >> by page exchanging, os is able to provide more physical memory for one >> process or another.Native memory allocation failure is the direct result >> from shortage of system physical memory : os cannot commit enough physial >> memory for the required virtual pages. >> >> Since I am not an expert in the area, excuse me if I miss something.:) >> On 2/3/07, LvJimmy,Jing wrote: >>> 2007/2/2, Xiao-Feng Li : >>>> Thank you, Jimmy and Leo, for the clear explanations. >>>> >>>> I think temporarily you can put the System.gc() there to workaround >>>> this issue. Yes, this is only a workaround since we cannot rely on it >>>> without explicit guarantee in its semantics. >>>> >>>> To really solve this problem, we might need a careful design. Your >>>> proposed approach looks like a possible solution. Have you thought to >>>> solve the problem by using GC to manage the raw memory blocks? Without >>>> deep understanding of the problem, I have some wild idea below. Please >>>> see if it is reasonable or stupid. I will read the spec of ByteBuffer >>>> to get more understanding. >>>> >>>> Since what the ByeBuffer needs are starting address and capacity, it >>>> doesn't really care if this piece of memory is in Java heap or in >>>> native runtime memory ( I assume.) We probably can provide some >>>> special kind of virtual Java object that serves as raw memory block to >>>> nio. In this works, we need not monitor the native runtime memory >>>> usage. >>>> >>> Ah, this solution sounds interesting to me, yes, if we malloc memory >>> in heap then we no longer need to monitor the native memory. But we >>> may need a kit of methods offer this heap-memory service: >>> 1) a function of malloc_in_heap() >>> 2) a function of free_heap_block() >>> 3) heap monitor should record this malloc/free action >>> 4) ... >>> However I do think this may be a good solution, and it still depends >>> on how easy can it be implemented. Note not only direct ByteBuffer >>> require this native memory usage. >>> >>>> This approach need certain contract between Java classes and GC about >>>> the special kind of Java object. Probably we can write a layer of Java >>>> class wrapper for raw memory allocation, which hides the contract from >>>> other common classes. >>>> >>> As we have a portlib, this can be implemented in such hymem_malloc or so. >>> >>>> At the same time, I am thinking of your proposed approach, and will >>>> get back later. >>>> >>> Thanks, waiting for you the GC guru's appoach towards heap memory >>> allocation :) >>> >>>> Thanks, >>>> xiaofeng >>>> >>>> On 2/2/07, Leo Li wrote: >>>>> Hi, Xiao-Feng: >>>>> Excuse me if I am confusing you. >>>>> The direct byte buffer holds a block of native memory space used >>> in nio >>>>> operation. The native byte buffer will be freed through the byte >>> buffer is >>>>> gc collected and added to ReferenceQueue monitored by MemorySpy >>> series >>>>> classes. >>>>> But the gc will not be triggered if the java heap is still >>> empty. >>> Here >>>>> is an example: I allocate a large block of native memory for a byte >>> buffer >>>>> while the byte buffer itself is quite small. Thus quickly native heap >>> is >>>>> depleted while VM still does not think gc is needed since java heap >>> has a >>>>> lot of free memory. >>>>> >>>>> >>>>> import java.nio.*; >>>>> public class Test { >>>>> public static void main(String[] args) throws Exception { >>>>> for(int i = 0;i<1000;i++) >>>>> { >>>>> ByteBuffer byteBuffer = ByteBuffer.allocateDirect >>> (10240000); >>>>> //System.gc(); >>>>> } >>>>> } >>>>> } >>>>> RI runs well but Harmony will soon throw OutOfMemory exception. >>>>> But if remove the comment before System.gc(), Harmony will become ok. >>>>> So my point is that we need a mechanism to notify VM to start gc >>> if we >>> have >>>>> no more native memory because sometimes problem will be solved if gc >>> is >>>>> fired. Of cause we cannot avoid a user always malloc space but never >>> free >>>>> them.:) >>>>> >>>>> Good luck! >>>>> >>>>> On 2/2/07, Xiao-Feng Li wrote: >>>>>> Leo, I can't fully understand your problem and proposal. I am >>> working >>>>>> in DRLVM GC component, I want to understand how GC can tackle the >>>>>> issue you meet. >>>>>> >>>>>> You said the native memory is allocated (probably with malloc) for >>>>>> byte buffer. I wonder how the process goes. I assume it's in native >>>>>> code. Then you use malloc to allocate a block of memory, and put >>> some >>>>>> Java object (byte buffer) into it? Is this what you do? I wonder >>> how >>>>>> do you manipulate Java object placement in native code; and if the >>>>>> native memory block is out of Java heap, do you expect GC to manage >>>>>> it? >>>>>> >>>>>> You said when GC reclaims those byte buffer object, the native >>> memory >>>>>> block can be freed in native code. How do you know if an object is >>>>>> reclaimed by GC or not? >>>>>> >>>>>> Thanks, >>>>>> xiaofeng >>>>>> >>>>>> On 2/2/07, Leo Li < liyilei1979@gmail.com> wrote: >>>>>>> Hi, all: >>>>>>> After applying patch3073, I added the support for direct byte >>> buffer >>>>>> in >>>>>>> nio module as spec requires. But now here exists a problem: The >>> native >>>>>>> memory allocated to the byte buffer can be released when gc >>> releases the >>>>>>> byte buffer object, however, if the native heap is full while >>> java >>> heap >>>>>>> still has space, gc will not be triggered. >>>>>>> It seems that RI will start gc before native memory heap is >>> depleted >>>>>> and >>>>>>> thus prevents out-of-memory error. >>>>>>> >>>>>>> Then our work focuses on: >>>>>>> 1. When gc is required. >>>>>>> 2. Trigger a gc. >>>>>>> The first one requires that we get support from operating >>> system, >>>>>> since >>>>>>> the memory allocated in the native code, for example by >>> malloc, is >>> out >>>>>> of >>>>>>> the control of java VM.( I have ever thought of counting the used >>> memory >>>>>> in >>>>>>> hymemory_allocate, but the plan fails since hymemory_free will >>> not >>>>>> report >>>>>>> how much space is released.) >>>>>>> The second one needs the help from VM. System.gc() is not so >>> reliable >>>>>> as >>>>>>> spec says, so it is necessary to have a internal channel to >>> notify >>> VM to >>>>>>> start gc. >>>>>>> >>>>>>> One solution, I think, is to let a monitor thread in VM to >>> check >>>>>> whether >>>>>>> OS physical memory is low. For example, the >>>>>> QueryMemoryResourceNotification >>>>>>> of win32 API is a candidate. Although the interrupt model is more >>>>>> effective: >>>>>>> win32 SDK provides a CreateMemoryResourceNotification to get a >>> handler >>>>>> on >>>>>>> which the monitor thread can wait, maybe on other platforms, OS >>> does not >>>>>>> supply such a convenience. So the monitor thread in the VM might >>> have to >>>>>>> check the OS resource once for a while and if necessary the >>> monitor >>>>>> thread >>>>>>> will call a GC. >>>>>>> >>>>>>> My suggestion is first to add some function to monitor memory >>> in >>>>>> portlib, >>>>>>> since it is highly related to platforms and in portlib there has >>> been >>>>>>> some useful tools.(Thanks Mark to point out that.) On the other >>> hand, we >>>>>> can >>>>>>> negotiate a channel to trigger gc in VM. Actually I am not an >>> expert on >>>>>> VM >>>>>>> since I am not sure whether there has been some monitor thread >>> and >>> the >>>>>> load >>>>>>> on performance if such a monitor thread is added to the VM... >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Leo Li >>>>>>> China Software Development Lab, IBM >>>>>>> >>>>>>> >>>>> >>>>> >>>>> -- >>>>> Leo Li >>>>> China Software Development Lab, IBM >>>>> >>>>> >>> >>> -- >>> >>> Best Regards! >>> >>> Jimmy, Jing Lv >>> China Software Development Lab, IBM >>> >> >> > >