lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: ThreadLocal causing memory leak with J2EE applications
Date Thu, 11 Sep 2008 01:53:32 GMT
Here is SafeThreadLocal. There is a method purge() that can be called  
after a IndexReader is closed. Due to GC, this may not be sufficient,  
better code would look like this:

	SomeLargeObject slo; // maybe a RAMDirectory?
	try {
		slo = new SomeLargeObject(); // or other creation mechanism;
	} catch (OutOfMemoryException e) {
		SafeThreadLocal.purge();
		// now try again
		slo = new SomeLargeObject(); // or other creation mechanism;
	}

		slo = new SomeLargeObject(); // or other creation mechanism;

Even without the above, it is far more aggressive in releasing the  
memory, since any access of the SafeThreadLocal will purge stale  
entries (WeakHashMap functionality).

SafeThreadLocal.java

package org.apache.lucene.util;

import java.lang.ref.WeakReference;
import java.util.*;

/**
  * version of ThreadLocal that is more deterministic in memory usage
  */
public class SafeThreadLocal {
     private final static Set locals = new HashSet();
     private final Map values = new WeakHashMap();

     public SafeThreadLocal() {
	synchronized(SafeThreadLocal.class) {
	    locals.add(new WeakReference(this));
	}
	// maybe call purge() here ? makes things slower, but a bit safer
     }

     protected Object initialValue() {
         return null;
     }

     public final synchronized Object get() {
         final Object key = Thread.currentThread();
         Object value;
         if(!values.containsKey(key)) {
             value = initialValue();
             values.put(key,value);
         } else {
             value = values.get(key);
         }
         return value;
     }

     public final synchronized void set(Object value) {
         values.put(Thread.currentThread(),value);
     }

     /**
      * clear any stale entries across all thread locals
      */
     public static synchronized void purge() {
	for(Iterator i = locals.iterator();i.hasNext();) {
	    SafeThreadLocal sfl = (SafeThreadLocal) ((WeakReference)i.next 
()).get();
	    if(sfl==null)
		i.remove();
	    else {
		synchronized(sfl) {
		    sfl.locals.size(); // causes stale entries to be purged
		}
	    }
	}
     }
}

		

On Sep 10, 2008, at 12:43 PM, Chris Lu wrote:

> SafeThreadLocal is very interesting. It'll be good not only for  
> Lucene, but also other projects.
>
> Could you please post it?
>
> -- 
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
> index.php?title=Create_Lucene_Database_Search_in_3_minutes
> DBSight customer, a shopping comparison site, (anonymous per  
> request) got 2.6 Million Euro funding!
>
>
> On Wed, Sep 10, 2008 at 9:41 AM, robert engels  
> <rengels@ix.netcom.com> wrote:
> The other thing Lucene can do is create a SafeThreadLocal - it is  
> rather trivial, and have that integrate at a higher-level, allowing  
> for manual clean-up across all threads.
>
> It MIGHT  be a bit slower than the JDK version (since that uses  
> heuristics to clear stale entries), and so doesn't always clear.
>
> But it will be far more deterministic.
>
> If someone is interested I can post the class, but I think it is  
> well within the understanding of the core Lucene developers.
>
>
> On Sep 10, 2008, at 11:10 AM, robert engels wrote:
>
>> You do not need to create a new RAMDirectory - just write to the  
>> existing one, and then reopen() the IndexReader using it.
>>
>> This will prevent lots of big objects being created. This may be  
>> the source of your problem.
>>
>> Even if the Segment is closed, the ThreadLocal will no longer be  
>> referenced, but there will still be a reference to the  
>> SegmentTermEnum (which will be cleared when the thread dies, or  
>> "most likely" when new thread locals on that thread a created, so  
>> here is a potential problem.
>>
>> Thread 1 does a search, creates a thread local that references the  
>> RAMDir (A).
>> Thread 2 does a search, creates a thread local that references the  
>> RAMDir (A).
>>
>> All readers, are closed on RAMDir (A).
>>
>> A new RAMDir (B) is opened.
>>
>> There may still be references in the thread local maps to RAMDir A  
>> (since no new thread local have been created yet).
>>
>> So you may get OOM depending on the size of the RAMDir (since you  
>> would need room for more than 1).  If you extend this out with  
>> lots of threads that don't run very often, you can see how you  
>> could easily run out of memory.  "I think" that ThreadLocal should  
>> use a ReferenceQueue so stale object slots can be reclaimed as  
>> soon as the key is dereferenced - but that is an issue for SUN.
>>
>> This is why you don't want to create new RAMDirs.
>>
>> A good rule of thumb - don't keep references to large objects in  
>> ThreadLocal (especially indirectly).  If needed, use a "key", and  
>> then read the cache using a the "key".
>> This would be something for the Lucene folks to change.
>>
>> On Sep 10, 2008, at 10:44 AM, Chris Lu wrote:
>>
>>> I am really want to find out where I am doing wrong, if that's  
>>> the case.
>>>
>>> Yes. I have made certain that I closed all Readers/Searchers, and  
>>> verified that through memory profiler.
>>>
>>> Yes. I am creating new RAMDirectory. But that's the problem. I  
>>> need to update the content. Sure, if no content update and  
>>> everything the same, of course no OOM.
>>>
>>> Yes. No guarantee of the thread schedule. But that's the problem.  
>>> If Lucene is using ThreadLocal to cache lots of things by the  
>>> Thread as the key, and no idea when it'll be released. Of course  
>>> ThreadLocal is not Lucene's problem...
>>>
>>> Chris
>>>
>>> On Wed, Sep 10, 2008 at 8:34 AM, robert engels  
>>> <rengels@ix.netcom.com> wrote:
>>> It is basic Java. Threads are not guaranteed to run on any sort  
>>> of schedule. If you create lots of large objects in one thread,  
>>> releasing them in another, there is a good chance you will get an  
>>> OOM (since the releasing thread may not run before the OOM  
>>> occurs)...  This is not Lucene specific by any means.
>>>
>>> It is a misunderstanding on your part about how GC works.
>>>
>>> I assume you must at some point be creating new RAMDirectories -  
>>> otherwise the memory would never really increase, since the  
>>> IndexReader/enums/etc are not very large...
>>>
>>> When you create a new RAMDirectories, you need to BE CERTAIN !!!  
>>> that the other IndexReaders/Searchers using the old RAMDirectory  
>>> are ALL CLOSED, otherwise their memory will still be in use,  
>>> which leads to your OOM...
>>>
>>>
>>> On Sep 10, 2008, at 10:16 AM, Chris Lu wrote:
>>>
>>>> I do not believe I am making any mistake. Actually I just got an  
>>>> email from another user, complaining about the same thing. And I  
>>>> am having the same usage pattern.
>>>>
>>>> After the reader is opened, the RAMDirectory is shared by  
>>>> several objects.
>>>> There is one instance of RAMDirectory in the memory, and it is  
>>>> holding lots of memory, which is expected.
>>>>
>>>> If I close the reader in the same thread that has opened it, the  
>>>> RAMDirectory is gone from the memory.
>>>> If I close the reader in other threads, the RAMDirectory is left  
>>>> in the memory, referenced along the tree I draw in the first email.
>>>>
>>>> I do not think the usage is wrong. Period.
>>>>
>>>> -------------------------------------
>>>> Hi,
>>>>
>>>>    i found a forum post from you here [1] where you mention that  
>>>> you
>>>> have a memory leak using the lucene ram directory. I'd like to  
>>>> ask you
>>>> if you already have resolved the problem and how you did it or  
>>>> maybe
>>>> you know where i can read about the solution. We are using
>>>> RAMDirectory too and figured out, that over time the memory
>>>> consumption raises and raises until the system breaks down but only
>>>> when we performing much index updates. if we only create the  
>>>> index and
>>>> don't do nothing except searching it, it work fine.
>>>>
>>>> maybe you can give me a hint or a link,
>>>> greetz,
>>>> -------------------------------------
>>>>
>>>> -- 
>>>> Chris Lu
>>>> -------------------------
>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>> site: http://www.dbsight.net
>>>> demo: http://search.dbsight.com
>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>> DBSight customer, a shopping comparison site, (anonymous per  
>>>> request) got 2.6 Million Euro funding!
>>>>
>>>> On Wed, Sep 10, 2008 at 7:12 AM, robert engels  
>>>> <rengels@ix.netcom.com> wrote:
>>>> Sorry, but I am fairly certain you are mistaken.
>>>>
>>>> If you only have a single IndexReader, the RAMDirectory will be  
>>>> shared in all cases.
>>>>
>>>> The only memory growth is any buffer space allocated by an  
>>>> IndexInput (used in many places and cached).
>>>>
>>>> Normally the IndexInput created by a RAMDirectory do not have  
>>>> any buffer allocated, since the underlying store is already in  
>>>> memory.
>>>>
>>>> You have some other problem in your code...
>>>>
>>>> On Sep 10, 2008, at 1:10 AM, Chris Lu wrote:
>>>>
>>>>> Actually, even I only use one IndexReader, some resources are  
>>>>> cached via the ThreadLocal cache, and can not be released  
>>>>> unless all threads do the close action.
>>>>>
>>>>> SegmentTermEnum itself is small, but it holds RAMDirectory  
>>>>> along the path, which is big.
>>>>>
>>>>> -- 
>>>>> Chris Lu
>>>>> -------------------------
>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>> site: http://www.dbsight.net
>>>>> demo: http://search.dbsight.com
>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>> DBSight customer, a shopping comparison site, (anonymous per  
>>>>> request) got 2.6 Million Euro funding!
>>>>>
>>>>> On Tue, Sep 9, 2008 at 10:43 PM, robert engels  
>>>>> <rengels@ix.netcom.com> wrote:
>>>>> You do not need a pool of IndexReaders...
>>>>>
>>>>> It does not matter what class it is, what matters is the class  
>>>>> that ultimately holds the reference.
>>>>>
>>>>> If the IndexReader is never closed, the SegmentReader(s) is  
>>>>> never closed, so the thread local in TermInfosReader is not  
>>>>> cleared (because the thread never dies). So you will get one  
>>>>> SegmentTermEnum, per thread * per segment.
>>>>>
>>>>> The SegmentTermEnum is not a large object, so even if you had  
>>>>> 100 threads, and 100 segments, for 10k instances, seems hard to  
>>>>> believe that is the source of your memory issue.
>>>>>
>>>>> The SegmentTermEnum is cached by thread since it needs to  
>>>>> enumerate the terms, not having a per thread cache, would lead  
>>>>> to lots of random access when multiple threads read the index -  
>>>>> very slow.
>>>>>
>>>>> You need to keep in mind, what if every thread was executing a  
>>>>> search simultaneously - you would still have 100x100  
>>>>> SegmentTermEnum instances anyway !  The only way to prevent  
>>>>> that would be to create and destroy the SegmentTermEnum on each  
>>>>> call (opening and seeking to the proper spot) - which would be  
>>>>> SLOW SLOW SLOW.
>>>>>
>>>>> On Sep 10, 2008, at 12:19 AM, Chris Lu wrote:
>>>>>
>>>>>> I have tried to create an IndexReader pool and dynamically  
>>>>>> create searcher. But the memory leak is the same. It's not  
>>>>>> related to the Searcher class specifically, but the  
>>>>>> SegmentTermEnum in TermInfosReader.
>>>>>>
>>>>>> -- 
>>>>>> Chris Lu
>>>>>> -------------------------
>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>> site: http://www.dbsight.net
>>>>>> demo: http://search.dbsight.com
>>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
>>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>> DBSight customer, a shopping comparison site, (anonymous per  
>>>>>> request) got 2.6 Million Euro funding!
>>>>>>
>>>>>> On Tue, Sep 9, 2008 at 10:14 PM, robert engels  
>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>> A searcher uses an IndexReader - the IndexReader is slow to  
>>>>>> open, not a Searcher. And searchers can share an IndexReader.
>>>>>>
>>>>>> You want to create a single shared (across all threads/users)  
>>>>>> IndexReader (usually), and create an Searcher as needed and  
>>>>>> dispose.  It is VERY CHEAP to create the Searcher.
>>>>>>
>>>>>> I am fairly certain the javadoc on Searcher is incorrect.  The  
>>>>>> warning "For performance reasons it is recommended to open  
>>>>>> only one IndexSearcher and use it for all of your searches" is  
>>>>>> not true in the case where an IndexReader is passed to the ctor.
>>>>>>
>>>>>> Any caching should USUALLY be performed at the IndexReader level.
>>>>>>
>>>>>> You are most likely using the "path" ctor, and that is the  
>>>>>> source of your problems, as multiple IndexReader instances are  
>>>>>> being created, and thus the memory use.
>>>>>>
>>>>>>
>>>>>> On Sep 9, 2008, at 11:44 PM, Chris Lu wrote:
>>>>>>
>>>>>>> On J2EE environment, usually there is a searcher pool with  
>>>>>>> several searchers open.
>>>>>>> The speed to opening a large index for every user is not  
>>>>>>> acceptable.
>>>>>>>
>>>>>>> -- 
>>>>>>> Chris Lu
>>>>>>> -------------------------
>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>> site: http://www.dbsight.net
>>>>>>> demo: http://search.dbsight.com
>>>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/

>>>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>> DBSight customer, a shopping comparison site, (anonymous per
 
>>>>>>> request) got 2.6 Million Euro funding!
>>>>>>>
>>>>>>> On Tue, Sep 9, 2008 at 9:03 PM, robert engels  
>>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>>> You need to close the searcher within the thread that is  
>>>>>>> using it, in order to have it cleaned up quickly... usually 

>>>>>>> right after you display the page of results.
>>>>>>>
>>>>>>> If you are keeping multiple searcher refs across multiple  
>>>>>>> threads for paging/whatever, you have not coded it correctly.
>>>>>>>
>>>>>>> Imagine 10,000 users - storing a searcher for each one is not
 
>>>>>>> going to work...
>>>>>>>
>>>>>>> On Sep 9, 2008, at 10:21 PM, Chris Lu wrote:
>>>>>>>
>>>>>>>> Right, in a sense I can not release it from another thread.
 
>>>>>>>> But that's the problem.
>>>>>>>>
>>>>>>>> It's a J2EE environment, all threads are kind of equal. It's
 
>>>>>>>> simply not possible to iterate through all threads to close
 
>>>>>>>> the searcher, thus releasing the ThreadLocal cache.
>>>>>>>> Unless Lucene is not recommended for J2EE environment, this
 
>>>>>>>> has to be fixed.
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Chris Lu
>>>>>>>> -------------------------
>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>>> site: http://www.dbsight.net
>>>>>>>> demo: http://search.dbsight.com
>>>>>>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/

>>>>>>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>> DBSight customer, a shopping comparison site, (anonymous
per  
>>>>>>>> request) got 2.6 Million Euro funding!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 9, 2008 at 8:14 PM, robert engels  
>>>>>>>> <rengels@ix.netcom.com> wrote:
>>>>>>>> Your code is not correct. You cannot release it on another
 
>>>>>>>> thread - the first thread may creating hundreds/thousands
of  
>>>>>>>> instances before the other thread ever runs...
>>>>>>>>
>>>>>>>> On Sep 9, 2008, at 10:10 PM, Chris Lu wrote:
>>>>>>>>
>>>>>>>>> If I release it on the thread that's creating the searcher,
 
>>>>>>>>> by setting searcher=null, everything is fine, the memory
is  
>>>>>>>>> released very cleanly.
>>>>>>>>> My load test was to repeatedly create a searcher on a
 
>>>>>>>>> RAMDirectory and release it on another thread. The test
 
>>>>>>>>> will quickly go to OOM after several runs. I set the
heap  
>>>>>>>>> size to be 1024M, and the RAMDirectory is of size 250M.
 
>>>>>>>>> Using some profiling tool, the used size simply stepped
up  
>>>>>>>>> pretty obviously by 250M.
>>>>>>>>>
>>>>>>>>> I think we should not rely on something that's a "maybe"
 
>>>>>>>>> behavior, especially for a general purpose library.
>>>>>>>>>
>>>>>>>>> Since it's a multi-threaded env, the thread that's creating
 
>>>>>>>>> the entries in the LRU cache may not go away quickly

>>>>>>>>> (actually most, if not all, application servers will
try to  
>>>>>>>>> reuse threads), so the LRU cache, which uses thread as
the  
>>>>>>>>> key, can not be released, so the SegmentTermEnum which
is  
>>>>>>>>> in the same class can not be released.
>>>>>>>>>
>>>>>>>>> And yes, I close the RAMDirectory, and the fileMap is
 
>>>>>>>>> released. I verified that through the profiler by directly
 
>>>>>>>>> checking the values in the snapshot.
>>>>>>>>>
>>>>>>>>> Pretty sure the reference tree wasn't like this using
code  
>>>>>>>>> before this commit, because after close the searcher
in  
>>>>>>>>> another thread, the RAMDirectory totally disappeared
from  
>>>>>>>>> the memory snapshot.
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Chris Lu
>>>>>>>>> -------------------------
>>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>>>> site: http://www.dbsight.net
>>>>>>>>> demo: http://search.dbsight.com
>>>>>>>>> Lucene Database Search in 3 minutes: http:// 
>>>>>>>>> wiki.dbsight.com/index.php? 
>>>>>>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>>> DBSight customer, a shopping comparison site, (anonymous
 
>>>>>>>>> per request) got 2.6 Million Euro funding!
>>>>>>>>>
>>>>>>>>> On Tue, Sep 9, 2008 at 5:03 PM, Michael McCandless  
>>>>>>>>> <lucene@mikemccandless.com> wrote:
>>>>>>>>>
>>>>>>>>> Chris Lu wrote:
>>>>>>>>>
>>>>>>>>> The problem should be similar to what's talked about
on  
>>>>>>>>> this discussion.
>>>>>>>>> http://lucene.markmail.org/message/keosgz2c2yjc7qre?

>>>>>>>>> q=ThreadLocal
>>>>>>>>>
>>>>>>>>> The "rough" conclusion of that thread is that, technically,
 
>>>>>>>>> this isn't a memory leak but rather a "delayed freeing"
 
>>>>>>>>> problem.  Ie, it may take longer, possibly much longer,
 
>>>>>>>>> than you want for the memory to be freed.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There is a memory leak for Lucene search from Lucene-1195.

>>>>>>>>> (svn r659602, May23,2008)
>>>>>>>>>
>>>>>>>>> This patch brings in a ThreadLocal cache to TermInfosReader.
>>>>>>>>>
>>>>>>>>> One thing that confuses me: TermInfosReader was already
 
>>>>>>>>> using a ThreadLocal to cache the SegmentTermEnum instance.
  
>>>>>>>>> What was added in this commit (for LUCENE-1195) was an
LRU  
>>>>>>>>> cache storing Term -> TermInfo instances.  But it
seems  
>>>>>>>>> like it's the SegmentTermEnum instance that you're tracing
 
>>>>>>>>> below.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It's usually recommended to keep the reader open, and
reuse  
>>>>>>>>> it when
>>>>>>>>> possible. In a common J2EE application, the http requests
 
>>>>>>>>> are usually
>>>>>>>>> handled by different threads. But since the cache is
 
>>>>>>>>> ThreadLocal, the cache
>>>>>>>>> are not really usable by other threads. What's worse,
the  
>>>>>>>>> cache can not be
>>>>>>>>> cleared by another thread!
>>>>>>>>>
>>>>>>>>> This leak is not so obvious usually. But my case is using
 
>>>>>>>>> RAMDirectory,
>>>>>>>>> having several hundred megabytes. So one un-released
 
>>>>>>>>> resource is obvious to
>>>>>>>>> me.
>>>>>>>>>
>>>>>>>>> Here is the reference tree:
>>>>>>>>> org.apache.lucene.store.RAMDirectory
>>>>>>>>>  |- directory of org.apache.lucene.store.RAMFile
>>>>>>>>>     |- file of org.apache.lucene.store.RAMInputStream
>>>>>>>>>         |- base of  
>>>>>>>>> org.apache.lucene.index.CompoundFileReader$CSIndexInput
>>>>>>>>>             |- input of  
>>>>>>>>> org.apache.lucene.index.SegmentTermEnum
>>>>>>>>>                 |- value of java.lang.ThreadLocal 
>>>>>>>>> $ThreadLocalMap$Entry
>>>>>>>>>
>>>>>>>>> So you have a RAMDir that has several hundred MB stored
in  
>>>>>>>>> it, that you're done with yet through this path Lucene
is  
>>>>>>>>> keeping it alive?
>>>>>>>>>
>>>>>>>>> Did you close the RAMDir?  (which will null its fileMap
and  
>>>>>>>>> should also free your memory).
>>>>>>>>>
>>>>>>>>> Also, that reference tree doesn't show the ThreadResources
 
>>>>>>>>> class that was added in that commit -- are you sure this
 
>>>>>>>>> reference tree wasn't before the commit?
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> --------------------------------------------------------------

>>>>>>>>> -------
>>>>>>>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>>>>>>>> For additional commands, e-mail: java-dev- 
>>>>>>>>> help@lucene.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -- 
>>>>>>>>> Chris Lu
>>>>>>>>> -------------------------
>>>>>>>>> Instant Scalable Full-Text Search On Any Database/Application
>>>>>>>>> site: http://www.dbsight.net
>>>>>>>>> demo: http://search.dbsight.com
>>>>>>>>> Lucene Database Search in 3 minutes: http:// 
>>>>>>>>> wiki.dbsight.com/index.php? 
>>>>>>>>> title=Create_Lucene_Database_Search_in_3_minutes
>>>>>>>>> DBSight customer, a shopping comparison site, (anonymous
 
>>>>>>>>> per request) got 2.6 Million Euro funding!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> -- 
>>> Chris Lu
>>> -------------------------
>>> Instant Scalable Full-Text Search On Any Database/Application
>>> site: http://www.dbsight.net
>>> demo: http://search.dbsight.com
>>> Lucene Database Search in 3 minutes: http://wiki.dbsight.com/ 
>>> index.php?title=Create_Lucene_Database_Search_in_3_minutes
>>> DBSight customer, a shopping comparison site, (anonymous per  
>>> request) got 2.6 Million Euro funding!
>>
>
>
>
>


Mime
View raw message