camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Willem Jiang <willem.ji...@gmail.com>
Subject Re: [DISCUSS] - Camel 2.0 - About Endpoints with lenient properties eating to much memory
Date Tue, 30 Jun 2009 09:15:28 GMT
Hi Claus,

I think we need to implements a method for get unique name from the 
endpoint. It will base on the endpoint's parameter not the customer 
parameter. With this method we could avoid to register to much temp 
endpoint with different customer parameter.

In this case we need find a way to trim the customer parameter from the 
endpoint URI. If we could do that, it will save us lots room for the 
store the fake temp http endpoints (we could share the endpoints in the 
ProducerCache/ConsumerCache with different customer parameter) and make 
the LRUCache more effective.

Just my two cents.

Willem

Claus Ibsen wrote:
> Hi
> 
> Tickets
> ======
> I am looking into two issues related to Camel eating memory
> https://issues.apache.org/activemq/browse/CAMEL-1771
> https://issues.apache.org/activemq/browse/CAMEL-1738
> 
> 
> The problem
> =========
> It all boils down to using a lot of http endpoints with unique urls.
> So over time Camel accumulate a lot of created endpoints in its
> internal endpoint registry and as well as JMX Beans
> That consumes memory, and for people with millions unique endpoints
> over time that consumes to much memory
> 
> 
> Solutions
> =======
> To remedy this I have addressed in two areas
> 
> a) failsafe with a limited size of 1000 entries
> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end
> user that its much better)
> - Using the LRUCache in CamelContext for its endpoint reigstry
> 
> I assume having more than 1000 unique endpoints, producers or
> consumers is not within normal usage for a given CamelContext?
> And it does not bring harm as Camel will just recreate the object if
> not already cached.
> 
> 
> b)
> - Not registering endpoints with lenient properties in JMX or camel
> context. That is if only they use properties that Camel does not know
> about (= lenient properties)
> as these endpoints is highly not reusable and short lived. And its
> only a few endpoints that support lenient properties (http, restlet,
> cxf, atom, rss)
> 
> So if you setup an endpoint with custom parameters (not Camel options)
> then its a lenient property and Camel will not cache/register it.
> For example:
> 
> http://myserver/mypath?id=1
> http://myserver/mypath?id=2
> http://myserver/mypath?id=3
> ...
> 
> where id is a lenient property that is not a Camel option.
> So these will not be registered.
> 
> 
> But if you use
>     http://myserver/mypath
> then it will be registered as its does not contain any lenient
> properties at all.
> 
> 
> c)
> - Only registering a single JMX bean.
> Otherwise the JMX registry will be cluttered with millions endpoints.
> So what we do not is to register the same parent endpoint for the
> endpoints.
> We use the new getEndpointKey() on Endpoint to provide the key to use
> for JMX registration. This allows us to provide the same key for all
> the http endpoints.
> 
> 
> Questions
> ========
> 1)
> Option a and c is already nearly done. I am running further unit tests
> and do a bit of code polish.
> 
> 2)
> As option b is a bit controversial, I am wondering if that is
> feasible. Should we just go ahead with the LRUCache being able to
> filter out the eldest endpoints?
> And I wonder if people just create a few endpoints with lenient
> properties then we still have "room" to store them in the cache so I
> wonder if we should do this at all?
> I do think we could keep this as a thought for the future, and just
> let the LRUCache handle it, so when people use millions of unqiue http
> endpoints we let the chace
> filter out the not used ones.
> 
> 
> 
> 


Mime
View raw message