camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: [DISCUSS] - Camel 2.0 - About Endpoints with lenient properties eating to much memory
Date Wed, 01 Jul 2009 04:37:04 GMT
On Tue, Jun 30, 2009 at 1:50 PM, James Strachan<james.strachan@gmail.com> wrote:
> I'm liking a) and allowing c) (a single mbean for all endpoints of a
> certain type) make sense to me.
Just for the record. Option a + c is currently implemented and
reported as fixed the issues by the end user.


>
> Using an LRU cache for non-singleton endpoints makes sense too; though
> I wonder if an endpoint might want to mark itself as not cachable if
> it knows its not really worth the trouble (as it has no state?).
>
>
> 2009/6/30 Willem Jiang <willem.jiang@gmail.com>:
>> Hi Claus,
>>
>> I think we need to implements a method for get unique name from the
>> endpoint. It will base on the endpoint's parameter not the customer
>> parameter. With this method we could avoid to register to much temp endpoint
>> with different customer parameter.
>>
>> In this case we need find a way to trim the customer parameter from the
>> endpoint URI. If we could do that, it will save us lots room for the store
>> the fake temp http endpoints (we could share the endpoints in the
>> ProducerCache/ConsumerCache with different customer parameter) and make the
>> LRUCache more effective.
>>
>> Just my two cents.
>>
>> Willem
>>
>> Claus Ibsen wrote:
>>>
>>> Hi
>>>
>>> Tickets
>>> ======
>>> I am looking into two issues related to Camel eating memory
>>> https://issues.apache.org/activemq/browse/CAMEL-1771
>>> https://issues.apache.org/activemq/browse/CAMEL-1738
>>>
>>>
>>> The problem
>>> =========
>>> It all boils down to using a lot of http endpoints with unique urls.
>>> So over time Camel accumulate a lot of created endpoints in its
>>> internal endpoint registry and as well as JMX Beans
>>> That consumes memory, and for people with millions unique endpoints
>>> over time that consumes to much memory
>>>
>>>
>>> Solutions
>>> =======
>>> To remedy this I have addressed in two areas
>>>
>>> a) failsafe with a limited size of 1000 entries
>>> - Using the LRUCache for ProducerCache/ConsumerCache (verified by end
>>> user that its much better)
>>> - Using the LRUCache in CamelContext for its endpoint reigstry
>>>
>>> I assume having more than 1000 unique endpoints, producers or
>>> consumers is not within normal usage for a given CamelContext?
>>> And it does not bring harm as Camel will just recreate the object if
>>> not already cached.
>>>
>>>
>>> b)
>>> - Not registering endpoints with lenient properties in JMX or camel
>>> context. That is if only they use properties that Camel does not know
>>> about (= lenient properties)
>>> as these endpoints is highly not reusable and short lived. And its
>>> only a few endpoints that support lenient properties (http, restlet,
>>> cxf, atom, rss)
>>>
>>> So if you setup an endpoint with custom parameters (not Camel options)
>>> then its a lenient property and Camel will not cache/register it.
>>> For example:
>>>
>>> http://myserver/mypath?id=1
>>> http://myserver/mypath?id=2
>>> http://myserver/mypath?id=3
>>> ...
>>>
>>> where id is a lenient property that is not a Camel option.
>>> So these will not be registered.
>>>
>>>
>>> But if you use
>>>    http://myserver/mypath
>>> then it will be registered as its does not contain any lenient
>>> properties at all.
>>>
>>>
>>> c)
>>> - Only registering a single JMX bean.
>>> Otherwise the JMX registry will be cluttered with millions endpoints.
>>> So what we do not is to register the same parent endpoint for the
>>> endpoints.
>>> We use the new getEndpointKey() on Endpoint to provide the key to use
>>> for JMX registration. This allows us to provide the same key for all
>>> the http endpoints.
>>>
>>>
>>> Questions
>>> ========
>>> 1)
>>> Option a and c is already nearly done. I am running further unit tests
>>> and do a bit of code polish.
>>>
>>> 2)
>>> As option b is a bit controversial, I am wondering if that is
>>> feasible. Should we just go ahead with the LRUCache being able to
>>> filter out the eldest endpoints?
>>> And I wonder if people just create a few endpoints with lenient
>>> properties then we still have "room" to store them in the cache so I
>>> wonder if we should do this at all?
>>> I do think we could keep this as a thought for the future, and just
>>> let the LRUCache handle it, so when people use millions of unqiue http
>>> endpoints we let the chace
>>> filter out the not used ones.
>>>
>>>
>>>
>>>
>>
>>
>
>
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/
>



-- 
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message