camel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Components setting data on OUT
Date Tue, 27 Jan 2009 06:23:25 GMT
On Tue, Jan 27, 2009 at 6:02 AM, William Tam <email.wtam@gmail.com> wrote:
> On Mon, Jan 26, 2009 at 11:34 PM, Hadrian Zbarcea <hzbarcea@gmail.com> wrote:
>> I think having dots ('.') in property names is not a good idea as they don't
>> get along iirc with some technologies.
>>
>
> Sorry one more thing, the property name is just key to exchange
> property map.  I believe we already use dots in exchange property
> names in some places today.
>
We have, in the Camel 2.0 design
(http://camel.apache.org/camel-20-design.html) a bullet, that we
should resolve the header key name issue.
James suggested to use a new naming convention CamelComponentName that I like.
We should really remove all dots and usage with long packagenames etc.
as the name names dont apply with JMS spec and some other transports.

And I think its something we should do prior to 2.0 release as its a
change that affects many components and internal parts in Camel.

I am glad that I kinda started this ball, this thread sure has proven
that we needed a little heads up and consensus on the
header/properties issue in Camel. Keep the ball rolling.

This is early morning and I am not juiced up for long and deep thoughts :)


>> Secondly, my point was that at this time, I am against the idea of
>> separating the headers into two maps.  I don't see a compelling reason to do
>> so and we should properly use properties as Roman and I think Claus
>> suggested.  I'm confident that we'll find a good way to do it.
>>
>> Hadrian
>>
>>
>> On Jan 26, 2009, at 11:09 PM, William Tam wrote:
>>
>>> On Mon, Jan 26, 2009 at 5:34 PM, Hadrian Zbarcea <hzbarcea@gmail.com>
>>> wrote:
>>>>
>>>> On Jan 26, 2009, at 4:17 PM, William Tam wrote:
>>>>
>>>>> On Mon, Jan 26, 2009 at 1:37 PM, Hadrian Zbarcea <hzbarcea@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> I don't disagree, I was just suggesting that they should then travel
as
>>>>>> properties.
>>>>>
>>>>> Why shouldn't they (protocol headers) travel as headers (to avoids
>>>>> unnecessary copying between exchange properties and protocol headers)?
>>>>
>>>> There should be no unnecessary copying, I agree.  If a header (such as
>>>> username/password) has broadly known and accepted semantics, the endpoint
>>>> or
>>>> associated policy could set it as a property from start.  Similarly, an
>>>> endpoint should look at both properties and headers when sending a
>>>> message.
>>>
>>> I think I finally get it.   Your example of username/password are
>>> really properties in my view.   E.g, we can have a property
>>> org.apache.camel.headers.username.  It is broadly known in Camel.
>>> Sure, we can stick it in exchange properties and let endpoint read it
>>> and create a protocol header with it (like, USER=foo).  I am fine with
>>> that idea.  That does not make org.apache.camel.headers.username=foo a
>>> protocol header, though.   It really isn't.
>>>
>>> I am concerned about true custom protocol header that is totally
>>> unknown to Camel (say, foo=bar).  I want my app to be able to define
>>> and insert some header in protocol message header.  I argue that
>>> foo=bar should be set in the header not in exchange properties.
>>>
>>>>
>>>>>
>>>>>
>>>>>> Whatever we name them, and whatever mechanism we decide to use,
>>>>>> as pointed out before, we need to distinguish between headers that
are
>>>>>> endpoint/protocol specific and have no semantics outside the endpoint
>>>>>> and
>>>>>> headers (which we called properties and didn't use consistently)
that
>>>>>> must
>>>>>> be carried over the lifetime of the Exchange.
>>>>>
>>>>> Custom protocol header fits into "no semantics outside the endpoint"
>>>>> (or else it won't get propagated) and so it goes to the exchange
>>>>> properties.   How does the next component (in the pipeline) know what
>>>>> one of the properties are intended to be sent in protocol header?
>>>>> Wouldn't it be better to let protocol headers travel as headers so
>>>>> only the "header candidates" will be potentially be sent?
>>>>
>>>> Not sure I understand your question, but I think we should use policies
>>>> for
>>>> handling custom headers.
>>>
>>> Sorry about the typos and grammars.  All I am saying is if we put user
>>> defined header in message headers, filter strategy (or policy if you
>>> will) won't have to iterate through the exchange properties for it.
>>> It is already in headers where it belongs.  All we need to do is to
>>> apply filters to headers.  It is much cleaner.  In your example, if
>>> endpoint wants to read a specific property (e.g.
>>> org.apache.camel.headers.username) and make a header out of a
>>> property, it can certainly do so.
>>>
>>> (However, I am not sure there are really that many Camel well known
>>> properties at least in the presence.)
>>>
>>>
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Hadrian
>>>>>>
>>>>>>
>>>>>> On Jan 26, 2009, at 1:07 PM, William Tam wrote:
>>>>>>
>>>>>>> On Mon, Jan 26, 2009 at 10:35 AM, Hadrian Zbarcea <hzbarcea@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This headers business is a bit of a tricky one.  I hit it
last year
>>>>>>>> in
>>>>>>>> the
>>>>>>>> context of security.
>>>>>>>>
>>>>>>>> I agree with the view that headers should only exist in the
context
>>>>>>>> of
>>>>>>>> an
>>>>>>>> endpoint.  I think outside of that there is no guarantee
that the
>>>>>>>> semantics
>>>>>>>> of a header is preserved.  I am not sure if headers should
be
>>>>>>>> propagated
>>>>>>>> from one endpoint to another at all.
>>>>>>>
>>>>>>> There are certainly use cases that protocol headers DO need to
be
>>>>>>> propagated between endpoints.  If users want to integrate with
some
>>>>>>> management and/or security tools like Actional, users are required
to
>>>>>>> include custom headers in protocol headers.  These custom headers
>>>>>>> travel with messages to allow trust zone enforcement and message
>>>>>>> correlation.  They need to be preserved and propagated across
hops
>>>>>>> which are potentially over different transport protocols.
>>>>>>>
>>>>>>>> Properties should be used instead.
>>>>>>>> Coming back to security, if http is used for instance there
are
>>>>>>>> several
>>>>>>>> ways
>>>>>>>> of handling that.  If basic auth is used for instance one
gets a
>>>>>>>> user/pass,
>>>>>>>> but that may need to be translated to something else at endpoint
>>>>>>>> boundaries.
>>>>>>>> I don't think that the "Authorization" header should exist
outside
>>>>>>>> the
>>>>>>>> http
>>>>>>>> endpoint for instance.
>>>>>>>>
>>>>>>>> Yes, we do propagate properties today, no issue there.  But
then some
>>>>>>>> policies need to be defined per endpoint to deal with known
>>>>>>>> headers/properties, and camel specific properties should
be defined
>>>>>>>> to
>>>>>>>> deal
>>>>>>>> with know headers.  Even better, endpoints should set protocol
>>>>>>>> specific
>>>>>>>> headers that are known as required to propagate (such as
the auth
>>>>>>>> stuff)
>>>>>>>> as
>>>>>>>> properties from start.
>>>>>>>>
>>>>>>>> My $0.02
>>>>>>>> Hadrian
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 26, 2009, at 9:44 AM, Claus Ibsen wrote:
>>>>>>>>
>>>>>>>>> On Mon, Jan 26, 2009 at 3:37 PM, Roman Kalukiewicz
>>>>>>>>> <roman.kalukiewicz@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Why don't we talk about exchange properties here?
My feeling here
>>>>>>>>>> is
>>>>>>>>>> that properties should be used as user-headers, while
headers are
>>>>>>>>>> always protocol headers. In fact it works this way
right now: If I
>>>>>>>>>> want to keep some value through the whole flow I
put it into
>>>>>>>>>> properties.
>>>>>>>>>>
>>>>>>>>>> By current convention if I put something on a header
it is sent as
>>>>>>>>>> protocol-specific header (JMS property, HTTP header),
and out
>>>>>>>>>> headers
>>>>>>>>>> are filled also with protocol headers (JSM properties
of out
>>>>>>>>>> emssage,
>>>>>>>>>> HTTP response headers). In this case headers shouldn't
be
>>>>>>>>>> propagated,
>>>>>>>>>> as there is no way to distinguish things propagated,
from things
>>>>>>>>>> retrieved. And out headers ARE different than in
headers.
>>>>>>>>>>
>>>>>>>>>> It is a matter of naming, but currently headers are
(what you call)
>>>>>>>>>> protocol/system headers, while properties are user-headers
(work as
>>>>>>>>>> variables). Do we really need to extend it further?
If someone mix
>>>>>>>>>> those two concepts then it is problem of documentation,
but not
>>>>>>>>>> lack
>>>>>>>>>> of functionality. I would just extend DSL a little
to be able to
>>>>>>>>>> retrieve a property (instead using header()).
>>>>>>>>>>
>>>>>>>>>> What do you think, guys? Maybe we should clearly
communicate what
>>>>>>>>>> things are for and what are the consequences of using
one or
>>>>>>>>>> another.
>>>>>>>>>>
>>>>>>>>>> Roman
>>>>>>>>>>
>>>>>>>>>> PS. Pipeline should propagate all headers of course,
but I believe
>>>>>>>>>> an
>>>>>>>>>> endpoint is a place where we shouldn't guarantee
that headers will
>>>>>>>>>> be
>>>>>>>>>> propagated by stating it clearly.
>>>>>>>>>
>>>>>>>>> Properties have just lived in the dark and end users
does not really
>>>>>>>>> know they exists. We have some builder methods to set/get
>>>>>>>>> properties.
>>>>>>>>> I guess we need to document and maybe make sure the Spring
DSL also
>>>>>>>>> has support for accessing properties as well.
>>>>>>>>>
>>>>>>>>> To my knowledge properties is always preserved so I doubt
we have an
>>>>>>>>> issue there.
>>>>>>>>>
>>>>>>>>> So users should just start learn using properties as
well :)
>>>>>>>>> However then we have the ProducerTemplate that has one
liners for
>>>>>>>>> sending an Exchange. We dont have a sendBodyAndProperties
method.
>>>>>>>>> But
>>>>>>>>> yet again it has too many methods already. They can just
use
>>>>>>>>> send("xxx", Exchange) and have exchange populated with
the
>>>>>>>>> properties
>>>>>>>>> of choice.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2009/1/26 Claus Ibsen <claus.ibsen@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jan 24, 2009 at 9:08 PM, William Tam
>>>>>>>>>>> <email.wtam@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> What we have stored in Headers today
in Camel is both:
>>>>>>>>>>>>> - user headers
>>>>>>>>>>>>> - and system headers (added by Camel
itself).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am starting to be more and more convinced
that we should
>>>>>>>>>>>>> separate
>>>>>>>>>>>>> the two.
>>>>>>>>>>>>> So any headers that a users has enforced
to be set should be
>>>>>>>>>>>>> kept
>>>>>>>>>>>>> in
>>>>>>>>>>>>> one Map and the others that the components
set internally (such
>>>>>>>>>>>>> as
>>>>>>>>>>>>> SQL
>>>>>>>>>>>>> number of rows returned, or whatnot we
have, there are many) in
>>>>>>>>>>>>> another Map.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It means that a component would have to look
for header in more
>>>>>>>>>>>> than
>>>>>>>>>>>> one place.   Besides, the distinction of
user vs system header is
>>>>>>>>>>>> not
>>>>>>>>>>>> always clear.  For example, the operation
name header for cxf
>>>>>>>>>>>> endpoint
>>>>>>>>>>>> can be set by user but it is also created
by cxf component.   I
>>>>>>>>>>>> am
>>>>>>>>>>>> sure there are many more examples.  There
is another header
>>>>>>>>>>>> category:
>>>>>>>>>>>> protocol headers.  A protocol header is not
really a user or
>>>>>>>>>>>> system
>>>>>>>>>>>> header.  Protocol headers are header propagated
from protocol
>>>>>>>>>>>> like
>>>>>>>>>>>> HTTP, which we do want to preserve in message
header.
>>>>>>>>>>>>
>>>>>>>>>>>>> The user headers is always preserved
and copied along in the
>>>>>>>>>>>>> routing.
>>>>>>>>>>>>> User can always clear/remove unwanted
headers.
>>>>>>>>>>>>> The system headers should be short lived
as they are not really
>>>>>>>>>>>>> useable. So they are "alive" in the next
step (process) in the
>>>>>>>>>>>>> route,
>>>>>>>>>>>>> and when the pipeline invokes next route
thereafter these
>>>>>>>>>>>>> information
>>>>>>>>>>>>> is cleared.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Separating these will also make the routing/tracing
a bit easier
>>>>>>>>>>>>> as
>>>>>>>>>>>>> Users can recognize their own headers
instead its mixed with all
>>>>>>>>>>>>> the
>>>>>>>>>>>>> noise the Camel components add.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I wonder we can leverage/extend the HeaderFilterStrategy
>>>>>>>>>>>> mechanism.
>>>>>>>>>>>> Currently, it is only used for filtering
unwanted headers (in
>>>>>>>>>>>> both
>>>>>>>>>>>> request and response direction) when we propagate
headers between
>>>>>>>>>>>> Camel and external messages (like HTTP).
  HeaderFilterStrategy
>>>>>>>>>>>> is
>>>>>>>>>>>> (or
>>>>>>>>>>>> will be) associated with an endpoint.  We
could make
>>>>>>>>>>>> HeaderFilterStrategy available to the exchange
object.  So, when
>>>>>>>>>>>> an
>>>>>>>>>>>> endpoint creates an exchange, the exchange
gets a header filter
>>>>>>>>>>>> strategy.  Then, pipeline can do something
like this to filter
>>>>>>>>>>>> unwanted header: message.filterHeaders().
  The header filter
>>>>>>>>>>>> strategy
>>>>>>>>>>>> is highly customizable for each endpoint
(can have a component
>>>>>>>>>>>> wide
>>>>>>>>>>>> default) and it can be looked up from registry.
>>>>>>>>>>>>
>>>>>>>>>>> Good pointers William.
>>>>>>>>>>>
>>>>>>>>>>> Yeah we can revist it after you have moved the
header filters to
>>>>>>>>>>> the
>>>>>>>>>>> endpoint.
>>>>>>>>>>>
>>>>>>>>>>> Then we can check up upon how to leverage it
as you suggest.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>
>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> Apache Camel Committer
>>>>>>>>>
>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
>



-- 
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/

Mime
View raw message