Return-Path: X-Original-To: apmail-synapse-dev-archive@www.apache.org Delivered-To: apmail-synapse-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 662639A38 for ; Thu, 19 Apr 2012 16:55:05 +0000 (UTC) Received: (qmail 85487 invoked by uid 500); 19 Apr 2012 16:55:05 -0000 Delivered-To: apmail-synapse-dev-archive@synapse.apache.org Received: (qmail 85409 invoked by uid 500); 19 Apr 2012 16:55:05 -0000 Mailing-List: contact dev-help@synapse.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@synapse.apache.org Delivered-To: mailing list dev@synapse.apache.org Received: (qmail 85402 invoked by uid 99); 19 Apr 2012 16:55:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Apr 2012 16:55:05 +0000 X-ASF-Spam-Status: No, hits=2.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hiranya911@gmail.com designates 209.85.212.42 as permitted sender) Received: from [209.85.212.42] (HELO mail-vb0-f42.google.com) (209.85.212.42) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Apr 2012 16:54:59 +0000 Received: by vbjk13 with SMTP id k13so6071365vbj.15 for ; Thu, 19 Apr 2012 09:54:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=NgA2yLqcJpx0Msj2L3cZlqvx9wQktsna+/WVOoHqzc8=; b=j04i4x5GWh4KbYXXo/RM23TJ0zWG7G2IM4V0Z9DbA3xjNXO4CGZwfGJozyXLbc3wzb zIOJ88gTPgRVTQyuZ8DuuV2UzLDr2LkC9optpnHa+J/CxX2mwAGyiQcaJJOZDg3J9H6M te3YHIfAZQ6tbKxzpDOlHcKVKLAOcy+8AAgqYsl0aE7s6YGoGUCxjMqpC0hzCWB0USeI gNdRjbD7BYn513WunkZagIFjIXtp9JQJyj3KBMkItickXKzZfICP7JHeCaxRgSH5AZf5 GWUTjLAnocCIN+cVJHQglLz2q4nrpKya5kHl2Qeb3TsrDkKJhdHvc5ov34d7hQllt9u9 zBgg== MIME-Version: 1.0 Received: by 10.52.179.168 with SMTP id dh8mr1186236vdc.120.1334854478278; Thu, 19 Apr 2012 09:54:38 -0700 (PDT) Received: by 10.220.75.11 with HTTP; Thu, 19 Apr 2012 09:54:38 -0700 (PDT) In-Reply-To: References: Date: Thu, 19 Apr 2012 22:24:38 +0530 Message-ID: Subject: Re: Message Relay Support for Synapse From: Hiranya Jayathilaka To: dev@synapse.apache.org Content-Type: multipart/alternative; boundary=bcaec51a7c0801320304be0b0727 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51a7c0801320304be0b0727 Content-Type: text/plain; charset=ISO-8859-1 Hi Supun, On Thu, Apr 19, 2012 at 7:14 PM, Supun Kamburugamuva wrote: > Hi Hiranya, > > I have some ideas in my mind to improve the Pass through transport. Mainly > it is to stream a single source to multiple targets. If you are planning to > move it quickly I can start on these improvements. > Yes, my plan is to do the code move soon. Hopefully sometime over the next few days. You can also grab the current source from WSO2 SVN and try out your modifications on that if you want. Thanks, Hiranya > > Thanks, > Supun.. > > > On Tue, Apr 17, 2012 at 9:36 AM, indika kumara wrote: > >> Thanks Hiranaya for the information. However, It may be a killer feature >> if we can support the streaming with most of the mediation scenarios even >> with some limitations. >> >> Thanks, >> >> Indika >> >> >> On Tue, Apr 17, 2012 at 8:41 PM, Hiranya Jayathilaka < >> hiranya911@gmail.com> wrote: >> >>> >>> >>> On Tue, Apr 17, 2012 at 3:45 PM, indika kumara wrote: >>> >>>> Not sure that it is Impossible. Where are the limitations - ours, XSLT >>>> engine, XSLT spec, etc.? >>> >>> >>> It's not one thing Indika. It's many things. Synapse has to work with >>> many WS-* modules, XSLT engines, XPath engines, XQuery and other >>> transformation engines and much more. Our Axiom representation can work >>> with most of these stuff and where that's not possible we can convert it >>> into a standard model like DOM or SAX. But the bottom-line is we have to >>> parse the message by some means to work with these libraries/APIs. Relay >>> module basically keeps the message in a completely un-parsed state. None of >>> the above APIs can work with such raw data. >>> >>> But I agree that for certain selected scenarios we might be able to work >>> in the relay mode all the time. In fact I once wrote a STX mediator using >>> Joost which can work withe binary representation provided by the message >>> relay builder. >>> >>> Thanks, >>> Hiranya >>> >>> >>>> >>>> I believe it should be possible. One way may >>>> be identifying XML fragments and transform them on the fly by fragment by >>>> fragment - read one fragment from input, transform it, and write it >>>> to output...I did my self a case study for a specific customer when I was >>>> at WSO2. He was happy. Of course, it is just a case study for one user. >>>> But, someone should be able to come up with a more generalized >>>> and improved solution. >>>> >>>> I can see some research on this topic. There should be >>>> some interesting ideas. >>>> >>>> http://www.springerlink.com/content/l156j0255347k356/ >>>> http://www.sciencedirect.com/science/article/pii/S0167642304001194 >>>> http://dl.acm.org/citation.cfm?id=1042051 >>>> >>>> On Tue, Apr 17, 2012 at 6:38 PM, Hiranya Jayathilaka < >>>> hiranya911@gmail.com> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Apr 17, 2012 at 10:49 AM, indika kumara >>>> > wrote: >>>>> >>>>>> Ohh... I thought we have a solution to get >>>>>> or preserve the benefits offered by the binary relay when it is used for >>>>>> other mediation scenarios - for example, with some kind of >>>>>> on-demand streaming transformation. >>>>> >>>>> >>>>> That's not practically possible. For security to work, Rampart should >>>>> be able to find the necessary headers in the SOAP infoset. For XSLT to work >>>>> the XSLT engine should be able to parse the given XML infoset. So for these >>>>> cases we have to build the message as usual and let the normal processing >>>>> take place. What we have implemented is kind of a compiler optimization for >>>>> Synapse language, where by looking at the configuration provided we insert >>>>> the build instruction to the sequence. >>>>> >>>>> Thanks, >>>>> Hiranya >>>>> >>>>> >>>>>> >>>>>> >>>>>> ~ Indika >>>>>> >>>>>> On Tue, Apr 17, 2012 at 2:24 PM, Hiranya Jayathilaka < >>>>>> hiranya911@gmail.com> wrote: >>>>>> >>>>>>> Hi Folks, >>>>>>> >>>>>>> Just to briefly explain how other messaging scenarios are made to >>>>>>> work with the relay: >>>>>>> >>>>>>> 1. A special handler checks whether security or some other content >>>>>>> aware module is engaged on the target service and builds the messages if >>>>>>> necessary. >>>>>>> 2. Each mediator knows whether they should build the message or not >>>>>>> (depending on their configuration). If necessary, mediators will build the >>>>>>> message. >>>>>>> >>>>>>> As a result a sequence such as the following will run in the pure >>>>>>> pass through mode: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Something like the following will cause the message to be built at >>>>>>> the log mediator: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> In the following proxy service, the in-sequence will be in the pass >>>>>>> through mode. The out-sequence will build the message at the xslt mediator: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Hiranya >>>>>>> >>>>>>> On Tue, Apr 17, 2012 at 9:32 AM, indika kumara < >>>>>>> indika.kuma@gmail.com> wrote: >>>>>>> >>>>>>>> +1 great donation! Would love to see the support for any mediation >>>>>>>> scenario with it too. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Indika >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 16, 2012 at 4:17 PM, Hiranya Jayathilaka < >>>>>>>> hiranya911@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Folks, >>>>>>>>> >>>>>>>>> Message relay is a performance enhancer we have developed for WSO2 >>>>>>>>> ESB. We are now thinking about contributing this code to Synapse. There are >>>>>>>>> two main components for this: >>>>>>>>> >>>>>>>>> 1. A builder-formatter pair for mediating messages without >>>>>>>>> building the Axiom infoset >>>>>>>>> 2. A new HTTP transport (pass through transport) >>>>>>>>> >>>>>>>>> These can be used independently of each other. One of the existing >>>>>>>>> limitations of this code is that it restricts the scenarios that we can run >>>>>>>>> on Synapse to pass through scenarios. But I've been working on making this >>>>>>>>> code work with any mediation scenario (transformation, security etc). So >>>>>>>>> that way we will be able to run Synapse with these optimizers by default >>>>>>>>> and still all the messaging scenarios would work out of the box. >>>>>>>>> >>>>>>>>> A performance comparison of the current NHTTP transport and the >>>>>>>>> above optimizers can be found at [1]. >>>>>>>>> >>>>>>>>> Appreciate your feedback on this. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Hiranya >>>>>>>>> >>>>>>>>> [1] - >>>>>>>>> http://wso2.org/library/articles/2012/03/performance-benchmarking-wso2-esb-different-message-transfer-mechanisms >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Hiranya Jayathilaka >>>>>>>>> Associate Technical Lead; >>>>>>>>> WSO2 Inc.; http://wso2.org >>>>>>>>> E-mail: hiranya@wso2.com; Mobile: +94 77 633 3491 >>>>>>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Hiranya Jayathilaka >>>>>>> Associate Technical Lead; >>>>>>> WSO2 Inc.; http://wso2.org >>>>>>> E-mail: hiranya@wso2.com; Mobile: +94 77 633 3491 >>>>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Hiranya Jayathilaka >>>>> Associate Technical Lead; >>>>> WSO2 Inc.; http://wso2.org >>>>> E-mail: hiranya@wso2.com; Mobile: +94 77 633 3491 >>>>> Blog: http://techfeast-hiranya.blogspot.com >>>>> >>>> >>>> >>> >>> >>> -- >>> Hiranya Jayathilaka >>> Associate Technical Lead; >>> WSO2 Inc.; http://wso2.org >>> E-mail: hiranya@wso2.com; Mobile: +94 77 633 3491 >>> Blog: http://techfeast-hiranya.blogspot.com >>> >> >> > > > -- > Supun Kamburugamuva > Member, Apache Software Foundation; http://www.apache.org > E-mail: supun06@gmail.com ; Mobile: +94 77 431 3585 > Blog: http://supunk.blogspot.com > > > > -- Hiranya Jayathilaka Associate Technical Lead; WSO2 Inc.; http://wso2.org E-mail: hiranya@wso2.com; Mobile: +94 77 633 3491 Blog: http://techfeast-hiranya.blogspot.com --bcaec51a7c0801320304be0b0727 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Supun,

On Thu, Apr 19, 2012 at 7:14 PM= , Supun Kamburugamuva <supun06@gmail.com> wrote:
Hi Hiranya,

I have some ideas in my mind to improve the = Pass through transport. Mainly it is to stream a single source to multiple = targets. If you are planning to move it quickly I can start on these improv= ements.

Yes, my plan is to do the code move soon. = Hopefully sometime over the next few days. You can also grab the current so= urce from WSO2 SVN and try out your modifications on that if you want.

Thanks,
Hiranya
=A0

Thanks,
Supun..

On Tue, Apr 17, 2012 at 9:36 AM, indika kumara = <indika.kuma@gmail.com> wrote:
Thanks Hiranaya for the information. However= , It may be a killer feature if we can=A0support=A0the streaming with most = of the mediation=A0scenarios=A0even with some limitations.=A0

Thanks,

Indika


On Tue, Apr 17, 2012 at = 8:41 PM, Hiranya Jayathilaka <hiranya911@gmail.com> wrote= :


On Tue, Apr 17, 2012 at 3:45 PM, in= dika kumara <indika.kuma@gmail.com> wrote:
Not sure that it is Impossible. Where are the=A0limitations - ours, XSLT en= gine, XSLT spec, etc.?

It's not o= ne thing Indika. It's many things. Synapse has to work with many WS-* m= odules, XSLT engines, XPath engines, XQuery and other transformation engine= s and much more. Our Axiom representation can work with most of these stuff= and where that's not possible we can convert it into a standard model = like DOM or SAX. But the bottom-line is we have to parse the message by som= e means to work with these libraries/APIs. Relay module basically keeps the= message in a completely un-parsed state. None of the above APIs can work w= ith such raw data.=A0

But I agree that for certain selected scenarios we migh= t be able to work in the relay mode all the time. In fact I once wrote a ST= X mediator using Joost which can work withe binary representation provided = by the message relay builder.

Thanks,
Hiranya
=A0
=

I=A0believe=A0it should = be possible. One way may be=A0identifying=A0XML=A0fragments and=A0transform= =A0them on the fly by=A0fragment=A0by fragment - read one fragment from inp= ut, transform it, and write it to=A0output...I did my self a case study for= a=A0specific=A0customer when I was at WSO2. He was happy. Of course, it is= just a case=A0study=A0for one user. But, someone should be able to come up= with a more generalized and=A0improved=A0solution.

I can see some research on this topic. There should be = some=A0interesting=A0ideas.


On Tue, Apr 17, 2012 at 6:38 PM, Hiranya Jay= athilaka <hiranya911@gmail.com> wrote:


On Tue, Apr 17, 2012 at 10:49 AM, i= ndika kumara <indika.kuma@gmail.com> wrote:
Ohh... I thought we have a solution to get or=A0preserve=A0the=A0benefits= =A0offered=A0by the binary relay when it is used for other mediation scenar= ios - for example, with some kind of on-demand=A0streaming=A0transformation= .

That's not practically possible. For security= to work, Rampart should be able to find the necessary headers in the SOAP = infoset. For XSLT to work the XSLT engine should be able to parse the given= XML infoset. So for these cases we have to build the message as usual and = let the normal processing take place. What we have implemented is kind of a= compiler optimization for Synapse language, where by looking at the config= uration provided we insert the build instruction to the sequence.

Thanks,
Hiranya
=A0
=
=A0

~ Indika

On Tue, Apr 17, 2012 at 2:24 PM, Hiranya Jay= athilaka <hiranya911@gmail.com> wrote:
Hi Folks,

Just to briefly explain how other messaging sc= enarios are made to work with the relay:

1. A spec= ial handler checks whether security or some other content aware module is e= ngaged on the target service and builds the messages if necessary.
2. Each mediator knows whether they should build the message or not (d= epending on their configuration). If necessary, mediators will build the me= ssage.

As a result a sequence such as the followin= g will run in the pure pass through mode:

<sequence>
=A0 =A0 <log/>
=A0 =A0 <send/>
</sequence>

Something like the following will cause the message to be built at the l= og mediator:

<sequence>
=A0 =A0<log level=3D"= ;full"/>
=A0 =A0<send/>
</sequence>=

In the following proxy service, the in-sequence w= ill be in the pass through mode. The out-sequence will build the message at= the xslt mediator:

<proxy name=3D"foo">
=A0 =A0&= lt;target endpoint=3D"bar">
=A0 =A0 =A0 <inSequen= ce>
=A0 =A0 =A0 =A0 =A0 <log/>
=A0 =A0 =A0 <= ;/inSequence>
=A0 =A0 =A0 <outSequence>
=A0 =A0 =A0 =A0 =A0 <log/><= /div>
=A0 =A0 =A0 =A0 =A0 <xslt key=3D"baz"/>
=A0 =A0 =A0 =A0 =A0 <send/>
=A0 =A0 =A0 </outSequence&g= t;
=A0 =A0</target>
</proxy>

Thanks,
Hiranya

On Tue, Apr 17, 2012 at 9:32 AM, = indika kumara <indika.kuma@gmail.com> wrote:
+1=A0great=A0donation! =A0Would love to see = the support for=A0any=A0mediation scenario=A0with it too.
=
Regards,

Indika


On Mon, Apr 16, 2012 at 4:17 PM, Hiranya Jayathilaka <hiranya= 911@gmail.com> wrote:
Hi Folks,

Message relay i= s a performance enhancer we have developed for WSO2 ESB. We are now thinkin= g about contributing this code to Synapse. There are two main components fo= r this:

1. A builder-formatter pair for mediating messages without building th= e Axiom infoset
2. A new HTTP transport (pass through transport)<= /div>

These can be used independently of each other. One= of the existing limitations of this code is that it restricts the scenario= s that we can run on Synapse to pass through scenarios. But I've been w= orking on making this code work with any mediation scenario (transformation= , security etc). So that way we will be able to run Synapse with these opti= mizers by default and still all the messaging scenarios would work out of t= he box.

A performance comparison of the current NHTTP transport= and the above optimizers can be found at [1].

App= reciate your feedback on this.

Thanks,
Hiranya


--
Hiranya Jayathilaka
Associate Technical Lead;=
WSO2 Inc.;=A0 http://wso2= .org
E-mail: h= iranya@wso2.com;=A0 Mobile: +94 77 633 3491
Blog: h= ttp://techfeast-hiranya.blogspot.com




--
Hiranya Jaya= thilaka
Associate Technical Lead;
WSO2 Inc.;=A0 http://wso2.org
E-mail: hiranya@wso2.com;=A0 Mobile: += 94 77 633 3491
Blog: h= ttp://techfeast-hiranya.blogspot.com



=

--
Hiranya Jayathilaka
Associate Technical Lead;
W= SO2 Inc.;=A0 http://wso2.org<= /a>
E-mail: hiranya@wso2.= com;=A0 Mobile: +94 77 633 3491
Blog: h= ttp://techfeast-hiranya.blogspot.com




--
Hiranya Jayathilaka
Associate Technical Lead;
WSO2 Inc.;=A0= http://wso2.org
E-mail: hiranya@wso2.= com;=A0 Mobile: +94 77 633 3491
Blog: h= ttp://techfeast-hiranya.blogspot.com




--
Supun Kamburugamuva
Member, Apache Software Foundation= ; http://www.apache.org=
E-mail: supun06@gmail.c= om; =A0Mobile: +94 77 431 3585
Blog: http://supun= k.blogspot.com






--
Hiranya Jaya= thilaka
Associate Technical Lead;
WSO2 Inc.;=A0 http://wso2.org
E-mail: hiranya@wso2.com;=A0 Mobile: +94 77 = 633 3491
Blog: h= ttp://techfeast-hiranya.blogspot.com
--bcaec51a7c0801320304be0b0727--