Return-Path: Delivered-To: apmail-ws-axis-c-dev-archive@www.apache.org Received: (qmail 24681 invoked from network); 15 Jul 2008 10:27:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 15 Jul 2008 10:27:14 -0000 Received: (qmail 77969 invoked by uid 500); 15 Jul 2008 10:27:14 -0000 Delivered-To: apmail-ws-axis-c-dev-archive@ws.apache.org Received: (qmail 77951 invoked by uid 500); 15 Jul 2008 10:27:14 -0000 Mailing-List: contact axis-c-dev-help@ws.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: List-Id: "Apache AXIS C Developers List" Reply-To: "Apache AXIS C Developers List" Delivered-To: mailing list axis-c-dev@ws.apache.org Received: (qmail 77940 invoked by uid 99); 15 Jul 2008 10:27:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jul 2008 03:27:14 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of csethil@gmail.com designates 64.233.166.181 as permitted sender) Received: from [64.233.166.181] (HELO py-out-1112.google.com) (64.233.166.181) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jul 2008 10:26:18 +0000 Received: by py-out-1112.google.com with SMTP id a29so2843201pyi.0 for ; Tue, 15 Jul 2008 03:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=4dfE6Nv59WjwFD05ry4lhqxa5X5uchrKMuFP4HVLchI=; b=KFyGXOjaOxEeoP08f0OwTs48QLeftPQhZt08y3XT4zg9j9vUmt249x/4rk2Esra2oF InoV1D/1TLKHVxSVvlzxIO5fxzpxh+GLytPkjfzGJ0qmHJUcADYPzN5OIufvROkM1dNk TrrCzf7wxDxn822SQcpi6otrwbNWFav3nubGk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=rWu4mUfzo/nP7YdIZM92DoSLGzRboJFYGSLXOk7PT3gmD5gY1OoFQ45LJzJGfBNAPs 0/+f2l9wuspwwrotAvGe8dCnd8GA75tE671Tcdt9Pr8NKzwz8A6NUgwGZh3IChpASRm0 Nd+Ox+TeDj/1pMjYk0yIbOjpAbE7n3Qn/iFOI= Received: by 10.115.76.1 with SMTP id d1mr18693922wal.207.1216117600545; Tue, 15 Jul 2008 03:26:40 -0700 (PDT) Received: by 10.115.94.13 with HTTP; Tue, 15 Jul 2008 03:26:40 -0700 (PDT) Message-ID: Date: Tue, 15 Jul 2008 03:26:40 -0700 From: "Thilina Gunarathne" To: "Apache AXIS C Developers List" Subject: Re: New method for creating xml_reader In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2908_15108095.1216117600537" References: <1215869034.10844.21.camel@manjula> <22a059a60807120701n24adcaf5jf3ebaacc54a83429@mail.gmail.com> <1215877903.10844.57.camel@manjula> <1216113490.6024.21.camel@manjula> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_2908_15108095.1216117600537 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi Manjula, One thing you guys can try to do is to break up the stream at the MIME parser level. Then the MIME parser only gives the SOAP Envelope part to the XML parser and nothing beyond. This can be achieved by faking a input stream. MIME parser gets the original input stream and then gives the XML parser a faked inputstream, something which provides the same interface and functionality as the original inputstream, but the MIME parser has control over what goes in to it. For an example, if reading an InputStream is reading a memory pointer (please excuse my very bad knowledge about C I/O) which gets filled by the transport layer, then the MIME parser can also write the data in to some memory location which will act as the input stream location for XMLReader. If the above is possible, then the MIME parser can stream the actual input stream directly to the faked inputstream till the SOAP part boundary is reached or till the user requests another MIME part. Streaming MIME parsing will happen throughout the process to limit the FAKED input stream to the contents of the SOAP part. If the user requests a MIME part while the XML reader is only half way in to parsing the SOAP envelope, then the MIME parser should read ahead the SOAP part, store it in memory and should provide it as needed to the FAKED input stream. One other way to visualize this is to think MIME parsing happens at a level below the XML parsing, just like transport level. I'm not sure how hard to do this in Axis2/C (or whether it's impossible)... Let me know if I'm not clear with the idea. thanks, Thilina On Tue, Jul 15, 2008 at 3:04 AM, Supun Kamburugamuva wrote: > >> But then again the problem will occur when xml_parser give the control >> back to mime_parser. Because xml_parser will read beyond the soap part >> when parsing. So the parser should be modified to return the the >> unwanted data it read beyond the soap part while parsing. I think >> Guththila can be modified for this requirement, but does any one know >> how to do this with libxml2. >> > > I assume by "read beyond the soap part" you mean consuming the stream > beyond the soap part. But I think this is unavoidable in any parser. > > 1. Parser don't know where the xml ends. > 2. The parser doesn't read character by chareacter from the stream (if it > does it'll be very inefficient) and it reads chunks of data from the stream. > > Becuase of the above reasons the parser will consume the stream beyond the > soap part. > > Supun.. > >> >> >> >> On Sat, 2008-07-12 at 21:21 +0530, Manjula Peiris wrote: >> > On Sat, 2008-07-12 at 19:31 +0530, Milinda Pathirage wrote: >> > > As Thilina mentioned in above thread Axis2/Java keeps a instance of >> > > mime parser in it's xml builder. So I think it's ok to follow that >> > > method. AFAIK, I think if we enable MTOM caching we have to have this >> > > feature or is anybody have ideas about how to solve this without going >> > > to this method?. Manjula can you point out the changes we need in the >> > > parser. Then we can discuss the problem and available solutions. This >> > > will help us to avoid any problems occur in the future because of the >> > > changes. >> > > If we have two methods for xml reader creation, how we can use both of >> > > them in the Axis2/C engine. Are we going to check the MIME headers and >> > > create the required xml reader? >> > >> > In the MTOM case the serialized message is most probably in the >> > following format. >> > >> > --MimeBoundary >> > mime_header1:xxxxx >> > mime_header2:xxxxx >> > >> > ....... >> > --MimeBoundary >> > mime_header1:xxxx >> > mime_header2:xxxx >> > >> > yyyyyyyyybinarydatayyyyyyyyyyyy >> > --MimeBoundary-- >> > >> > We have the information form http_headers whether this is a multipart >> > message or not. So will call the soap_builder straightaway with a flag >> > telling this is a multipart message. Since the modified soap_builder >> > will have a reference to mime_parser instance when seeing this flag it >> > will call the mime_parser. mime_parser will search for \r\n\r\n sequence >> > between SOAP part and first mime_headers set. When searching for this it >> > may read some part of the SOAP. So after this search we can't >> > straightaway call either xml_reader_create_for_io or >> > xml_reader_create_for_memory, because none of them will not get correct >> > xml. So what we need is first read from a buffer and then when that >> > buffer is finished read from the network stream. This will cause to >> > define and implement the new xml_reader function and corresponding >> > modifications at parser level as I mentioned in my previous mail. >> > >> > -Manjula. >> > >> > >> > > >> > > Thanks, >> > > Milinda... >> > > >> > > On Sat, Jul 12, 2008 at 6:53 PM, Manjula Peiris >> > > wrote: >> > > Hi all, >> > > >> > > MTOM caching stuff is finished in both send and receiving >> > > sides.It is >> > > available here [1]. But due to this [2] problem, it cannot be >> > > integrated >> > > to Axis2/C trunk. >> > > >> > > In order to solve that problem we need to keep a reference of >> > > mime_parser instance in the soap_builder similarly >> > > stax_builder. So when >> > > parsing the soap part the xml_reader will not just read from >> > > the stream, >> > > but also need to read from the buffer which will be available >> > > after >> > > parsing mime_headrs. So we need to create a parser instance >> > > which will >> > > first read from the buffer and when that buffer is not >> > > containing any >> > > more data, it will read from the stream. So we may need to add >> > > a new >> > > function say, >> > > xml_reader_create_for_memory_and_io and change some of the >> > > logic in >> > > parser. >> > > Your suggestions are highly appreciated. >> > > >> > > [1] >> https://svn.apache.org/repos/asf/webservices/axis2/branches/c/post_1_4_mtom/c >> > > >> > > [2] >> http://marc.info/?l=axis-c-dev&m=121320959601769&w=2 >> > > >> > > Thanks, >> > > -Manjula. >> > > >> > > >> > > -- >> > > Manjula Peiris: http://manjula-peiris.blogspot.com/ >> > > >> > > >> > > >> --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org >> > > For additional commands, e-mail: >> axis-c-dev-help@ws.apache.org >> > > >> > > >> > > >> > > >> > > -- >> > > http://mpathirage.com >> > > http://wso2.org "Oxygen for Web Service Developers" >> > > http://wsaxc.blogspot.com "Web Services With Axis2/C" >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org >> > For additional commands, e-mail: axis-c-dev-help@ws.apache.org >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org >> For additional commands, e-mail: axis-c-dev-help@ws.apache.org >> >> > -- Thilina Gunarathne - http://thilinag.blogspot.com ------=_Part_2908_15108095.1216117600537 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi Manjula,
One thing you guys can try to do is to break up the stream at the MIME parser level. Then the MIME parser only gives the SOAP Envelope part to the XML parser and nothing beyond. This can be achieved by faking a input stream. MIME parser gets the original input stream and then gives the XML parser a faked inputstream, something which provides the same interface and functionality as the original inputstream, but the  MIME parser has control over what goes in to it.

For an example, if reading an InputStream is reading a memory pointer (please excuse my very bad knowledge about C I/O) which gets filled by the transport layer, then the MIME parser can also write the data in to some memory location  which will act as the input stream location for XMLReader.

If the above is possible, then the MIME parser can stream the actual input stream directly to the faked inputstream till the SOAP part boundary is reached or till the user requests another MIME part. Streaming MIME parsing will happen throughout the process to limit the FAKED input stream to the contents of the SOAP part.  If the user requests a MIME part while the XML reader is only half way in to parsing the SOAP envelope, then the MIME parser should read ahead the SOAP part, store it in memory and should provide it as needed to the FAKED input stream. One other way to visualize this is to think MIME parsing happens at a level below the XML parsing, just like transport level.

I'm not sure how hard to do this in Axis2/C (or whether it's impossible)... Let me know if I'm not clear with the idea.

thanks,
Thilina


On Tue, Jul 15, 2008 at 3:04 AM, Supun Kamburugamuva <supun06@gmail.com> wrote:

But then again the problem will occur when xml_parser give the control
back to mime_parser. Because xml_parser will read beyond the soap part
when parsing. So the parser should be modified to return the the
unwanted data it read beyond the soap part while parsing. I think
Guththila can be modified for this requirement, but does any one know
how to do this with libxml2.
 
I assume by "read beyond the soap part" you mean consuming the stream beyond the soap part. But I think this is unavoidable in any parser. 
 
1. Parser don't know where the xml ends.
2. The parser doesn't read character by chareacter from the stream (if it does it'll be very inefficient) and it reads chunks of data from the stream.
 
Becuase of the above reasons the parser will consume the stream beyond the soap part. 
 
Supun..  


On Sat, 2008-07-12 at 21:21 +0530, Manjula Peiris wrote:
> On Sat, 2008-07-12 at 19:31 +0530, Milinda Pathirage wrote:
> > As Thilina mentioned in above thread Axis2/Java keeps a instance of
> > mime parser in it's xml builder. So I think it's ok to follow that
> > method. AFAIK, I think if we enable MTOM caching we have to have this
> > feature or is anybody have ideas about how to solve this without going
> > to this method?. Manjula can you point out the changes we need in the
> > parser. Then we can discuss the problem and available solutions. This
> > will help us to avoid any problems occur in the future because of the
> > changes.
> > If we have two methods for xml reader creation, how we can use both of
> > them in the Axis2/C engine. Are we going to check the MIME headers and
> > create the required xml reader?
>
> In the MTOM case the serialized message is most probably in the
> following format.
>
> --MimeBoundary
> mime_header1:xxxxx
> mime_header2:xxxxx
>
> <soapenv:Envelope>.......</soapenv:Envelope>
> --MimeBoundary
> mime_header1:xxxx
> mime_header2:xxxx
>
> yyyyyyyyybinarydatayyyyyyyyyyyy
> --MimeBoundary--
>
> We have the information form http_headers whether this is a multipart
> message or not. So will call the soap_builder straightaway with a flag
> telling this is a multipart message. Since the modified soap_builder
> will have a reference to mime_parser instance when seeing this flag it
> will call the mime_parser. mime_parser will search for \r\n\r\n sequence
> between SOAP part and first mime_headers set. When searching for this it
> may read some part of the SOAP. So after this search we can't
> straightaway call either xml_reader_create_for_io or
> xml_reader_create_for_memory, because none of them will not get correct
> xml. So what we need is first read from a buffer and then when that
> buffer is finished read from the network stream. This will cause to
> define and implement the new xml_reader function and corresponding
> modifications at parser level as I mentioned in my previous mail.
>
> -Manjula.
>
>
> >
> > Thanks,
> > Milinda...
> >
> > On Sat, Jul 12, 2008 at 6:53 PM, Manjula Peiris <manjula@wso2.com>
> > wrote:
> >         Hi all,
> >
> >         MTOM caching stuff is finished in both send and receiving
> >         sides.It is
> >         available here [1]. But due to this [2] problem, it cannot be
> >         integrated
> >         to Axis2/C trunk.
> >
> >         In order to solve that problem we need to keep a reference of
> >         mime_parser instance in the soap_builder similarly
> >         stax_builder. So when
> >         parsing the soap part the xml_reader will not just read from
> >         the stream,
> >         but also need to read from the buffer which will be available
> >         after
> >         parsing mime_headrs. So we need to create a parser instance
> >         which will
> >         first read from the buffer and when that buffer is not
> >         containing any
> >         more data, it will read from the stream. So we may need to add
> >         a new
> >         function say,
> >         xml_reader_create_for_memory_and_io and change some of the
> >         logic in
> >         parser.
> >         Your suggestions are highly appreciated.
> >
> >         [1]https://svn.apache.org/repos/asf/webservices/axis2/branches/c/post_1_4_mtom/c
> >
> >         [2]http://marc.info/?l=axis-c-dev&amp;m=121320959601769&amp;w=2
> >
> >         Thanks,
> >         -Manjula.
> >
> >
> >         --
> >         Manjula Peiris: http://manjula-peiris.blogspot.com/
> >
> >
> >         ---------------------------------------------------------------------
> >         To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> >         For additional commands, e-mail: axis-c-dev-help@ws.apache.org
> >
> >
> >
> >
> > --
> > http://mpathirage.com
> > http://wso2.org "Oxygen for Web Service Developers"
> > http://wsaxc.blogspot.com "Web Services With Axis2/C"
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
> For additional commands, e-mail: axis-c-dev-help@ws.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: axis-c-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: axis-c-dev-help@ws.apache.org





--
Thilina Gunarathne - http://thilinag.blogspot.com ------=_Part_2908_15108095.1216117600537--