synapse-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruwan Linton <ruwan.lin...@gmail.com>
Subject Re: VFS - Synapse Memory Leak
Date Sun, 08 Mar 2009 11:47:08 GMT
I agree and can understand the time factor and also +1 for reusing stuff
than trying to invent the wheel again :-)

Thanks,
Ruwan

On Sun, Mar 8, 2009 at 4:08 PM, Andreas Veithen
<andreas.veithen@gmail.com>wrote:

> Ruwan,
>
> It's not a question of possibility, it is a question of available time :-)
>
> Also note that some of the features that we might want to implement
> have some similarities with what is done for attachments in Axiom
> (except that an attachment is only available once, while a file over
> VFS can be read several times). I think there is also some existing
> code in Axis2 that might be useful. We should not reimplement these
> things but try to make the existing code reusable. This however is
> only realistic for the next release after 1.3.
>
> Andreas
>
> On Sun, Mar 8, 2009 at 03:47, Ruwan Linton <ruwan.linton@gmail.com> wrote:
> > Andreas,
> >
> > Can we have the caching at the file system as a property to support the
> > multiple layers touching the full message and is it possible make it to
> > specify a threshold for streaming? For example if the message is touched
> > several time we might still need streaming but not for the 100KB or
> lesser
> > files.
> >
> > Thanks,
> > Ruwan
> >
> > On Sun, Mar 8, 2009 at 1:12 AM, Andreas Veithen <
> andreas.veithen@gmail.com>
> > wrote:
> >>
> >> I've done an initial implementation of this feature. It is available
> >> in trunk and should be included in the next nightly build. In order to
> >> enable this in your configuration, you need to add the following
> >> property to the proxy:
> >>
> >> <parameter name="transport.vfs.Streaming">true</parameter>
> >>
> >> You also need to add the following mediators just before the <send>
> >> mediator:
> >>
> >> <property action="remove" name="transportNonBlocking" scope="axis2"/>
> >> <property action="set" name="OUT_ONLY" value="true"/>
> >>
> >> With this configuration Synapse will stream the data directly from the
> >> incoming to the outgoing transport without storing it in memory or in
> >> a temporary file. Note that this has two other side effects:
> >> * The incoming file (or connection in case of a remote file) will only
> >> be opened on demand. In this case this happens during execution of the
> >> <send> mediator.
> >> * If during the mediation the content of the file is needed several
> >> time (which is not the case in your example), it will be read several
> >> times. The reason is of course that the content is not cached.
> >>
> >> I tested the solution with a 2GB file and it worked fine. The
> >> performance of the implementation is not yet optimal, but at least the
> >> memory consumption is constant.
> >>
> >> Some additional comments:
> >> * The transport.vfs.Streaming property has no impact on XML and SOAP
> >> processing: this type of content is processed exactly as before.
> >> * With the changes described here, we have now two different policies
> >> for plain text and binary content processing: in-memory caching + no
> >> streaming (transport.vfs.Streaming=false) and no caching + deferred
> >> connection + streaming (transport.vfs.Streaming=true). Probably we
> >> should define a wider range of policies in the future, including file
> >> system caching + streaming.
> >> * It is necessary to remove the transportNonBlocking property
> >> (MessageContext.TRANSPORT_NON_BLOCKING) to prevent the <send> mediator
> >> (more precisely the OperationClient) from executing the outgoing
> >> transport in a separate thread. This property is set by the incoming
> >> transport. I think this is a bug since I don't see any valid reason
> >> why the transport that handles the incoming request should determine
> >> the threading behavior of the transport that sends the outgoing
> >> request to the target service. Maybe Asankha can comment on this?
> >>
> >> Andreas
> >>
> >> On Thu, Mar 5, 2009 at 07:21, kimhorn <kim.horn@icsglobal.net> wrote:
> >> >
> >> > Thats good; as this stops us using Synapse.
> >> >
> >> >
> >> >
> >> > Asankha C. Perera wrote:
> >> >>
> >> >>
> >> >>> Exception in thread "vfs-Worker-4" java.lang.OutOfMemoryError:
Java
> >> >>> heap
> >> >>> space
> >> >>>         at
> >> >>>
> >> >>>
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
> >> >>>         at
> >> >>>
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:518)
> >> >>>         at java.lang.StringBuffer.append(StringBuffer.java:307)
> >> >>>         at java.io.StringWriter.write(StringWriter.java:72)
> >> >>>         at
> org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129)
> >> >>>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:1104)
> >> >>>         at org.apache.commons.io.IOUtils.copy(IOUtils.java:1078)
> >> >>>         at org.apache.commons.io.IOUtils.toString(IOUtils.java:382)
> >> >>>         at
> >> >>>
> >> >>>
> org.apache.synapse.format.PlainTextBuilder.processDocument(PlainTextBuilder.java:68)
> >> >>>
> >> >> Since the content type is text, the plain text formatter is trying
to
> >> >> use a String to parse as I see.. which is a problem for large
> content..
> >> >>
> >> >> A definite bug we need to fix ..
> >> >>
> >> >> cheers
> >> >> asankha
> >> >>
> >> >> --
> >> >> Asankha C. Perera
> >> >> AdroitLogic, http://adroitlogic.org
> >> >>
> >> >> http://esbmagic.blogspot.com
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> >> >> For additional commands, e-mail: dev-help@synapse.apache.org
> >> >>
> >> >>
> >> >>
> >> >
> >> > --
> >> > View this message in context:
> >> >
> http://www.nabble.com/VFS---Synapse-Memory-Leak-tp22344176p22345904.html
> >> > Sent from the Synapse - Dev mailing list archive at Nabble.com.
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> >> > For additional commands, e-mail: dev-help@synapse.apache.org
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> >> For additional commands, e-mail: dev-help@synapse.apache.org
> >>
> >
> >
> >
> > --
> > Ruwan Linton
> > http://wso2.org - "Oxygenating the Web Services Platform"
> > http://ruwansblog.blogspot.com/
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
> For additional commands, e-mail: dev-help@synapse.apache.org
>
>


-- 
Ruwan Linton
http://wso2.org - "Oxygenating the Web Services Platform"
http://ruwansblog.blogspot.com/

Mime
View raw message