camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (Confluence)" <conflue...@apache.org>
Subject [CONF] Apache Camel > Stream caching
Date Wed, 17 Jul 2013 16:10:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/en/2176/1/1/_/styles/combined.css?spaceKey=CAMEL&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/CAMEL/Stream+caching">Stream
caching</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~davsclaus">Claus
Ibsen</a>
    </h4>
        <br/>
                         <h4>Changes (2)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >{info} <br> <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">{tip}
<br>If using Camel 2.12 onwards then see about StreamCachingStrategy further below which
is the recommended way to configure stream caching options. <br>{tip} <br> <br></td></tr>
            <tr><td class="diff-unchanged" >h3. Enabling stream caching <br>
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br> <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">h3.
Using StreamCachingStrategy <br>*Available as of Camel 2.12* <br> <br>[Stream
caching] is from Camel 2.12 onwards intended to be configured using {{org.apache.camel.spi.StreamCachingStrategy}}.
<br>The old kind of configuration using properties on the [CamelContext] has been marked
as deprecated. <br> <br>The strategy has the following options: <br> <br>||
Option || Default || Description || <br>| temporaryDirectory | java.io.tmpdir | Base
directory where temporary files for spooled streams should be stored. | <br>| spoolThreshold
| 128kb | Size in bytes when the stream should be spooled to disk instead of keeping in memory.
Use a value of 0 or negative to disable it all together so streams is always kept in memory
regardless of their size. | <br>| spoolChiper | null | If set, the temporary files are
encrypted using the specified cipher transformation (i.e., a valid stream or 8-bit cipher
name such as &quot;RC4&quot;, &quot;AES/CTR/NoPadding&quot;. An empty name
&quot;&quot; is treated as null).| <br>| bufferSize | 4096 | Size in bytes of
the buffer used in the stream. | <br>| removeTemporaryDirectoryWhenStopping | true |
Whether to remove the temporary directory when stopping [CamelContext]. | <br> <br>h4.
Using StreamCachingStrategy in Java <br> <br>You can configure the {{StreamCachingStrategy}}
in Java as shown below: <br>{code} <br>context.getStreamCachingStrategy().setTemporaryDirectory&quot;/tmp/cachedir&quot;);
<br>context.getStreamCachingStrategy().setSpoolThreshold(64 * 1024); <br>context.getStreamCachingStrategy().setBufferSize(16
* 1024); <br>// to enable encryption using RC4 <br>// context.getStreamCachingStrategy().setSpoolChiper(&quot;RC4&quot;);
<br>{code} <br> <br>And remember to enable [Stream caching] on the [CamelContext]
or on routes <br>{code} <br>context.setStreamCaching(true); <br>{code} <br>
<br>h4. Using StreamCachingStrategy in XML <br> <br>And in XML you do <br>{code:xml}
<br>&lt;!-- define a bean of type StreamCachingStrategy which CamelContext will
automatic use --&gt; <br>&lt;bean id=&quot;streamStrategy&quot; class=&quot;org.apache.camel.impl.DefaultStreamCachingStrategy&quot;&gt;
<br>  &lt;property name=&quot;temporaryDirectory&quot; value=&quot;/tmp/cachedir&quot;/&gt;
<br>  &lt;property name=&quot;spoolThreshold&quot; value=&quot;65536&quot;/&gt;
<br>  &lt;property name=&quot;bufferSize&quot; value=&quot;16384&quot;/&gt;
<br>&lt;/bean&gt; <br> <br>&lt;!-- remember to enable stream
caching --&gt; <br>&lt;camelContext streamCaching=&quot;true&quot; xmlns=&quot;http://camel.apache.org/schema/spring&quot;&gt;
<br>{code} <br> <br> <br></td></tr>
            <tr><td class="diff-unchanged" >h3. How it works? <br>In order
to determine if a type requires caching, we leverage the [type converter | CAMEL:Type Converter]
feature.  Any type that requires stream caching can be converted into an *{{org.apache.camel.StreamCache}}*
instance. <br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h2><a name="Streamcaching-Streamcaching"></a>Stream caching</h2>
<p>While stream types (like StreamSource, InputStream and Reader) are commonly used
in messaging for performance reasons, they also have an important drawback: they can only
be read once.  In order to be able to work with message content multiple times, the stream
needs to be cached.</p>

<p>Streams are caching in memory. In Camel 2.0, large stream messages (over 64 Kb in
Camel 2.11 or older, and 128 kb from Camel 2.12 onwards) will be cached in a temporary file
instead &#8211; Camel itself will handle deleting the temporary file once the cached stream
is no longer necessary.</p>

<p>In Camel 2.0 stream cache is default <b>disabled</b> out of the box.<br/>
In Camel 1.x stream cache is default <b>enabled</b> out of the box.</p>

<div class='panelMacro'><table class='infoMacro'><colgroup><col width='24'><col></colgroup><tr><td
valign='top'><img src="/confluence/images/icons/emoticons/information.gif" width="16"
height="16" align="absmiddle" alt="" border="0"></td><td><b>StreamCache
Affects your payload object</b><br />The StreamCache will affect your payload
object as it will replace the Stream payload with a <tt>org.apache.camel.StreamCache</tt>
object.<br/>
This <tt>StreamCache</tt> is capable of being re-readable and thus possible to
better be routed within Camel using redelivery or <a href="/confluence/display/CAMEL/Content+Based+Router"
title="Content Based Router">Content Based Router</a> or the likes.

<p>However to not change the payload under the covers without the end user really knowing
we changed the default in Camel 2.0 to <b>disabled</b>. So in Camel 2.0 you have
to explicit enable it if you want to use it.</p></td></tr></table></div>

<div class='panelMacro'><table class='tipMacro'><colgroup><col width='24'><col></colgroup><tr><td
valign='top'><img src="/confluence/images/icons/emoticons/check.gif" width="16" height="16"
align="absmiddle" alt="" border="0"></td><td>If using Camel 2.12 onwards then
see about StreamCachingStrategy further below which is the recommended way to configure stream
caching options.</td></tr></table></div>

<h3><a name="Streamcaching-Enablingstreamcaching"></a>Enabling stream caching</h3>

<p>In Apache Camel, you can explicitly enable stream caching for a single route with
the <b><tt>streamCaching</tt></b> DSL method:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
from("jbi:service:http://foo.bar.org/MyService")
    .streamCaching()
    .to("jbi:service:http://foo.bar.org/MyOtherService");
</pre>
</div></div>

<p>In Spring XML you enable it by setting the <tt>streamCache="true"</tt>
attribute on the <tt>route</tt> tag.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;route streamCache="true"&gt;
   &lt;from uri="jbi:service:http://foo.bar.org/MyService"/&gt;
   &lt;to uri="jbi:service:http://foo.bar.org/MyOtherService"/&gt;
&lt;/route&gt;
</pre>
</div></div>

<h3><a name="Streamcaching-Scopes"></a>Scopes</h3>
<p>StreamCache supports the global and per route scope. So by setting the streamCache
attribute on camelContext you can enable/disable it globally.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;camelContext streamCache="true"&gt;
   ...
&lt;/camelContext&gt;
</pre>
</div></div>

<p>The route scope is configured by the <tt>streamCache</tt> attribute on
the <tt>&lt;route&gt;</tt> tag such as:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;route streamCache="true"&gt;
   &lt;from uri="jbi:service:http://foo.bar.org/MyService"/&gt;
   &lt;to uri="jbi:service:http://foo.bar.org/MyOtherService"/&gt;
&lt;/route&gt;
</pre>
</div></div>

<p>You can mix and match for instance you can enable it globally and disable it on a
particular route such as:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;camelContext streamCache="true"&gt;
  &lt;route&gt;
    &lt;from uri="jbi:service:http://foo.bar.org/MyService"/&gt;
    &lt;to uri="jbi:service:http://foo.bar.org/MyOtherService"/&gt;
  &lt;/route&gt;
  
  &lt;route streamCache="false"&gt;
    &lt;from uri="jms:queue:foo"/&gt;
    &lt;to uri="jms:queue:bar"/&gt;
  &lt;/route&gt;

&lt;/camelContext&gt;
</pre>
</div></div>

<h4><a name="Streamcaching-EnablingfromJavaDSL"></a>Enabling from Java DSL</h4>
<p>You can enable stream cache by setting the property on CamelContext, for instance
in a RouteBuilder class:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
  context.setStreamCache(true);

  from("jbi:service:http://foo.bar.org/MyService")
     .to("jbi:service:http://foo.bar.org/MyOtherService");
</pre>
</div></div>

<h3><a name="Streamcaching-Disablestreamcachingexplicitly"></a>Disable stream
caching explicitly</h3>
<p>If you have enabled stream caching globally on the CamelContext and you want to disable
it for certain routes in a selective manner, you can use the following syntax.</p>

<p><b>Spring DSL:</b></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;camelContext streamCache="true"&gt;
  &lt;route streamCache="false"&gt;
    &lt;from uri="jetty:http://0.0.0.0:9090"/&gt;
    &lt;to uri="file:target/incoming"/&gt;
  &lt;/route&gt;
&lt;/camelContext&gt;
</pre>
</div></div>

<p><b>Java DSL:</b></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
  context.setStreamCache(true);

  from("jetty:http://0.0.0.0:9090").noStreamCaching()
     .to("file:target/incoming");
</pre>
</div></div>


<h3><a name="Streamcaching-Streamingcachetofiles"></a>Streaming cache to
files</h3>
<p>When stream cache is enabled it will by default spool big streams to files instead
of keeping them in memory. The default threshold is 64kb but you can configure it with the
following properties:</p>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Property </th>
<th class='confluenceTh'> Default </th>
<th class='confluenceTh'> Description </th>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamBufferSize </td>
<td class='confluenceTd'> 2kb </td>
<td class='confluenceTd'> <b>Camel 2.9.4, 2.10.2, 2.11.0:</b> Size in bytes
of the buffer used in the stream. </td>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamThreshold </td>
<td class='confluenceTd'> 64kb or 128kb </td>
<td class='confluenceTd'> 64kb for <b>Camel 2.11</b> or older. 128kb for
<b>Camel 2.12</b> onwards. Size in bytes when the stream should be spooled to
disk instead of keeping in memory. Use a value of 0 or negative to disable it all together
so streams is always kept in memory regardless of their size. </td>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamOutputDirectory </td>
<td class='confluenceTd'> java.io.tmpdir </td>
<td class='confluenceTd'> Base directory where temporary files for spooled streams should
be stored. </td>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamCipherTransformation </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> <b>Camel 2.11.0:</b> If set, the temporary files
are encrypted using the specified cipher transformation (i.e., a valid stream or 8-bit cipher
name such as "RC4", "AES/CTR/NoPadding". An empty name "" is treated as null).</td>
</tr>
</tbody></table>
</div>


<p>You set these properties on the CamelContext as shown below, where we use a 1mb threshold
to spool to disk for messages bigger than 1mb:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
context.getProperties().put(CachedOutputStream.TEMP_DIR, "/tmp/cachedir");
context.getProperties().put(CachedOutputStream.THRESHOLD, "1048576");
context.getProperties().put(CachedOutputStream.BUFFER_SIZE, "131072");
// to enable encryption using RC4
// context.getProperties().put(CachedOutputStream.CIPHER_TRANSFORMATION, "RC4");
</pre>
</div></div>

<p>And in XML you do</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;camelContext xmlns="http://camel.apache.org/schema/blueprint"&gt;

  &lt;!-- disable stream caching spool to disk --&gt;
  &lt;properties&gt;
    &lt;property key="CamelCachedOutputStreamOutputDirectory" value="/tmp/cachedir"/&gt;
    &lt;property key="CamelCachedOutputStreamThreshold" value="1048576"/&gt;
    &lt;property key="CamelCachedOutputStreamBufferSize" value="131072"/&gt;
  &lt;/properties&gt;
</pre>
</div></div>


<h4><a name="Streamcaching-Disablingspoolingtodisk"></a>Disabling spooling
to disk</h4>
<p>You can disable spooling to disk by setting a threshold of 0 or a negative value.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
// disable spooling to disk
context.getProperties().put(CachedOutputStream.THRESHOLD, "-1");
</pre>
</div></div>

<p>And in XML you do</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;camelContext xmlns="http://camel.apache.org/schema/blueprint"&gt;

  &lt;!-- disable stream caching spool to disk --&gt;
  &lt;properties&gt;
    &lt;property key="CamelCachedOutputStreamThreshold" value="-1"/&gt;
  &lt;/properties&gt;
</pre>
</div></div>


<h3><a name="Streamcaching-UsingStreamCachingStrategy"></a>Using StreamCachingStrategy</h3>
<p><b>Available as of Camel 2.12</b></p>

<p><a href="/confluence/display/CAMEL/Stream+caching" title="Stream caching">Stream
caching</a> is from Camel 2.12 onwards intended to be configured using <tt>org.apache.camel.spi.StreamCachingStrategy</tt>.<br/>
The old kind of configuration using properties on the <a href="/confluence/display/CAMEL/CamelContext"
title="CamelContext">CamelContext</a> has been marked as deprecated.</p>

<p>The strategy has the following options:</p>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Option </th>
<th class='confluenceTh'> Default </th>
<th class='confluenceTh'> Description </th>
</tr>
<tr>
<td class='confluenceTd'> temporaryDirectory </td>
<td class='confluenceTd'> java.io.tmpdir </td>
<td class='confluenceTd'> Base directory where temporary files for spooled streams should
be stored. </td>
</tr>
<tr>
<td class='confluenceTd'> spoolThreshold </td>
<td class='confluenceTd'> 128kb </td>
<td class='confluenceTd'> Size in bytes when the stream should be spooled to disk instead
of keeping in memory. Use a value of 0 or negative to disable it all together so streams is
always kept in memory regardless of their size. </td>
</tr>
<tr>
<td class='confluenceTd'> spoolChiper </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> If set, the temporary files are encrypted using the specified
cipher transformation (i.e., a valid stream or 8-bit cipher name such as "RC4", "AES/CTR/NoPadding".
An empty name "" is treated as null).</td>
</tr>
<tr>
<td class='confluenceTd'> bufferSize </td>
<td class='confluenceTd'> 4096 </td>
<td class='confluenceTd'> Size in bytes of the buffer used in the stream. </td>
</tr>
<tr>
<td class='confluenceTd'> removeTemporaryDirectoryWhenStopping </td>
<td class='confluenceTd'> true </td>
<td class='confluenceTd'> Whether to remove the temporary directory when stopping <a
href="/confluence/display/CAMEL/CamelContext" title="CamelContext">CamelContext</a>.
</td>
</tr>
</tbody></table>
</div>


<h4><a name="Streamcaching-UsingStreamCachingStrategyinJava"></a>Using StreamCachingStrategy
in Java</h4>

<p>You can configure the <tt>StreamCachingStrategy</tt> in Java as shown
below:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
context.getStreamCachingStrategy().setTemporaryDirectory"/tmp/cachedir");
context.getStreamCachingStrategy().setSpoolThreshold(64 * 1024);
context.getStreamCachingStrategy().setBufferSize(16 * 1024);
// to enable encryption using RC4
// context.getStreamCachingStrategy().setSpoolChiper("RC4");
</pre>
</div></div>

<p>And remember to enable <a href="/confluence/display/CAMEL/Stream+caching" title="Stream
caching">Stream caching</a> on the <a href="/confluence/display/CAMEL/CamelContext"
title="CamelContext">CamelContext</a> or on routes</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
context.setStreamCaching(true);
</pre>
</div></div>

<h4><a name="Streamcaching-UsingStreamCachingStrategyinXML"></a>Using StreamCachingStrategy
in XML</h4>

<p>And in XML you do</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: xml; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;!-- define a bean of type StreamCachingStrategy which CamelContext will automatic
use --&gt;
&lt;bean id="streamStrategy" class="org.apache.camel.impl.DefaultStreamCachingStrategy"&gt;
  &lt;property name="temporaryDirectory" value="/tmp/cachedir"/&gt;
  &lt;property name="spoolThreshold" value="65536"/&gt;
  &lt;property name="bufferSize" value="16384"/&gt;
&lt;/bean&gt;

&lt;!-- remember to enable stream caching --&gt;
&lt;camelContext streamCaching="true" xmlns="http://camel.apache.org/schema/spring"&gt;
</pre>
</div></div>


<h3><a name="Streamcaching-Howitworks%3F"></a>How it works?</h3>
<p>In order to determine if a type requires caching, we leverage the <a href="/confluence/display/CAMEL/Type+Converter"
title="Type Converter">type converter </a> feature.  Any type that requires stream
caching can be converted into an <b><tt>org.apache.camel.StreamCache</tt></b>
instance.</p>
    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;" class="grey">
                        <a href="https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=CAMEL">Stop
watching space</a>
            <span style="padding: 0px 5px;">|</span>
                <a href="https://cwiki.apache.org/confluence/users/editmyemailsettings.action">Change
email notification preferences</a>
</div>
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/Stream+caching">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=112456&revisedVersion=17&originalVersion=16">View
Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/Stream+caching?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message