camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Camel > Stream caching
Date Fri, 21 Jun 2013 09:29:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/2042/9/1/_/styles/combined.css?spaceKey=CAMEL&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/CAMEL/Stream+caching">Stream
caching</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~davsclaus">Claus
Ibsen</a>
    </h4>
        <br/>
                         <h4>Changes (7)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >| CamelCachedOutputStreamCipherTransformation
| null | *Camel 2.11.0:* If set, the temporary files are encrypted using the specified cipher
transformation (i.e., a valid stream or 8-bit cipher name such as &quot;RC4&quot;,
&quot;AES/CTR/NoPadding&quot;. An empty name &quot;&quot; is treated as null).|
<br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">You
set these properties on the CamelContext as shown below: <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">You
set these properties on the CamelContext as shown below, where we use a 1mb threshold to spool
to disk for messages bigger than 1mb: <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{code:java} <br>context.getProperties().put(CachedOutputStream.TEMP_DIR,
&quot;/tmp/cachedir&quot;); <br></td></tr>
            <tr><td class="diff-changed-lines" >context.getProperties().put(CachedOutputStream.THRESHOLD,
<span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">&quot;1024&quot;);</span>
<span class="diff-added-words"style="background-color: #dfd;">&quot;1048576&quot;);</span>
<br></td></tr>
            <tr><td class="diff-changed-lines" >context.getProperties().put(CachedOutputStream.BUFFER_SIZE,
<span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">&quot;4096&quot;);</span>
<span class="diff-added-words"style="background-color: #dfd;">&quot;131072&quot;);</span>
<br></td></tr>
            <tr><td class="diff-unchanged" >// to enable encryption using RC4
<br>// context.getProperties().put(CachedOutputStream.CIPHER_TRANSFORMATION, &quot;RC4&quot;);
<br>{code} <br> <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">And
in XML you do <br>{code:xml} <br>&lt;camelContext xmlns=&quot;http://camel.apache.org/schema/blueprint&quot;&gt;
<br> <br>  &lt;!-- disable stream caching spool to disk --&gt; <br>
 &lt;properties&gt; <br>    &lt;property key=&quot;CamelCachedOutputStreamOutputDirectory&quot;
value=&quot;/tmp/cachedir&quot;/&gt; <br>    &lt;property key=&quot;CamelCachedOutputStreamThreshold&quot;
value=&quot;1048576&quot;/&gt; <br>    &lt;property key=&quot;CamelCachedOutputStreamBufferSize&quot;
value=&quot;131072&quot;/&gt; <br>  &lt;/properties&gt; <br>{code}
<br> <br> <br></td></tr>
            <tr><td class="diff-unchanged" >h4. Disabling spooling to disk <br>You
can disable spooling to disk by setting a threshold of 0 or a negative value. <br>{code}
<br>// disable spooling to disk <br></td></tr>
            <tr><td class="diff-changed-lines" >context.getProperties().put(CachedOutputStream.THRESHOLD,
<span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">&quot;0&quot;);</span>
<span class="diff-added-words"style="background-color: #dfd;">&quot;-1&quot;);</span>
<br></td></tr>
            <tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">And
in XML you do <br>{code:xml} <br>&lt;camelContext xmlns=&quot;http://camel.apache.org/schema/blueprint&quot;&gt;
<br> <br>  &lt;!-- disable stream caching spool to disk --&gt; <br>
 &lt;properties&gt; <br>    &lt;property key=&quot;CamelCachedOutputStreamThreshold&quot;
value=&quot;-1&quot;/&gt; <br>  &lt;/properties&gt; <br>{code}
<br> <br> <br></td></tr>
            <tr><td class="diff-unchanged" >h3. How it works? <br>In order
to determine if a type requires caching, we leverage the [type converter | CAMEL:Type Converter]
feature.  Any type that requires stream caching can be converted into an *{{org.apache.camel.StreamCache}}*
instance. <br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h2><a name="Streamcaching-Streamcaching"></a>Stream caching</h2>
<p>While stream types (like StreamSource, InputStream and Reader) are commonly used
in messaging for performance reasons, they also have an important drawback: they can only
be read once.  In order to be able to work with message content multiple times, the stream
needs to be cached.</p>

<p>Streams are caching in memory. In Camel 2.0, large stream messages (over 64 Kb in
Camel 2.11 or older, and 128 kb from Camel 2.12 onwards) will be cached in a temporary file
instead &#8211; Camel itself will handle deleting the temporary file once the cached stream
is no longer necessary.</p>

<p>In Camel 2.0 stream cache is default <b>disabled</b> out of the box.<br/>
In Camel 1.x stream cache is default <b>enabled</b> out of the box.</p>

<div class='panelMacro'><table class='infoMacro'><colgroup><col width='24'><col></colgroup><tr><td
valign='top'><img src="/confluence/images/icons/emoticons/information.gif" width="16"
height="16" align="absmiddle" alt="" border="0"></td><td><b>StreamCache
Affects your payload object</b><br />The StreamCache will affect your payload
object as it will replace the Stream payload with a <tt>org.apache.camel.StreamCache</tt>
object.<br/>
This <tt>StreamCache</tt> is capable of being re-readable and thus possible to
better be routed within Camel using redelivery or <a href="/confluence/display/CAMEL/Content+Based+Router"
title="Content Based Router">Content Based Router</a> or the likes.

<p>However to not change the payload under the covers without the end user really knowing
we changed the default in Camel 2.0 to <b>disabled</b>. So in Camel 2.0 you have
to explicit enable it if you want to use it.</p></td></tr></table></div>

<h3><a name="Streamcaching-Enablingstreamcaching"></a>Enabling stream caching</h3>

<p>In Apache Camel, you can explicitly enable stream caching for a single route with
the <b><tt>streamCaching</tt></b> DSL method:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
from(<span class="code-quote">"jbi:service:http:<span class="code-comment">//foo.bar.org/MyService"</span>)
</span>    .streamCaching()
    .to(<span class="code-quote">"jbi:service:http:<span class="code-comment">//foo.bar.org/MyOtherService"</span>);</span>
</pre>
</div></div>

<p>In Spring XML you enable it by setting the <tt>streamCache="true"</tt>
attribute on the <tt>route</tt> tag.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;route streamCache=<span class="code-quote">"true"</span>&gt;</span>
   <span class="code-tag">&lt;from uri=<span class="code-quote">"jbi:service:http://foo.bar.org/MyService"</span>/&gt;</span>
   <span class="code-tag">&lt;to uri=<span class="code-quote">"jbi:service:http://foo.bar.org/MyOtherService"</span>/&gt;</span>
<span class="code-tag">&lt;/route&gt;</span>
</pre>
</div></div>

<h3><a name="Streamcaching-Scopes"></a>Scopes</h3>
<p>StreamCache supports the global and per route scope. So by setting the streamCache
attribute on camelContext you can enable/disable it globally.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;camelContext streamCache=<span class="code-quote">"true"</span>&gt;</span>
   ...
<span class="code-tag">&lt;/camelContext&gt;</span>
</pre>
</div></div>

<p>The route scope is configured by the <tt>streamCache</tt> attribute on
the <tt>&lt;route&gt;</tt> tag such as:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;route streamCache=<span class="code-quote">"true"</span>&gt;</span>
   <span class="code-tag">&lt;from uri=<span class="code-quote">"jbi:service:http://foo.bar.org/MyService"</span>/&gt;</span>
   <span class="code-tag">&lt;to uri=<span class="code-quote">"jbi:service:http://foo.bar.org/MyOtherService"</span>/&gt;</span>
<span class="code-tag">&lt;/route&gt;</span>
</pre>
</div></div>

<p>You can mix and match for instance you can enable it globally and disable it on a
particular route such as:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;camelContext streamCache=<span class="code-quote">"true"</span>&gt;</span>
  <span class="code-tag">&lt;route&gt;</span>
    <span class="code-tag">&lt;from uri=<span class="code-quote">"jbi:service:http://foo.bar.org/MyService"</span>/&gt;</span>
    <span class="code-tag">&lt;to uri=<span class="code-quote">"jbi:service:http://foo.bar.org/MyOtherService"</span>/&gt;</span>
  <span class="code-tag">&lt;/route&gt;</span>
  
  <span class="code-tag">&lt;route streamCache=<span class="code-quote">"false"</span>&gt;</span>
    <span class="code-tag">&lt;from uri=<span class="code-quote">"jms:queue:foo"</span>/&gt;</span>
    <span class="code-tag">&lt;to uri=<span class="code-quote">"jms:queue:bar"</span>/&gt;</span>
  <span class="code-tag">&lt;/route&gt;</span>

<span class="code-tag">&lt;/camelContext&gt;</span>
</pre>
</div></div>

<h4><a name="Streamcaching-EnablingfromJavaDSL"></a>Enabling from Java DSL</h4>
<p>You can enable stream cache by setting the property on CamelContext, for instance
in a RouteBuilder class:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
  context.setStreamCache(<span class="code-keyword">true</span>);

  from(<span class="code-quote">"jbi:service:http:<span class="code-comment">//foo.bar.org/MyService"</span>)
</span>     .to(<span class="code-quote">"jbi:service:http:<span class="code-comment">//foo.bar.org/MyOtherService"</span>);</span>
</pre>
</div></div>

<h3><a name="Streamcaching-Disablestreamcachingexplicitly"></a>Disable stream
caching explicitly</h3>
<p>If you have enabled stream caching globally on the CamelContext and you want to disable
it for certain routes in a selective manner, you can use the following syntax.</p>

<p><b>Spring DSL:</b></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;camelContext streamCache=<span class="code-quote">"true"</span>&gt;</span>
  <span class="code-tag">&lt;route streamCache=<span class="code-quote">"false"</span>&gt;</span>
    <span class="code-tag">&lt;from uri=<span class="code-quote">"jetty:http://0.0.0.0:9090"</span>/&gt;</span>
    <span class="code-tag">&lt;to uri=<span class="code-quote">"file:target/incoming"</span>/&gt;</span>
  <span class="code-tag">&lt;/route&gt;</span>
<span class="code-tag">&lt;/camelContext&gt;</span>
</pre>
</div></div>

<p><b>Java DSL:</b></p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
  context.setStreamCache(<span class="code-keyword">true</span>);

  from(<span class="code-quote">"jetty:http:<span class="code-comment">//0.0.0.0:9090"</span>).noStreamCaching()
</span>     .to(<span class="code-quote">"file:target/incoming"</span>);
</pre>
</div></div>


<h3><a name="Streamcaching-Streamingcachetofiles"></a>Streaming cache to
files</h3>
<p>When stream cache is enabled it will by default spool big streams to files instead
of keeping them in memory. The default threshold is 64kb but you can configure it with the
following properties:</p>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Property </th>
<th class='confluenceTh'> Default </th>
<th class='confluenceTh'> Description </th>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamBufferSize </td>
<td class='confluenceTd'> 2kb </td>
<td class='confluenceTd'> <b>Camel 2.9.4, 2.10.2, 2.11.0:</b> Size in bytes
of the buffer used in the stream. </td>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamThreshold </td>
<td class='confluenceTd'> 64kb or 128kb </td>
<td class='confluenceTd'> 64kb for <b>Camel 2.11</b> or older. 128kb for
<b>Camel 2.12</b> onwards. Size in bytes when the stream should be spooled to
disk instead of keeping in memory. Use a value of 0 or negative to disable it all together
so streams is always kept in memory regardless of their size. </td>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamOutputDirectory </td>
<td class='confluenceTd'> java.io.tmpdir </td>
<td class='confluenceTd'> Base directory where temporary files for spooled streams should
be stored. </td>
</tr>
<tr>
<td class='confluenceTd'> CamelCachedOutputStreamCipherTransformation </td>
<td class='confluenceTd'> null </td>
<td class='confluenceTd'> <b>Camel 2.11.0:</b> If set, the temporary files
are encrypted using the specified cipher transformation (i.e., a valid stream or 8-bit cipher
name such as "RC4", "AES/CTR/NoPadding". An empty name "" is treated as null).</td>
</tr>
</tbody></table>
</div>


<p>You set these properties on the CamelContext as shown below, where we use a 1mb threshold
to spool to disk for messages bigger than 1mb:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
context.getProperties().put(CachedOutputStream.TEMP_DIR, <span class="code-quote">"/tmp/cachedir"</span>);
context.getProperties().put(CachedOutputStream.THRESHOLD, <span class="code-quote">"1048576"</span>);
context.getProperties().put(CachedOutputStream.BUFFER_SIZE, <span class="code-quote">"131072"</span>);
<span class="code-comment">// to enable encryption using RC4
</span><span class="code-comment">// context.getProperties().put(CachedOutputStream.CIPHER_TRANSFORMATION,
<span class="code-quote">"RC4"</span>);</span>
</pre>
</div></div>

<p>And in XML you do</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;camelContext xmlns=<span class="code-quote">"http://camel.apache.org/schema/blueprint"</span>&gt;</span>

  <span class="code-tag"><span class="code-comment">&lt;!-- disable stream
caching spool to disk --&gt;</span></span>
  <span class="code-tag">&lt;properties&gt;</span>
    <span class="code-tag">&lt;property key=<span class="code-quote">"CamelCachedOutputStreamOutputDirectory"</span>
value=<span class="code-quote">"/tmp/cachedir"</span>/&gt;</span>
    <span class="code-tag">&lt;property key=<span class="code-quote">"CamelCachedOutputStreamThreshold"</span>
value=<span class="code-quote">"1048576"</span>/&gt;</span>
    <span class="code-tag">&lt;property key=<span class="code-quote">"CamelCachedOutputStreamBufferSize"</span>
value=<span class="code-quote">"131072"</span>/&gt;</span>
  <span class="code-tag">&lt;/properties&gt;</span>
</pre>
</div></div>


<h4><a name="Streamcaching-Disablingspoolingtodisk"></a>Disabling spooling
to disk</h4>
<p>You can disable spooling to disk by setting a threshold of 0 or a negative value.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-java">
<span class="code-comment">// disable spooling to disk
</span>context.getProperties().put(CachedOutputStream.THRESHOLD, <span class="code-quote">"-1"</span>);
</pre>
</div></div>

<p>And in XML you do</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="code-xml">
<span class="code-tag">&lt;camelContext xmlns=<span class="code-quote">"http://camel.apache.org/schema/blueprint"</span>&gt;</span>

  <span class="code-tag"><span class="code-comment">&lt;!-- disable stream
caching spool to disk --&gt;</span></span>
  <span class="code-tag">&lt;properties&gt;</span>
    <span class="code-tag">&lt;property key=<span class="code-quote">"CamelCachedOutputStreamThreshold"</span>
value=<span class="code-quote">"-1"</span>/&gt;</span>
  <span class="code-tag">&lt;/properties&gt;</span>
</pre>
</div></div>


<h3><a name="Streamcaching-Howitworks%3F"></a>How it works?</h3>
<p>In order to determine if a type requires caching, we leverage the <a href="/confluence/display/CAMEL/Type+Converter"
title="Type Converter">type converter </a> feature.  Any type that requires stream
caching can be converted into an <b><tt>org.apache.camel.StreamCache</tt></b>
instance.</p>
    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;">
            <a href="https://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
        </div>
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/Stream+caching">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=112456&revisedVersion=16&originalVersion=15">View
Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/Stream+caching?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message