camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Babak Vahdat (Confluence)" <conflue...@apache.org>
Subject [CONF] Apache Camel > ETL Example
Date Mon, 21 Oct 2013 11:32:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/en/2176/1/1/_/styles/combined.css?spaceKey=CAMEL&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/CAMEL/ETL+Example">ETL
Example</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~bvahdat">Babak
Vahdat</a>
    </h4>
        <br/>
                         <h4>Changes (10)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br>|| Java Class || Purpose
|| <br></td></tr>
            <tr><td class="diff-changed-lines" >| <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[CustomerEntity|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[CustomerEntity|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java]</span>
| The JPA entity bean (i.e. a POJO with @Entity) | <br></td></tr>
            <tr><td class="diff-changed-lines" >| <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[PersonDocument|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[PersonDocument|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java]</span>
| The JAXB2 POJO used to parse the XML | <br></td></tr>
            <tr><td class="diff-changed-lines" >| <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[CustomerTransformer|http://svn.apache.org/repos/asf/acamel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[CustomerTransformer|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java]</span>
| The custom [Type Converter] used to convert a PersonDocument into a CustomerEntity | <br></td></tr>
            <tr><td class="diff-changed-lines" >| <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[EtlRoutes|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[EtlRoutes|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java]</span>
| The Camel routing [DSL] to define the flow from the files to the converter to the [JPA]
endpoint | <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-changed-lines" >The there is the spring configuration
file in <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[src/resources/META-INF/services/camel-context.xml|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/resources/META-INF/spring/camel-context.xml]</span>
<span class="diff-added-words"style="background-color: #dfd;">[src/resources/META-INF/services/camel-context.xml|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/resources/META-INF/spring/camel-context.xml]</span>
which defines the JPA template and tells Camel to look in the *org.apache.camel.example.etl*
package to find its routes. <br></td></tr>
            <tr><td class="diff-unchanged" > <br>h3. Code walkthrough  <br>
<br></td></tr>
            <tr><td class="diff-changed-lines" >So lets start with the route definition
in <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[EtlRoutes|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[EtlRoutes|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java]</span>
<br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">{snippet:id=example|lang=java|url=camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java}</span>
<span class="diff-added-words"style="background-color: #dfd;">{snippet:id=example|lang=java|url=https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java}</span>
<br></td></tr>
            <tr><td class="diff-unchanged" > <br>The above sets up a route
from the _src/data_ directory. Notice we&#39;re using the _noop_ mode of the [File|File2]
component so that the files are not moved or deleted when they are processed (so when the
tool is restarted they will be processed again). <br> <br></td></tr>
            <tr><td class="diff-changed-lines" >We&#39;re converting the body
of the message to a <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[PersonDocument|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[PersonDocument|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java]</span>
which since this POJO as an @XmlRootElement annotation from [JAXB] will kick in the [Type
Converter] to use JAXB to unmarshall the object. <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-changed-lines" >Then we send the message with
a PersonDocument body to the [JPA] endpoint. Notice how this endpoint specifies the expected
type. So the [Type Converter] is gonna try convert the PersonDocument to a <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[CustomerEntity|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java].</span>
<span class="diff-added-words"style="background-color: #dfd;">[CustomerEntity|hhttps://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java].</span>
Here Camel will find the <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">[CustomerTransformer|http://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java]</span>
<span class="diff-added-words"style="background-color: #dfd;">[CustomerTransformer|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java]</span>
class which has an @Converter method <br></td></tr>
            <tr><td class="diff-unchanged" > <br>{snippet:id=example|lang=java|url=camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java}
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">Please
note that when you run the example for the first time, the converter {{CustomerTransformer}}
will not be able to find any entities inside the database, so that along the logs written
into the console you should see: <br> <br>{code} <br>thread #0 - file://src/data]
CustomerTransformer            INFO  Created a new CustomerEntity Customer[userName: james
firstName: null surname: null] as no matching persisted entity found. <br>thread #0
- file://src/data] CustomerTransformer            INFO  Created a new CustomerEntity Customer[userName:
hiram firstName: null surname: null] as no matching persisted entity found. <br>{code}
<br> <br>However running the example for a second time, as the entites have been
already inserted into the database, the log should now say: <br> <br>{code} <br>thread
#0 - file://src/data] CustomerTransformer            INFO  Found a matching CustomerEntity
Customer[userName: james firstName: James surname: Strachan] having the userName james. <br>thread
#0 - file://src/data] CustomerTransformer            INFO  Found a matching CustomerEntity
Customer[userName: hiram firstName: Hiram surname: Chirino] having the userName hiram. <br>{code}
<br> <br></td></tr>
            <tr><td class="diff-unchanged" >Failing that you can run the Main
from inside your IDE if you prefer. Follow the [Building] instructions to create an Eclipse/IDEA
project to import <br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h2><a name="ETLExample-ExtractTransformLoad%28ETL%29Example"></a>Extract
Transform Load (ETL) Example </h2>

<p>The <a href="/confluence/display/CAMEL/ETL" title="ETL">ETL</a> (Extract,
Transform, Load) example shows how to load data into a database using Camel. In this example
we will poll for files, transform them and then store them in the database via the <a href="/confluence/display/CAMEL/JPA"
title="JPA">JPA</a> component.</p>

<h3><a name="ETLExample-Overview"></a>Overview</h3>

<p>This example lives in the <em>examples/camel-example-etl</em> directory
and will poll for XML files in the child <em>src/data</em> directory. When the
files are detected, they are converted, using the fallback <a href="/confluence/display/CAMEL/JAXB"
title="JAXB">JAXB</a> <a href="/confluence/display/CAMEL/Type+Converter" title="Type
Converter">Type Converter</a> to a PersonDocument class. This POJO is then transformed
using a custom <a href="/confluence/display/CAMEL/Type+Converter" title="Type Converter">Type
Converter</a> into a CustomerEntity bean using the CustomerTransformer class. Then this
bean is stored in the database via the <a href="/confluence/display/CAMEL/JPA" title="JPA">JPA</a>
component.</p>

<p>The code for this example is as follows</p>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Java Class </th>
<th class='confluenceTh'> Purpose </th>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java"
class="external-link" rel="nofollow">CustomerEntity</a> </td>
<td class='confluenceTd'> The JPA entity bean (i.e. a POJO with @Entity) </td>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java"
class="external-link" rel="nofollow">PersonDocument</a> </td>
<td class='confluenceTd'> The JAXB2 POJO used to parse the XML </td>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java"
class="external-link" rel="nofollow">CustomerTransformer</a> </td>
<td class='confluenceTd'> The custom <a href="/confluence/display/CAMEL/Type+Converter"
title="Type Converter">Type Converter</a> used to convert a PersonDocument into a
CustomerEntity </td>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java"
class="external-link" rel="nofollow">EtlRoutes</a> </td>
<td class='confluenceTd'> The Camel routing <a href="/confluence/display/CAMEL/DSL"
title="DSL">DSL</a> to define the flow from the files to the converter to the <a
href="/confluence/display/CAMEL/JPA" title="JPA">JPA</a> endpoint </td>
</tr>
</tbody></table>
</div>


<p>The there is the spring configuration file in <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/resources/META-INF/spring/camel-context.xml"
class="external-link" rel="nofollow">src/resources/META-INF/services/camel-context.xml</a>
which defines the JPA template and tells Camel to look in the <b>org.apache.camel.example.etl</b>
package to find its routes.</p>

<h3><a name="ETLExample-Codewalkthrough"></a>Code walkthrough </h3>

<p>So lets start with the route definition in <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java"
class="external-link" rel="nofollow">EtlRoutes</a></p>

<div class="error"><span class="error">Error formatting macro: snippet: java.lang.IllegalArgumentException:
Invalid url: must begin with a configured prefix.</span> </div>

<p>The above sets up a route from the <em>src/data</em> directory. Notice
we're using the <em>noop</em> mode of the <a href="/confluence/display/CAMEL/File2"
title="File2">File</a> component so that the files are not moved or deleted when
they are processed (so when the tool is restarted they will be processed again).</p>

<p>We're converting the body of the message to a <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java"
class="external-link" rel="nofollow">PersonDocument</a> which since this POJO as
an @XmlRootElement annotation from <a href="/confluence/display/CAMEL/JAXB" title="JAXB">JAXB</a>
will kick in the <a href="/confluence/display/CAMEL/Type+Converter" title="Type Converter">Type
Converter</a> to use JAXB to unmarshall the object.</p>

<p>Then we send the message with a PersonDocument body to the <a href="/confluence/display/CAMEL/JPA"
title="JPA">JPA</a> endpoint. Notice how this endpoint specifies the expected type.
So the <a href="/confluence/display/CAMEL/Type+Converter" title="Type Converter">Type
Converter</a> is gonna try convert the PersonDocument to a <a href="hhttps://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java"
class="external-link" rel="nofollow">CustomerEntity</a>. Here Camel will find the
<a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java"
class="external-link" rel="nofollow">CustomerTransformer</a> class which has an @Converter
method</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<script type="syntaxhighlighter" class="theme: Default; brush: java; gutter: false"><![CDATA[
@Converter
public class CustomerTransformer {

    private static final transient Logger LOG = LoggerFactory.getLogger(CustomerTransformer.class);

    /**
     * A transformation method to convert a person document into a customer
     * entity
     * @throws Exception 
     */
    @Converter
    public CustomerEntity toCustomer(PersonDocument doc, Exchange exchange) throws Exception
{
        JpaTemplate template = exchange.getIn().getHeader("CamelJpaTemplate", JpaTemplate.class);

        
        String user = doc.getUser();
        CustomerEntity customer = findCustomerByName(template, user);

        // let's convert information from the document into the entity bean
        customer.setUserName(user);
        customer.setFirstName(doc.getFirstName());
        customer.setSurname(doc.getLastName());
        customer.setCity(doc.getCity());

        LOG.debug("Created object customer: " + customer);
        return customer;
    }

    /**
     * Finds a customer for the given username
     */
    protected CustomerEntity findCustomerByName(JpaTemplate template, String user) throws
Exception {
        List&lt;CustomerEntity&gt; list = CastUtils.cast(template.find("select x from
customer"
                                                                 + " x where x.userName =
?1", user));
        if (list.isEmpty()) {
            CustomerEntity answer = new CustomerEntity();
            answer.setUserName(user);
            return answer;
        } else {
            return list.get(0);
        }
    }

}
]]></script>
</div></div>

<p>which performs the necessary conversion to an entity bean which is then stored in
the database</p>

<h3><a name="ETLExample-Runningtheexample"></a>Running the example</h3>

<p>To run the example we use the <a href="/confluence/display/CAMEL/Camel+Maven+Plugin"
title="Camel Maven Plugin">Camel Maven Plugin</a>. For example from the source or
binary distribution the following should work</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
cd examples/camel-example-etl
mvn camel:run
</pre>
</div></div>

<p>If you prefer you can just run the Main directly using</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
mvn compile exec:java
</pre>
</div></div>

<p>Please note that when you run the example for the first time, the converter <tt>CustomerTransformer</tt>
will not be able to find any entities inside the database, so that along the logs written
into the console you should see:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
thread #0 - file://src/data] CustomerTransformer            INFO  Created a new CustomerEntity
Customer[userName: james firstName: null surname: null] as no matching persisted entity found.
thread #0 - file://src/data] CustomerTransformer            INFO  Created a new CustomerEntity
Customer[userName: hiram firstName: null surname: null] as no matching persisted entity found.
</pre>
</div></div>

<p>However running the example for a second time, as the entites have been already inserted
into the database, the log should now say:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
thread #0 - file://src/data] CustomerTransformer            INFO  Found a matching CustomerEntity
Customer[userName: james firstName: James surname: Strachan] having the userName james.
thread #0 - file://src/data] CustomerTransformer            INFO  Found a matching CustomerEntity
Customer[userName: hiram firstName: Hiram surname: Chirino] having the userName hiram.
</pre>
</div></div>

<p>Failing that you can run the Main from inside your IDE if you prefer. Follow the
<a href="/confluence/display/CAMEL/Building" title="Building">Building</a> instructions
to create an Eclipse/IDEA project to import</p>
    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;" class="grey">
                        <a href="https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=CAMEL">Stop
watching space</a>
            <span style="padding: 0px 5px;">|</span>
                <a href="https://cwiki.apache.org/confluence/users/editmyemailsettings.action">Change
email notification preferences</a>
</div>
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/ETL+Example">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=62499&revisedVersion=13&originalVersion=12">View
Changes</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message