camel-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Babak Vahdat (Confluence)" <conflue...@apache.org>
Subject [CONF] Apache Camel > ETL Example
Date Mon, 21 Oct 2013 12:36:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/en/2176/1/1/_/styles/combined.css?spaceKey=CAMEL&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/CAMEL/ETL+Example">ETL
Example</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~bvahdat">Babak
Vahdat</a>
    </h4>
        <br/>
                         <h4>Changes (3)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >Then we send the message with a PersonDocument
body to the [JPA] endpoint. Notice how this endpoint specifies the expected type. So the [Type
Converter] is gonna try convert the PersonDocument to a [CustomerEntity|hhttps://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java].
Here Camel will find the [CustomerTransformer|https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java]
class which has an @Converter method <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">{snippet:id=example|lang=java|url=camel/trunk/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java}
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">{code}
<br>@Converter <br>public class CustomerTransformer { <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">
   private static final Logger LOG = LoggerFactory.getLogger(CustomerTransformer.class); <br>
<br>    /** <br>     * A transformation method to convert a person document into
a customer <br>     * entity <br>     */ <br>    @Converter <br> 
  public static CustomerEntity toCustomer(PersonDocument doc, Exchange exchange) throws Exception
{ <br>        EntityManager entityManager = exchange.getIn().getHeader(JpaConstants.ENTITYMANAGER,
EntityManager.class); <br>        TransactionTemplate transactionTemplate = exchange.getContext().getRegistry().lookupByNameAndType(&quot;transactionTemplate&quot;,
TransactionTemplate.class); <br> <br>        String user = doc.getUser(); <br>
       CustomerEntity customer = findCustomerByName(transactionTemplate, entityManager, user);
<br> <br>        // let&#39;s convert information from the document into the
entity bean <br>        customer.setUserName(user); <br>        customer.setFirstName(doc.getFirstName());
<br>        customer.setSurname(doc.getLastName()); <br>        customer.setCity(doc.getCity());
<br> <br>        LOG.info(&quot;Created object customer: {}&quot;, customer);
<br>        return customer; <br>    } <br> <br>    /** <br>
    * Finds a customer for the given username <br>     */ <br>    private static
CustomerEntity findCustomerByName(TransactionTemplate transactionTemplate, final EntityManager
entityManager, final String userName) throws Exception { <br>        return transactionTemplate.execute(new
TransactionCallback&lt;CustomerEntity&gt;() { <br>            public CustomerEntity
doInTransaction(TransactionStatus status) { <br>                entityManager.joinTransaction();
<br>                List&lt;CustomerEntity&gt; list = entityManager.createNamedQuery(&quot;findCustomerByUsername&quot;,
CustomerEntity.class).setParameter(&quot;userName&quot;, userName).getResultList();
<br>                CustomerEntity answer; <br>                if (list.isEmpty())
{ <br>                    answer = new CustomerEntity(); <br>                
   answer.setUserName(userName); <br>                    LOG.info(&quot;Created
a new CustomerEntity {} as no matching persisted entity found.&quot;, answer); <br>
               } else { <br>                    answer = list.get(0); <br>   
                LOG.info(&quot;Found a matching CustomerEntity {} having the userName
{}.&quot;, answer, userName); <br>                } <br> <br>      
         return answer; <br>            } <br>        }); <br>    } <br>
<br>} <br>{code} <br> <br></td></tr>
            <tr><td class="diff-unchanged" >which performs the necessary conversion
to an entity bean which is then stored in the database <br> <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h2><a name="ETLExample-ExtractTransformLoad%28ETL%29Example"></a>Extract
Transform Load (ETL) Example </h2>

<p>The <a href="/confluence/display/CAMEL/ETL" title="ETL">ETL</a> (Extract,
Transform, Load) example shows how to load data into a database using Camel. In this example
we will poll for files, transform them and then store them in the database via the <a href="/confluence/display/CAMEL/JPA"
title="JPA">JPA</a> component.</p>

<h3><a name="ETLExample-Overview"></a>Overview</h3>

<p>This example lives in the <em>examples/camel-example-etl</em> directory
and will poll for XML files in the child <em>src/data</em> directory. When the
files are detected, they are converted, using the fallback <a href="/confluence/display/CAMEL/JAXB"
title="JAXB">JAXB</a> <a href="/confluence/display/CAMEL/Type+Converter" title="Type
Converter">Type Converter</a> to a PersonDocument class. This POJO is then transformed
using a custom <a href="/confluence/display/CAMEL/Type+Converter" title="Type Converter">Type
Converter</a> into a CustomerEntity bean using the CustomerTransformer class. Then this
bean is stored in the database via the <a href="/confluence/display/CAMEL/JPA" title="JPA">JPA</a>
component.</p>

<p>The code for this example is as follows</p>

<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Java Class </th>
<th class='confluenceTh'> Purpose </th>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java"
class="external-link" rel="nofollow">CustomerEntity</a> </td>
<td class='confluenceTd'> The JPA entity bean (i.e. a POJO with @Entity) </td>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java"
class="external-link" rel="nofollow">PersonDocument</a> </td>
<td class='confluenceTd'> The JAXB2 POJO used to parse the XML </td>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java"
class="external-link" rel="nofollow">CustomerTransformer</a> </td>
<td class='confluenceTd'> The custom <a href="/confluence/display/CAMEL/Type+Converter"
title="Type Converter">Type Converter</a> used to convert a PersonDocument into a
CustomerEntity </td>
</tr>
<tr>
<td class='confluenceTd'> <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java"
class="external-link" rel="nofollow">EtlRoutes</a> </td>
<td class='confluenceTd'> The Camel routing <a href="/confluence/display/CAMEL/DSL"
title="DSL">DSL</a> to define the flow from the files to the converter to the <a
href="/confluence/display/CAMEL/JPA" title="JPA">JPA</a> endpoint </td>
</tr>
</tbody></table>
</div>


<p>The there is the spring configuration file in <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/resources/META-INF/spring/camel-context.xml"
class="external-link" rel="nofollow">src/resources/META-INF/services/camel-context.xml</a>
which defines the JPA template and tells Camel to look in the <b>org.apache.camel.example.etl</b>
package to find its routes.</p>

<h3><a name="ETLExample-Codewalkthrough"></a>Code walkthrough </h3>

<p>So lets start with the route definition in <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/EtlRoutes.java"
class="external-link" rel="nofollow">EtlRoutes</a></p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
public class EtlRoutes extends SpringRouteBuilder {
    public void configure() throws Exception {

        from("file:src/data?noop=true")
            .convertBodyTo(PersonDocument.class)
            .to("jpa:org.apache.camel.example.etl.CustomerEntity");

        // the following will dump the database to files
        from("jpa:org.apache.camel.example.etl.CustomerEntity?consumeDelete=false&amp;delay=3000&amp;consumeLockEntity=false")
            .setHeader(Exchange.FILE_NAME, el("${in.body.userName}.xml"))
            .to("file:target/customers");
    }
}
</pre>
</div></div>

<p>The above sets up a route from the <em>src/data</em> directory. Notice
we're using the <em>noop</em> mode of the <a href="/confluence/display/CAMEL/File2"
title="File2">File</a> component so that the files are not moved or deleted when
they are processed (so when the tool is restarted they will be processed again).</p>

<p>We're converting the body of the message to a <a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/PersonDocument.java"
class="external-link" rel="nofollow">PersonDocument</a> which since this POJO as
an @XmlRootElement annotation from <a href="/confluence/display/CAMEL/JAXB" title="JAXB">JAXB</a>
will kick in the <a href="/confluence/display/CAMEL/Type+Converter" title="Type Converter">Type
Converter</a> to use JAXB to unmarshall the object.</p>

<p>Then we send the message with a PersonDocument body to the <a href="/confluence/display/CAMEL/JPA"
title="JPA">JPA</a> endpoint. Notice how this endpoint specifies the expected type.
So the <a href="/confluence/display/CAMEL/Type+Converter" title="Type Converter">Type
Converter</a> is gonna try convert the PersonDocument to a <a href="hhttps://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerEntity.java"
class="external-link" rel="nofollow">CustomerEntity</a>. Here Camel will find the
<a href="https://github.com/apache/camel/blob/master/examples/camel-example-etl/src/main/java/org/apache/camel/example/etl/CustomerTransformer.java"
class="external-link" rel="nofollow">CustomerTransformer</a> class which has an @Converter
method</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
@Converter
public class CustomerTransformer {

    private static final Logger LOG = LoggerFactory.getLogger(CustomerTransformer.class);

    /**
     * A transformation method to convert a person document into a customer
     * entity
     */
    @Converter
    public static CustomerEntity toCustomer(PersonDocument doc, Exchange exchange) throws
Exception {
        EntityManager entityManager = exchange.getIn().getHeader(JpaConstants.ENTITYMANAGER,
EntityManager.class);
        TransactionTemplate transactionTemplate = exchange.getContext().getRegistry().lookupByNameAndType("transactionTemplate",
TransactionTemplate.class);

        String user = doc.getUser();
        CustomerEntity customer = findCustomerByName(transactionTemplate, entityManager, user);

        // let's convert information from the document into the entity bean
        customer.setUserName(user);
        customer.setFirstName(doc.getFirstName());
        customer.setSurname(doc.getLastName());
        customer.setCity(doc.getCity());

        LOG.info("Created object customer: {}", customer);
        return customer;
    }

    /**
     * Finds a customer for the given username
     */
    private static CustomerEntity findCustomerByName(TransactionTemplate transactionTemplate,
final EntityManager entityManager, final String userName) throws Exception {
        return transactionTemplate.execute(new TransactionCallback&lt;CustomerEntity&gt;()
{
            public CustomerEntity doInTransaction(TransactionStatus status) {
                entityManager.joinTransaction();
                List&lt;CustomerEntity&gt; list = entityManager.createNamedQuery("findCustomerByUsername",
CustomerEntity.class).setParameter("userName", userName).getResultList();
                CustomerEntity answer;
                if (list.isEmpty()) {
                    answer = new CustomerEntity();
                    answer.setUserName(userName);
                    LOG.info("Created a new CustomerEntity {} as no matching persisted entity
found.", answer);
                } else {
                    answer = list.get(0);
                    LOG.info("Found a matching CustomerEntity {} having the userName {}.",
answer, userName);
                }

                return answer;
            }
        });
    }

}
</pre>
</div></div>

<p>which performs the necessary conversion to an entity bean which is then stored in
the database</p>

<h3><a name="ETLExample-Runningtheexample"></a>Running the example</h3>

<p>To run the example we use the <a href="/confluence/display/CAMEL/Camel+Maven+Plugin"
title="Camel Maven Plugin">Camel Maven Plugin</a>. For example from the source or
binary distribution the following should work</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
cd examples/camel-example-etl
mvn camel:run
</pre>
</div></div>

<p>If you prefer you can just run the Main directly using</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
mvn compile exec:java
</pre>
</div></div>

<p>Please note that when you run the example for the first time, the converter <tt>CustomerTransformer</tt>
will not be able to find any entities inside the database, so that along the logs written
into the console you should see:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
thread #0 - file://src/data] CustomerTransformer            INFO  Created a new CustomerEntity
Customer[userName: james firstName: null surname: null] as no matching persisted entity found.
thread #0 - file://src/data] CustomerTransformer            INFO  Created a new CustomerEntity
Customer[userName: hiram firstName: null surname: null] as no matching persisted entity found.
</pre>
</div></div>

<p>However running the example for a second time, as the entites have been already inserted
into the database, the log should now say:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
thread #0 - file://src/data] CustomerTransformer            INFO  Found a matching CustomerEntity
Customer[userName: james firstName: James surname: Strachan] having the userName james.
thread #0 - file://src/data] CustomerTransformer            INFO  Found a matching CustomerEntity
Customer[userName: hiram firstName: Hiram surname: Chirino] having the userName hiram.
</pre>
</div></div>

<p>Failing that you can run the Main from inside your IDE if you prefer. Follow the
<a href="/confluence/display/CAMEL/Building" title="Building">Building</a> instructions
to create an Eclipse/IDEA project to import</p>
    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;" class="grey">
                        <a href="https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=CAMEL">Stop
watching space</a>
            <span style="padding: 0px 5px;">|</span>
                <a href="https://cwiki.apache.org/confluence/users/editmyemailsettings.action">Change
email notification preferences</a>
</div>
        <a href="https://cwiki.apache.org/confluence/display/CAMEL/ETL+Example">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=62499&revisedVersion=15&originalVersion=14">View
Changes</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message