apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Running application using apex CLI
Date Wed, 29 Jun 2016 18:47:10 GMT
Hi Akshay,

Thanks for contribution to apex-malhar about update to elastic search
We have a contributing guidelines on apex website (

One of the important steps in contributing to apex is to communicate on
dev@apex.apache.org and share your intent and approach with the community.
This is critical for your changes as there are dependency changes involved
in your PR and development community should be able to discuss and share
your feedback before changes can be merged.

About guava dependency, as such apex does not provide a way to override
default dependency.
I would not recommend to change guava dependency in apex. The guava
dependency is essentially coming from hadoop and not apex.

But to make your case work, there is a way to shade dependency of guava and
elastic search using maven. Please have a look at maven shade plugin:


On Wed, Jun 29, 2016 at 2:33 AM, Akshay S Harale <akshay.harale@synerzip.com
> wrote:

> In our application we are fetching data from kafka, doing some
> modifications in data and then persist that data into elastic search.
> As we are using elasticsearch verion 2.2.3, first we did upgrade of
> elastic search client API in malhar contrib from 1.1.2 to 2.3.3 [ here is
>  pull request : https://github.com/apache/apex-malhar/pull/325 ]
> With new elastic search api we able to run the application in local mode
> using junite test case which worked very well and able to see the data in
> elastic search.
> Now we are trying to run our application using APEX cli. First we get the
> exception about the googles guava library:
> *java.lang.NoSuchMethodError:
> com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>       at
> org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:190)
>   at
> org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:131)
>       at
> com.datatorrent.contrib.elasticsearch.ElasticSearchConnectable.connect(ElasticSearchConnectable.java:123)*
> The reason behind above exception is : elastic search java client
> requires guava 18.0 and the Hadoop 2.2 comes with guava 11.0.2.
> In our application package (apa file) we have guava 18.0 jar. But i think
> while launching application apex gives preference to the guava 11.0.2.
> So we updated the apex-engine.jar and changed guava version to 18.0
> manually which is not good practice :)
> After this we are able to see that application is running in hadoop
> cluster but in log there is an exception :
> *2016-06-28 15:54:28,721 ERROR com.datatorrent.netlet.AbstractClient:
> Exception in event loop {id=ProcessWideEventLoop, head=1, tail=1,
> capacity=1024} java.net.ConnectException: Connection refused         at
> sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)         at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>     at
> com.datatorrent.netlet.DefaultEventLoop.handleSelectedKey(DefaultEventLoop.java:371)
>       at
> com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet.forEach(OptimizedEventLoop.java:59)
>       at
> com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:192)*
> *We want to know about how to override the default apex dependencies with
> dependency that  we are using in our application.*
> Regards,
> Akshay S. Harale
> Software Developer @ Synerzip
> Skype – akshayharale
> This e-mail, including any attached files, may contain confidential and
> privileged information for the sole use of the intended recipient. Any
> review, use, distribution, or disclosure by others is strictly prohibited.
> If you are not the intended recipient (or authorized to receive information
> for the intended recipient), please contact the sender by reply e-mail and
> delete all copies of this message.

View raw message