apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)
Date Tue, 01 Mar 2016 15:23:41 GMT
Suhas,

Here is a bit more information on dealing with this class of errors, known
as
validation errors: Validation is a pre-deployment step to ensure that all
necessary constraints are met. It is discussed here:
http://docs.datatorrent.com/application_development/#validating-an-application

In this particular case

   -  the message is '*may not be null'*
   -  the originating class is
   *com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator*
   -  the property path is *consumer.zookeeperMap* and
   -  the "*leafBean*" is
   *com.datatorrent.contrib.kafka.SimpleKafkaConsumer*


If you look at the file:

*contrib/src/main/java/com/datatorrent/contrib/kafka/SimpleKafkaConsumer.java*
you'll see that it extends *KafkaConsumer*; that class has the following
property
which is the source of your problem:

*  /***
*   * A zookeeper map keyed by cluster id*
*   * It's mandatory field*
*   */*
*  @NotNull*
*  @Bind(JavaSerializer.class)*
*  SetMultimap<String, String> zookeeperMap;*

To set the value of this property, you can do it in the code or in a
properties
file as suggested by Ashish; there is another example in
*KafkaInputOperatorTest.java* near line 434:

*testMeta.operator.setZookeeper("cluster1::node0,node1,node2:2181,node3:2182;cluster2::node4:2181");*

In a properties file, it would be something like this:
*  <property>*
*    <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>*
*    <value>....</value>*
*  </property>*

Finally, this property is documented at:
  http://docs.datatorrent.com/operators/kafkaInputOperator/
in the section on *KafkaConsumer*.


Ram


On Mon, Feb 29, 2016 at 11:00 PM, Suhas Gogate <suhas@ampool.io> wrote:

> Thanks Thomas!  I could get the dependency right, but now stuck with some
> other problem :)  Not sure how to fix this?  Appreciate your help!
>
> An error occurred trying to launch the application. Server message:
> javax.validation.ConstraintViolationException: Operator MessageReader
> violates constraints
> [ConstraintViolationImpl{rootBean=com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator@3000595,
> propertyPath='consumer.zookeeperMap', message='may not be null',
> leafBean=com.datatorrent.contrib.kafka.SimpleKafkaConsumer@198e8b8a,
> value=null}] at
> com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1337)
> at com.datatorrent.stram.StramClient.<init>(StramClient.java:141) at
> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:508)
> at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2047)
> at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3450) at
> com.datatorrent.stram.cli.DTCli.access$7000(DTCli.java:106) at
> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1892) at
> com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449)
>
> From: Thomas Weise <thomas.weise@gmail.com>
> Reply-To: "users@apex.incubator.apache.org" <
> users@apex.incubator.apache.org>
> Date: Monday, February 29, 2016 at 9:44 PM
> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
> Subject: Re: Error finding KafkaSinglePortStringInputOperator
> (NoClassDefFoundError)
>
> Example here:
>
> https://github.com/tweise/apex-samples/blob/master/exactly-once/pom.xml
>
>
> On Mon, Feb 29, 2016 at 9:36 PM, Suhas Gogate <suhas@ampool.io> wrote:
>
>> How to add those dependencies? I followed the documentation of Kafka
>> Input Operator, created a simple java maven project, compiled to create
>> .apa package. So not sure how to add the malhar-contrib library in the
>> application package?
>>
>> Thanks for the help
>>
>> —Suhas
>>
>> From: Thomas Weise <thomas.weise@gmail.com>
>> Reply-To: "users@apex.incubator.apache.org" <
>> users@apex.incubator.apache.org>
>> Date: Monday, February 29, 2016 at 9:34 PM
>>
>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>> (NoClassDefFoundError)
>>
>> malhar-contrib and Kafka dependencies are missing.
>>
>>
>> On Mon, Feb 29, 2016 at 9:32 PM, Suhas Gogate <suhas@ampool.io> wrote:
>>
>>> Thanks for the reply Thomas!  I was just trying to workaround the
>>> problem although I agree that all dependencies should be part application
>>> package.  Here is info.
>>>
>>> Sumedhas-MacBook-Pro-2:Downloads sumedhag$  unzip -t
>>> AdStream-1.0-SNAPSHOT.apa
>>>
>>> Archive:  AdStream-1.0-SNAPSHOT.apa
>>>
>>>     testing: META-INF/                OK
>>>
>>>     testing: META-INF/MANIFEST.MF     OK
>>>
>>>     testing: app/                     OK
>>>
>>>     testing: app/AdStream-1.0-SNAPSHOT.jar   OK
>>>
>>>     testing: lib/                     OK
>>>
>>>     testing: lib/jetty-websocket-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/jetty-security-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/jetty-continuation-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/activemq-client-5.8.0.jar   OK
>>>
>>>     testing: lib/jetty-util-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/slf4j-api-1.7.5.jar   OK
>>>
>>>     testing: lib/jersey-client-1.9.jar   OK
>>>
>>>     testing: lib/jetty-servlet-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/malhar-library-3.1.1.jar   OK
>>>
>>>     testing: lib/named-regexp-0.2.3.jar   OK
>>>
>>>     testing: lib/fastutil-6.6.4.jar   OK
>>>
>>>     testing: lib/jetty-http-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar   OK
>>>
>>>     testing: lib/jetty-io-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/commons-logging-1.1.1.jar   OK
>>>
>>>     testing: lib/commons-beanutils-1.8.3.jar   OK
>>>
>>>     testing: lib/commons-lang3-3.1.jar   OK
>>>
>>>     testing: lib/jersey-core-1.9.jar   OK
>>>
>>>     testing: lib/jms-api-1.1-rev-1.jar   OK
>>>
>>>     testing: lib/geronimo-jms_1.1_spec-1.1.1.jar   OK
>>>
>>>     testing: lib/jetty-server-8.1.10.v20130312.jar   OK
>>>
>>>     testing: lib/hawtbuf-1.9.jar      OK
>>>
>>>     testing: lib/javax.mail-1.5.0.jar   OK
>>>
>>>     testing: lib/activation-1.1.jar   OK
>>>
>>>     testing: lib/commons-compiler-2.7.8.jar   OK
>>>
>>>     testing: conf/                    OK
>>>
>>>     testing: conf/my-app-conf1.xml    OK
>>>
>>>     testing: META-INF/properties.xml   OK
>>>
>>> No errors detected in compressed data of AdStream-1.0-SNAPSHOT.apa.
>>>
>>>
>>> From: Thomas Weise <thomas.weise@gmail.com>
>>> Reply-To: "users@apex.incubator.apache.org" <
>>> users@apex.incubator.apache.org>
>>> Date: Monday, February 29, 2016 at 9:29 PM
>>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>>> (NoClassDefFoundError)
>>>
>>> Suhas,
>>>
>>> Do not copy any operator libraries into the lib folder. These
>>> dependencies need to be packaged with the application into the .apa
>>> application package. The .apa will also need to contain the Kafka
>>> dependencies.
>>>
>>> Can you please share what .jar files are in your app package (unzip -l
>>> yourapp.apa)
>>>
>>> Thomas
>>>
>>>
>>> On Mon, Feb 29, 2016 at 9:10 PM, Suhas Gogate <suhas@ampool.io> wrote:
>>>
>>>> I have single node data torrent/Apex installation. Software installed
>>>> under "/opt/datatorrent/releases/3.2.0”. When I uploaded and launched my
>>>> application I get error finding the class
>>>>  “KafkaSinglePortStringInputOperator”.  I could compile the application
>>>> after adding the dt-contrib dependency but not sure how dt-contrib jars be
>>>> available for application when running on single node data torrent
>>>> installation.
>>>>
>>>> Appreciate help!
>>>>
>>>> —Suhas
>>>> PS: I explicitly copied the  dt-contrib-3.1.1.jar to
>>>> /opt/datatorrent/current/lib/dt-contrib-3.1.1.jar and restarted
>>>> the gateway..
>>>>
>>>>
>>>> Error launching the application:
>>>>
>>>> An error occurred trying to launch the application. Server message:
>>>> java.lang.NoClassDefFoundError:
>>>> com/datatorrent/contrib/kafka/KafkaSinglePortStringInputOperator at
>>>> io.ampool.demo.adtech.Application.populateDAG(Application.java:44) at
>>>> com.datatorrent.stram.plan.logical.LogicalPlanConfiguration.prepareDAG(LogicalPlanConfiguration.java:2108)
>>>> at
>>>> com.datatorrent.stram.client.StramAppLauncher$1.createApp(StramAppLauncher.java:407)
>>>> at
>>>> com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:482)
>>>> at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2047)
>>>> at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3450) at
>>>> com.datatorrent.stram.cli.DTCli.access$7000(DTCli.java:106) at
>>>> com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1892) at
>>>> com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449) Caused by:
>>>> java.lang.ClassNotFoundException:
>>>> com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator at
>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
>>>> java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:810) at
>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 9 more Fatal
>>>> error encountered
>>>>
>>>>
>>>>
>>>>
>>>> Application.java
>>>>
>>>>
>>>> import com.datatorrent.api.StreamingApplication;
>>>>
>>>> import com.datatorrent.api.DAG;
>>>>
>>>> import com.datatorrent.api.DAG.Locality;
>>>>
>>>> import com.datatorrent.lib.io.ConsoleOutputOperator;
>>>>
>>>> import com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator
>>>> ;
>>>>
>>>>
>>>> @ApplicationAnnotation(name = "AdStream")
>>>>
>>>> public class Application implements StreamingApplication
>>>>
>>>> {
>>>>
>>>>   @Override
>>>>
>>>>   public void populateDAG(DAG dag, Configuration entries)
>>>>
>>>>   {
>>>>
>>>>     KafkaSinglePortStringInputOperator input =  dag.addOperator(
>>>> "MessageReader", newKafkaSinglePortStringInputOperator());
>>>>
>>>>
>>>>     ConsoleOutputOperator output = dag.addOperator("Output", new
>>>> ConsoleOutputOperator());
>>>>
>>>>
>>>>     dag.addStream("MessageData", input.outputPort, output.input);
>>>>
>>>>   }
>>>>
>>>> }
>>>>
>>>>
>>>
>>
>

Mime
View raw message