apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suhas Gogate <su...@ampool.io>
Subject Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)
Date Tue, 01 Mar 2016 17:02:40 GMT
Thanks Ashish and Ram,

  1.  See the Application example at the end of this  http://docs.datatorrent.com/operators/kafkaInputOperator/
  (need correction?)
  2.  Also its bit confusing to identify property name string to set properties for the embedded
classes like consumer. IMO, All the properties  should be documented for top  level operator
w/ their fully qualified property name string and optional default value vs. mandatory status.
E.g. In Kafka input operator, there is  KafkaConsumer and SimpleKafkaConsumer so how do I
know the property strings for both?

Thanks & Regards, Suhas

From: Munagala Ramanath <ram@datatorrent.com<mailto:ram@datatorrent.com>>
Reply-To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>"
<users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Date: Tuesday, March 1, 2016 at 7:23 AM
To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>" <users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Subject: Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

Suhas,

Here is a bit more information on dealing with this class of errors, known as
validation errors: Validation is a pre-deployment step to ensure that all
necessary constraints are met. It is discussed here:
http://docs.datatorrent.com/application_development/#validating-an-application

In this particular case

  *    the message is 'may not be null'
  *    the originating class is com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator
  *    the property path is consumer.zookeeperMap and
  *    the "leafBean" is com.datatorrent.contrib.kafka.SimpleKafkaConsumer

If you look at the file:
  contrib/src/main/java/com/datatorrent/contrib/kafka/SimpleKafkaConsumer.java
you'll see that it extends KafkaConsumer; that class has the following property
which is the source of your problem:

  /**
   * A zookeeper map keyed by cluster id
   * It's mandatory field
   */
  @NotNull
  @Bind(JavaSerializer.class)
  SetMultimap<String, String> zookeeperMap;

To set the value of this property, you can do it in the code or in a properties
file as suggested by Ashish; there is another example in KafkaInputOperatorTest.java near
line 434:

testMeta.operator.setZookeeper("cluster1::node0,node1,node2:2181,node3:2182;cluster2::node4:2181");

In a properties file, it would be something like this:
  <property>
    <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>
    <value>....</value>
  </property>

Finally, this property is documented at:
  http://docs.datatorrent.com/operators/kafkaInputOperator/
in the section on KafkaConsumer.


Ram


On Mon, Feb 29, 2016 at 11:00 PM, Suhas Gogate <suhas@ampool.io<mailto:suhas@ampool.io>>
wrote:
Thanks Thomas!  I could get the dependency right, but now stuck with some other problem :)
 Not sure how to fix this?  Appreciate your help!

An error occurred trying to launch the application. Server message: javax.validation.ConstraintViolationException:
Operator MessageReader violates constraints [ConstraintViolationImpl{rootBean=com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator@3000595,
propertyPath='consumer.zookeeperMap', message='may not be null', leafBean=com.datatorrent.contrib.kafka.SimpleKafkaConsumer@198e8b8a,
value=null}] at com.datatorrent.stram.plan.logical.LogicalPlan.validate(LogicalPlan.java:1337)
at com.datatorrent.stram.StramClient.<init>(StramClient.java:141) at com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:508)
at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2047) at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3450)
at com.datatorrent.stram.cli.DTCli.access$7000(DTCli.java:106) at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1892)
at com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449)

From: Thomas Weise <thomas.weise@gmail.com<mailto:thomas.weise@gmail.com>>
Reply-To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>"
<users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Date: Monday, February 29, 2016 at 9:44 PM
To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>" <users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Subject: Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

Example here:

https://github.com/tweise/apex-samples/blob/master/exactly-once/pom.xml


On Mon, Feb 29, 2016 at 9:36 PM, Suhas Gogate <suhas@ampool.io<mailto:suhas@ampool.io>>
wrote:
How to add those dependencies? I followed the documentation of Kafka Input Operator, created
a simple java maven project, compiled to create .apa package. So not sure how to add the malhar-contrib
library in the application package?

Thanks for the help

—Suhas

From: Thomas Weise <thomas.weise@gmail.com<mailto:thomas.weise@gmail.com>>
Reply-To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>"
<users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Date: Monday, February 29, 2016 at 9:34 PM

To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>" <users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Subject: Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

malhar-contrib and Kafka dependencies are missing.


On Mon, Feb 29, 2016 at 9:32 PM, Suhas Gogate <suhas@ampool.io<mailto:suhas@ampool.io>>
wrote:
Thanks for the reply Thomas!  I was just trying to workaround the problem although I agree
that all dependencies should be part application package.  Here is info.


Sumedhas-MacBook-Pro-2:Downloads sumedhag$  unzip -t AdStream-1.0-SNAPSHOT.apa

Archive:  AdStream-1.0-SNAPSHOT.apa

    testing: META-INF/                OK

    testing: META-INF/MANIFEST.MF     OK

    testing: app/                     OK

    testing: app/AdStream-1.0-SNAPSHOT.jar   OK

    testing: lib/                     OK

    testing: lib/jetty-websocket-8.1.10.v20130312.jar   OK

    testing: lib/jetty-security-8.1.10.v20130312.jar   OK

    testing: lib/jetty-continuation-8.1.10.v20130312.jar   OK

    testing: lib/activemq-client-5.8.0.jar   OK

    testing: lib/jetty-util-8.1.10.v20130312.jar   OK

    testing: lib/slf4j-api-1.7.5.jar   OK

    testing: lib/jersey-client-1.9.jar   OK

    testing: lib/jetty-servlet-8.1.10.v20130312.jar   OK

    testing: lib/malhar-library-3.1.1.jar   OK

    testing: lib/named-regexp-0.2.3.jar   OK

    testing: lib/fastutil-6.6.4.jar   OK

    testing: lib/jetty-http-8.1.10.v20130312.jar   OK

    testing: lib/geronimo-j2ee-management_1.1_spec-1.0.1.jar   OK

    testing: lib/jetty-io-8.1.10.v20130312.jar   OK

    testing: lib/commons-logging-1.1.1.jar   OK

    testing: lib/commons-beanutils-1.8.3.jar   OK

    testing: lib/commons-lang3-3.1.jar   OK

    testing: lib/jersey-core-1.9.jar   OK

    testing: lib/jms-api-1.1-rev-1.jar   OK

    testing: lib/geronimo-jms_1.1_spec-1.1.1.jar   OK

    testing: lib/jetty-server-8.1.10.v20130312.jar   OK

    testing: lib/hawtbuf-1.9.jar      OK

    testing: lib/javax.mail-1.5.0.jar   OK

    testing: lib/activation-1.1.jar   OK

    testing: lib/commons-compiler-2.7.8.jar   OK

    testing: conf/                    OK

    testing: conf/my-app-conf1.xml    OK

    testing: META-INF/properties.xml   OK

No errors detected in compressed data of AdStream-1.0-SNAPSHOT.apa.


From: Thomas Weise <thomas.weise@gmail.com<mailto:thomas.weise@gmail.com>>
Reply-To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>"
<users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Date: Monday, February 29, 2016 at 9:29 PM
To: "users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>" <users@apex.incubator.apache.org<mailto:users@apex.incubator.apache.org>>
Subject: Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)

Suhas,

Do not copy any operator libraries into the lib folder. These dependencies need to be packaged
with the application into the .apa application package. The .apa will also need to contain
the Kafka dependencies.

Can you please share what .jar files are in your app package (unzip -l yourapp.apa)

Thomas


On Mon, Feb 29, 2016 at 9:10 PM, Suhas Gogate <suhas@ampool.io<mailto:suhas@ampool.io>>
wrote:
I have single node data torrent/Apex installation. Software installed under "/opt/datatorrent/releases/3.2.0”.
When I uploaded and launched my application I get error finding the class  “KafkaSinglePortStringInputOperator”.
 I could compile the application after adding the dt-contrib dependency but not sure how dt-contrib
jars be available for application when running on single node data torrent installation.

Appreciate help!

—Suhas
PS: I explicitly copied the  dt-contrib-3.1.1.jar to  /opt/datatorrent/current/lib/dt-contrib-3.1.1.jar
and restarted the gateway..


Error launching the application:

An error occurred trying to launch the application. Server message: java.lang.NoClassDefFoundError:
com/datatorrent/contrib/kafka/KafkaSinglePortStringInputOperator at io.ampool.demo.adtech.Application.populateDAG(Application.java:44)
at com.datatorrent.stram.plan.logical.LogicalPlanConfiguration.prepareDAG(LogicalPlanConfiguration.java:2108)
at com.datatorrent.stram.client.StramAppLauncher$1.createApp(StramAppLauncher.java:407) at
com.datatorrent.stram.client.StramAppLauncher.launchApp(StramAppLauncher.java:482) at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:2047)
at com.datatorrent.stram.cli.DTCli.launchAppPackage(DTCli.java:3450) at com.datatorrent.stram.cli.DTCli.access$7000(DTCli.java:106)
at com.datatorrent.stram.cli.DTCli$LaunchCommand.execute(DTCli.java:1892) at com.datatorrent.stram.cli.DTCli$3.run(DTCli.java:1449)
Caused by: java.lang.ClassNotFoundException: com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:810) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 more Fatal error encountered




Application.java



import com.datatorrent.api.StreamingApplication;

import com.datatorrent.api.DAG;

import com.datatorrent.api.DAG.Locality;

import com.datatorrent.lib.io.ConsoleOutputOperator;

import com.datatorrent.contrib.kafka.KafkaSinglePortStringInputOperator;


@ApplicationAnnotation(name = "AdStream")

public class Application implements StreamingApplication

{

  @Override

  public void populateDAG(DAG dag, Configuration entries)

  {

    KafkaSinglePortStringInputOperator input =  dag.addOperator("MessageReader", newKafkaSinglePortStringInputOperator());


    ConsoleOutputOperator output = dag.addOperator("Output", new ConsoleOutputOperator());


    dag.addStream("MessageData", input.outputPort, output.input);

  }

}





Mime
View raw message