apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)
Date Tue, 01 Mar 2016 21:01:43 GMT
I just ran a sample app with the following properties in the configuration
file:

  <property>
    <name>dt.operator.KafkaInput.prop.initialOffset</name>
    <value>earliest</value>
  </property>
  <property>
    <name>dt.operator.KafkaInput.prop.topic</name>
    <value>test-topic</value>
  </property>
  <property>
    <name>dt.operator.KafkaInput.prop.zookeeper</name>
    <value>node21:2181</value>
  </property>

The string "KafkaInput" is the first argument to the dag.addOperator() call.

These settings will apply to the operator regardless of application. To
apply only to a specific application you can use:

  <property>

<name>dt.application.KafkaApp.operator.KafkaInput.prop.initialOffset</name>
    <value>earliest</value>
  </property>
  <property>
    <name>dt.application.KafkaApp.operator.KafkaInput.prop.topic</name>
    <value>test-topic</value>
  </property>
  <property>
    <name>dt.application.KafkaApp.operator.KafkaInput.prop.zookeeper</name>
    <value>node21:2181</value>
  </property>

Let me know if you have any issues.

Ram

On Tue, Mar 1, 2016 at 11:15 AM, Suhas Gogate <suhas@ampool.io> wrote:

> I looked at following code for Application java and saw you used different
> names for class variable and operator name :)
>
>
> https://github.com/sandeshh/streaming-benchmarks/blob/master/apex-benchmarks/src/main/java/com/example/Application.java
>
> I did the same and gave same names and got it working.. But now its
> failing after launch, so have to debug..   :)
>
> Thanks for the help!
>
> Regards,
> —Suhas
>
> From: Sandesh Hegde <sandesh@datatorrent.com>
> Reply-To: "users@apex.incubator.apache.org" <
> users@apex.incubator.apache.org>
> Date: Tuesday, March 1, 2016 at 11:11 AM
> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
> Subject: Re: Error finding KafkaSinglePortStringInputOperator
> (NoClassDefFoundError)
>
> In my case, I used the same name in both the places.
>
> https://github.com/sandeshh/streaming-benchmarks/blob/squash/apex-benchmarks/src/main/java/com/example/Application.java
>
> Thanks for pointing out the mistakes in the doc, we will certainly look
> into it.
> Also, you are welcome to file jira for this,
> https://issues.apache.org/jira/browse/APEXCORE/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-pane
> l.
>
> Thanks
> Sandesh
>
> On Tue, Mar 1, 2016 at 10:44 AM Suhas Gogate <suhas@ampool.io> wrote:
>
>> Now these property names make sense but are they explained anyplace on
>> how to construct the names?  I see a logic but still may not be clear
>> unless it’s explained properly…    Again kafkaInput is a class variable and
>> not the name of the operator given during dg.addOperator e.g. MessageReader
>> in my case is name of operator and “input” as class variable :)
>>
>> Look at the Application example at the end that provides wrong
>> information..  http://docs.datatorrent.com/operators/kafkaInputOperator/
>> ,
>>
>> Regards,
>> *—Suhas*
>>
>> From: Sandesh Hegde <sandesh@datatorrent.com>
>> Reply-To: "users@apex.incubator.apache.org" <
>> users@apex.incubator.apache.org>
>> Date: Tuesday, March 1, 2016 at 9:34 AM
>>
>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>> (NoClassDefFoundError)
>>
>> Here is the sample from my working app,
>> https://github.com/sandeshh/streaming-benchmarks/blob/master/conf/apex.xml
>>
>> you need to switch the strings "consumer" and "prop"
>>
>> <property>
>> <name
>> >dt.application.Apex_Benchmark.operator.kafkaInput.prop.consumer.topic</
>> name>
>> <value>ad-events</value>
>> </property>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 1, 2016 at 9:30 AM Munagala Ramanath <ram@datatorrent.com>
>> wrote:
>>
>>> Suhas,
>>>
>>> Can you try removing the ".consumer" substring from the property name ?
>>>
>>> Ram
>>>
>>> On Tue, Mar 1, 2016 at 9:13 AM, Suhas Gogate <suhas@ampool.io> wrote:
>>>
>>>> Ram,  It seems following property also does not work..
>>>>
>>>> In a properties file, it would be something like this:
>>>> *  <property>*
>>>> *    <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>*
>>>> *    <value>....</value>*
>>>> *  </property>*
>>>>
>>>>
>>>> My properties,
>>>>
>>>> <?xml version="1.0" encoding="UTF-8" standalone="no"?>
>>>>
>>>> <configuration>
>>>>
>>>>   <property>
>>>>
>>>>     <name>dt.attr.MASTER_MEMORY_MB</name>
>>>>
>>>>     <value>1024</value>
>>>>
>>>>   </property>
>>>>
>>>>   <property>
>>>>
>>>>     <name>dt.operator.MessageReader.consumer.prop.initialOffset</name>
>>>>
>>>>     <value>earliest</value>
>>>>
>>>>   </property>
>>>>
>>>>   <property>
>>>>
>>>>     <name>dt.operator.MessageReader.consumer.prop.topic</name>
>>>>
>>>>     <value>ad-server-log-topic</value>
>>>>
>>>>   </property>
>>>>
>>>>   <property>
>>>>
>>>>     <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>
>>>>
>>>>     <value>ip-172-31-47-211.us-west-2.compute.internal:2181</value>
>>>>
>>>>   </property>
>>>>
>>>> </configuration>
>>>>
>>>> From: Munagala Ramanath <ram@datatorrent.com>
>>>> Reply-To: "users@apex.incubator.apache.org" <
>>>> users@apex.incubator.apache.org>
>>>> Date: Tuesday, March 1, 2016 at 7:23 AM
>>>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>>>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>>>> (NoClassDefFoundError)
>>>>
>>>> In a properties file, it would be something like this:
>>>> *  <property>*
>>>> *    <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>*
>>>> *    <value>....</value>*
>>>> *  </property>*
>>>>
>>>>
>>>

Mime
View raw message