apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <thomas.we...@gmail.com>
Subject Re: Error finding KafkaSinglePortStringInputOperator (NoClassDefFoundError)
Date Tue, 01 Mar 2016 21:31:48 GMT
Please make sure no hadoop jars are included into the .apa


On Tue, Mar 1, 2016 at 1:30 PM, Suhas Gogate <suhas@ampool.io> wrote:

> Oh I see, I include hadoop 2.2.0 jars in App package,  but my runtime is
> hadoop 2.7.1. So addDeprecation is only supported after 2.3.0 it seems..
> May be I have to change my pom.xml to use hadoop 2.7.1 jars?
>
> From: Thomas Weise <thomas.weise@gmail.com>
> Reply-To: "users@apex.incubator.apache.org" <
> users@apex.incubator.apache.org>
> Date: Tuesday, March 1, 2016 at 1:28 PM
>
> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
> Subject: Re: Error finding KafkaSinglePortStringInputOperator
> (NoClassDefFoundError)
>
> Can you please check if there is a hadoop jar file inside the .apa ?
>
> On Tue, Mar 1, 2016 at 1:18 PM, Suhas Gogate <suhas@ampool.io> wrote:
>
>> Thanks Ram, I could launch the application after correcting the property
>> names but just after launching application it fails as AppMaster fails two
>> times with following error,  Any clue?
>>
>> 2016-03-01 21:01:41,904 INFO  stram.StreamingAppMaster
>> (StreamingAppMaster.java:main(98)) - Initializing Application Master.
>>
>> 2016-03-01 21:01:41,905 ERROR stram.StreamingAppMaster
>> (StreamingAppMaster.java:main(105)) - Exiting Application Master
>>
>> java.lang.NoSuchMethodError:
>> org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
>>
>> at
>> org.apache.hadoop.yarn.conf.YarnConfiguration.addDeprecatedKeys(YarnConfiguration.java:82)
>>
>> at
>> org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:76)
>>
>> at
>> com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:100)
>>
>> End of LogType:dt.log
>>
>>
>> From: Munagala Ramanath <ram@datatorrent.com>
>> Reply-To: "users@apex.incubator.apache.org" <
>> users@apex.incubator.apache.org>
>> Date: Tuesday, March 1, 2016 at 1:01 PM
>>
>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>> (NoClassDefFoundError)
>>
>> I just ran a sample app with the following properties in the
>> configuration file:
>>
>>   <property>
>>     <name>dt.operator.KafkaInput.prop.initialOffset</name>
>>     <value>earliest</value>
>>   </property>
>>   <property>
>>     <name>dt.operator.KafkaInput.prop.topic</name>
>>     <value>test-topic</value>
>>   </property>
>>   <property>
>>     <name>dt.operator.KafkaInput.prop.zookeeper</name>
>>     <value>node21:2181</value>
>>   </property>
>>
>> The string "KafkaInput" is the first argument to the dag.addOperator()
>> call.
>>
>> These settings will apply to the operator regardless of application. To
>> apply only to a specific application you can use:
>>
>>   <property>
>>
>> <name>dt.application.KafkaApp.operator.KafkaInput.prop.initialOffset</name>
>>     <value>earliest</value>
>>   </property>
>>   <property>
>>     <name>dt.application.KafkaApp.operator.KafkaInput.prop.topic</name>
>>     <value>test-topic</value>
>>   </property>
>>   <property>
>>
>> <name>dt.application.KafkaApp.operator.KafkaInput.prop.zookeeper</name>
>>     <value>node21:2181</value>
>>   </property>
>>
>> Let me know if you have any issues.
>>
>> Ram
>>
>> On Tue, Mar 1, 2016 at 11:15 AM, Suhas Gogate <suhas@ampool.io> wrote:
>>
>>> I looked at following code for Application java and saw you used
>>> different names for class variable and operator name :)
>>>
>>>
>>> https://github.com/sandeshh/streaming-benchmarks/blob/master/apex-benchmarks/src/main/java/com/example/Application.java
>>>
>>> I did the same and gave same names and got it working.. But now its
>>> failing after launch, so have to debug..   :)
>>>
>>> Thanks for the help!
>>>
>>> Regards,
>>> —Suhas
>>>
>>> From: Sandesh Hegde <sandesh@datatorrent.com>
>>> Reply-To: "users@apex.incubator.apache.org" <
>>> users@apex.incubator.apache.org>
>>> Date: Tuesday, March 1, 2016 at 11:11 AM
>>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>>> (NoClassDefFoundError)
>>>
>>> In my case, I used the same name in both the places.
>>>
>>> https://github.com/sandeshh/streaming-benchmarks/blob/squash/apex-benchmarks/src/main/java/com/example/Application.java
>>>
>>> Thanks for pointing out the mistakes in the doc, we will certainly look
>>> into it.
>>> Also, you are welcome to file jira for this,
>>> https://issues.apache.org/jira/browse/APEXCORE/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-pane
>>> l.
>>>
>>> Thanks
>>> Sandesh
>>>
>>> On Tue, Mar 1, 2016 at 10:44 AM Suhas Gogate <suhas@ampool.io> wrote:
>>>
>>>> Now these property names make sense but are they explained anyplace on
>>>> how to construct the names?  I see a logic but still may not be clear
>>>> unless it’s explained properly…    Again kafkaInput is a class variable
and
>>>> not the name of the operator given during dg.addOperator e.g. MessageReader
>>>> in my case is name of operator and “input” as class variable :)
>>>>
>>>> Look at the Application example at the end that provides wrong
>>>> information..
>>>> http://docs.datatorrent.com/operators/kafkaInputOperator/,
>>>>
>>>> Regards,
>>>> *—Suhas*
>>>>
>>>> From: Sandesh Hegde <sandesh@datatorrent.com>
>>>> Reply-To: "users@apex.incubator.apache.org" <
>>>> users@apex.incubator.apache.org>
>>>> Date: Tuesday, March 1, 2016 at 9:34 AM
>>>>
>>>> To: "users@apex.incubator.apache.org" <users@apex.incubator.apache.org>
>>>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>>>> (NoClassDefFoundError)
>>>>
>>>> Here is the sample from my working app,
>>>>
>>>> https://github.com/sandeshh/streaming-benchmarks/blob/master/conf/apex.xml
>>>>
>>>> you need to switch the strings "consumer" and "prop"
>>>>
>>>> <property>
>>>> <name
>>>> >dt.application.Apex_Benchmark.operator.kafkaInput.prop.consumer.topic</
>>>> name>
>>>> <value>ad-events</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 1, 2016 at 9:30 AM Munagala Ramanath <ram@datatorrent.com>
>>>> wrote:
>>>>
>>>>> Suhas,
>>>>>
>>>>> Can you try removing the ".consumer" substring from the property name
?
>>>>>
>>>>> Ram
>>>>>
>>>>> On Tue, Mar 1, 2016 at 9:13 AM, Suhas Gogate <suhas@ampool.io>
wrote:
>>>>>
>>>>>> Ram,  It seems following property also does not work..
>>>>>>
>>>>>> In a properties file, it would be something like this:
>>>>>> *  <property>*
>>>>>> *    <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>*
>>>>>> *    <value>....</value>*
>>>>>> *  </property>*
>>>>>>
>>>>>>
>>>>>> My properties,
>>>>>>
>>>>>> <?xml version="1.0" encoding="UTF-8" standalone="no"?>
>>>>>>
>>>>>> <configuration>
>>>>>>
>>>>>>   <property>
>>>>>>
>>>>>>     <name>dt.attr.MASTER_MEMORY_MB</name>
>>>>>>
>>>>>>     <value>1024</value>
>>>>>>
>>>>>>   </property>
>>>>>>
>>>>>>   <property>
>>>>>>
>>>>>>     <name>dt.operator.MessageReader.consumer.prop.initialOffset
>>>>>> </name>
>>>>>>
>>>>>>     <value>earliest</value>
>>>>>>
>>>>>>   </property>
>>>>>>
>>>>>>   <property>
>>>>>>
>>>>>>     <name>dt.operator.MessageReader.consumer.prop.topic</name>
>>>>>>
>>>>>>     <value>ad-server-log-topic</value>
>>>>>>
>>>>>>   </property>
>>>>>>
>>>>>>   <property>
>>>>>>
>>>>>>     <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>
>>>>>>
>>>>>>     <value>ip-172-31-47-211.us-west-2.compute.internal:2181</value>
>>>>>>
>>>>>>   </property>
>>>>>>
>>>>>> </configuration>
>>>>>>
>>>>>> From: Munagala Ramanath <ram@datatorrent.com>
>>>>>> Reply-To: "users@apex.incubator.apache.org" <
>>>>>> users@apex.incubator.apache.org>
>>>>>> Date: Tuesday, March 1, 2016 at 7:23 AM
>>>>>> To: "users@apex.incubator.apache.org" <
>>>>>> users@apex.incubator.apache.org>
>>>>>> Subject: Re: Error finding KafkaSinglePortStringInputOperator
>>>>>> (NoClassDefFoundError)
>>>>>>
>>>>>> In a properties file, it would be something like this:
>>>>>> *  <property>*
>>>>>> *    <name>dt.operator.MessageReader.consumer.prop.zookeeper</name>*
>>>>>> *    <value>....</value>*
>>>>>> *  </property>*
>>>>>>
>>>>>>
>>>>>
>>
>

Mime
View raw message