flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shashank agarwal <shashank...@gmail.com>
Subject Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0
Date Wed, 20 Dec 2017 09:15:59 GMT
One more thing when i submit the job ir start yarn session it prints
following logs :

Using the result of 'hadoop classpath' to augment the Hadoop classpath:
/usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/flink/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


So i think it's adding Hadoop libs in classpath too cause it's able to
create the checkpointing directories from flink-conf file to HDFS.






‌

On Wed, Dec 20, 2017 at 2:31 PM, shashank agarwal <shashank734@gmail.com>
wrote:

> Hi,
>
> Please find attached list of jar file contents and flink/lib/ contents. I
> have removed my class files list from jar list and I have
> added flink-hadoop-compatibility_2.11-1.4.0.jar later in flink/lib/ but
> no success.
>
> I have tried by removing flink-shaded-hadoop2 from my project but still no
> success.
>
>
> Thanks
> Shashank
>
>
> On Wed, Dec 20, 2017 at 2:14 PM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>
>> Hi,
>>
>> Could you please list what exactly is in your submitted jar file, for
>> example using "jar tf my-jar-file.jar"? And also what files exactly are in
>> your Flink lib directory.
>>
>> Best,
>> Aljoscha
>>
>>
>> On 19. Dec 2017, at 20:10, shashank agarwal <shashank734@gmail.com>
>> wrote:
>>
>> Hi Timo,
>>
>> I am using Rocksdbstatebackend with hdfs path. I have following flink
>> dependencies in my sbt :
>>
>> "org.slf4j" % "slf4j-log4j12" % "1.7.21",
>>   "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
>>   "org.apache.flink" %% "flink-streaming-scala" % flinkVersion %
>> "provided",
>>   "org.apache.flink" %% "flink-cep-scala" % flinkVersion,
>>   "org.apache.flink" %% "flink-connector-kafka-0.10" % flinkVersion,
>>   "org.apache.flink" %% "flink-connector-filesystem" % flinkVersion,
>>   "org.apache.flink" %% "flink-statebackend-rocksdb" % flinkVersion,
>>   "org.apache.flink" %% "flink-connector-cassandra" % "1.3.2",
>>   "org.apache.flink" % "flink-shaded-hadoop2" % flinkVersion,
>>
>> when i start flink yarn session  it's working fine even it's creating
>> flink checkpointing directory and copying libs into hdfs.
>>
>> But when I submit the application to this yarn session it prints
>> following logs :
>>
>> Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.
<http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*
>>
>> But application fails contuniously with logs which i have sent earlier.
>>
>>
>>
>> ‌I have tried to add flink- hadoop-compability*.jar as suggested by Jorn
>> but it's not working.
>>
>>
>>
>> On Tue, Dec 19, 2017 at 5:08 PM, shashank agarwal <shashank734@gmail.com>
>>  wrote:
>>
>>> yes, it's working fine. now not getting compile time error.
>>>
>>> But when i trying to run this on cluster or yarn, getting following
>>> runtime error :
>>>
>>> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find
a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink
and no Hadoop file system to support this scheme could be loaded.
>>> 	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:405)
>>> 	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
>>> 	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
>>> 	at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
>>> 	at org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
>>> 	at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createStreamFactory(RocksDBStateBackend.java:273)
>>> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
>>> 	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
>>> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
>>> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
>>> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
>>> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
>>> 	at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop
File System abstraction does not support scheme 'hdfs'. Either no file system implementation
exists for that scheme, or the relevant classes are missing from the classpath.
>>> 	at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:102)
>>> 	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
>>> 	... 12 more
>>> Caused by: java.io.IOException: No FileSystem for scheme: hdfs
>>> 	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2786)
>>> 	at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:99)
>>> 	... 13 more
>>>
>>>
>>>
>>>
>>>
>>> while submitting job it's printing following logs so i think it's including hdoop
libs :
>>>
>>>
>>> Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/hdp/2.6.0.3-8/hadoop/conf:/usr/hdp/2.6.0.3-8/hadoop/lib/*:/usr/hdp/2.6.0.3-8/hadoop/.//*:/usr/hdp/2.6.0.
<http://2.6.0.0/>3-8/hadoop-hdfs/./:/usr/hdp/2.6.0.3-8/hadoop-hdfs/lib/*:/usr/hdp/2.6.0.3-8/hadoop-hdfs/.//*:/usr/hdp/2.6.0.3-8/hadoop-yarn/lib/*:/usr/hdp/2.6.0.3-8/hadoop-yarn/.//*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/lib/*:/usr/hdp/2.6.0.3-8/hadoop-mapreduce/.//*
>>>
>>>
>>> On Fri, Dec 8, 2017 at 9:24 PM, shashank agarwal <shashank734@gmail.com>
>>>  wrote:
>>>
>>>> Sure i’ll Try that. Thanks
>>>>
>>>> On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen <sewen@apache.org> wrote:
>>>>
>>>>> I would recommend to add "flink-shaded-hadoop2". That is a bundle of
>>>>> all Hadoop dependencies used by Flink.
>>>>>
>>>>>
>>>>> On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek <aljoscha@apache.org>
>>>>>  wrote:
>>>>>
>>>>>> I see, thanks for letting us know!
>>>>>>
>>>>>>
>>>>>> On 8. Dec 2017, at 15:42, shashank agarwal <shashank734@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> I had to include two dependencies.
>>>>>>
>>>>>> hadoop-hdfs (this for HDFS configuration)
>>>>>> hadoop-common (this for Path)
>>>>>>
>>>>>>
>>>>>> ‌
>>>>>>
>>>>>> On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek <aljoscha@apache.org
>>>>>> > wrote:
>>>>>>
>>>>>>> I think hadoop-hdfs might be sufficient.
>>>>>>>
>>>>>>>
>>>>>>> On 8. Dec 2017, at 14:48, shashank agarwal <shashank734@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Can you specifically guide which dependencies should I add to
extend
>>>>>>> this :
>>>>>>>
>>>>>>> https://github.com/apache/flink/blob/release-1.4.0-rc3/flink
>>>>>>> -connectors/flink-connector-filesystem/src/main/java/org/apa
>>>>>>> che/flink/streaming/connectors/fs/Bucketer.java
>>>>>>>
>>>>>>> is the Hadoop-core is sufficient.
>>>>>>>
>>>>>>>
>>>>>>> ‌
>>>>>>>
>>>>>>> On Fri, Dec 8, 2017 at 6:58 PM, shashank agarwal <
>>>>>>> shashank734@gmail.com> wrote:
>>>>>>>
>>>>>>>> It's a compilation error. I think I have to include the Hadoop
>>>>>>>> dependencies.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ‌
>>>>>>>>
>>>>>>>> On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek <
>>>>>>>> aljoscha@apache.org> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Is this a compilation error or at runtime. But in general,
yes you
>>>>>>>>> have to include the Hadoop dependencies if they're not
there.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Aljoscha
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 8. Dec 2017, at 14:10, shashank agarwal <shashank734@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am trying to test 1.4.0-RC3, Hadoop libraries removed
in this
>>>>>>>>> version. Actually, i have created custom Bucketer for
the bucketing sink.
>>>>>>>>> I am extending
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.flink.streaming.connectors.fs.bucketing.Bucketer
>>>>>>>>>
>>>>>>>>> in the class, i have to use org.apache.hadoop.fs.Path
 but as
>>>>>>>>> hadoop libraries removed it's giving error
>>>>>>>>>
>>>>>>>>> "object hadoop is not a member of package org.apache"
>>>>>>>>>
>>>>>>>>> Should i have to include Hadoop client libs in
>>>>>>>>> build.sbt dependencies.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Shashank
>>>>>>>>> ‌
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks Regards
>>>>>>>>
>>>>>>>> SHASHANK AGARWAL
>>>>>>>>  ---  Trying to mobilize the things....
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks Regards
>>>>>>>
>>>>>>> SHASHANK AGARWAL
>>>>>>>  ---  Trying to mobilize the things....
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks Regards
>>>>>>
>>>>>> SHASHANK AGARWAL
>>>>>>  ---  Trying to mobilize the things....
>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>> Sent from iPhone 5
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks Regards
>>>
>>> SHASHANK AGARWAL
>>>  ---  Trying to mobilize the things....
>>>
>>>
>>
>>
>> --
>> Thanks Regards
>>
>> SHASHANK AGARWAL
>>  ---  Trying to mobilize the things....
>>
>>
>>
>
>
> --
> Thanks Regards
>
> SHASHANK AGARWAL
>  ---  Trying to mobilize the things....
>
>


-- 
Thanks Regards

SHASHANK AGARWAL
 ---  Trying to mobilize the things....

Mime
View raw message