flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrea Sella <andrea.se...@radicalbit.io>
Subject Re: Flink and YARN ship folder
Date Wed, 16 Mar 2016 17:25:43 GMT
Hi,

After few tests I am able to write and read on Alluxio.
I am using Flink 1.0.0 and in my case external libraries are not loaded
from lib folder to classpath, it loads only flink-dist_2.11-1.0.0.jar. I
need to specify the folder with -yt parameter to load the others.

If I run `/bin/flink run -m yarn-cluster -yn 4 -yjm 2048 -ytm 4096 some.jar
--input alluxio://somepath` it will throws an exception related to a
missing library ie. Class alluxio.hadoop.FileSystem not found
If I run `./bin/flink run -m yarn-cluster -yt lib/ -yn 4 -yjm 2048 -ytm
4096 some.jar --input alluxio://somepath` with -yt params it will work fine.

is it a bug?

Cheers,
Andrea




2016-03-14 15:00 GMT+01:00 Andrea Sella <andrea.sella@radicalbit.io>:

> Hi Robert,
>
> Ok, thank you.
>
> 2016-03-14 11:13 GMT+01:00 Robert Metzger <rmetzger@apache.org>:
>
>> Hi Andrea,
>>
>> You don't have to manually replicate any operations on the slaves. All
>> files in the lib/ folder are transferred to all containers (Jobmanagers and
>> TaskManagers).
>>
>>
>> On Sat, Mar 12, 2016 at 3:25 PM, Andrea Sella <andrea.sella@radicalbit.io
>> > wrote:
>>
>>> Hi Ufuk,
>>>
>>> I'm trying to execute the WordCount batch example with input and output
>>> on Alluxio, i followed Running Flink on Alluxio
>>> <http://www.alluxio.org/documentation/en/Running-Flink-on-Alluxio.html>
and
>>> added the library to lib folder. Have I to replicate this operation on the
>>> slaves or YARN manage that and I must have the library just where I launch
>>> the job?
>>>
>>> Thanks,
>>> Andrea
>>>
>>> 2016-03-11 19:23 GMT+01:00 Ufuk Celebi <uce@apache.org>:
>>>
>>>> Everything in the lib folder should be added to the classpath. Can you
>>>> check the YARN client logs that the files are uploaded? Furthermore,
>>>> you can check the classpath of the JVM in the YARN logs of the
>>>> JobManager/TaskManager processes.
>>>>
>>>> – Ufuk
>>>>
>>>>
>>>> On Fri, Mar 11, 2016 at 5:33 PM, Andrea Sella
>>>> <andrea.sella@radicalbit.io> wrote:
>>>> > Hi,
>>>> >
>>>> > There is a way to add external dependencies to Flink Job,  running on
>>>> YARN,
>>>> > not using HADOOP_CLASSPATH?
>>>> > I am looking for a similar idea to standalone mode using lib folder.
>>>> >
>>>> > BR,
>>>> > Andrea
>>>>
>>>
>>>
>>
>

Mime
View raw message