flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
Date Tue, 22 Nov 2016 11:25:40 GMT
Hi Vinay,

I was referring to setting up an deploying a YARN cluster directly
through a Java program, instead of using the command-line interface.
When you do that, you typically construct a YarnClusterDescriptor and
parameterize it, then you call deploy() to deploy the cluster and
create a YarnClusterClient for interaction.

-Max


On Fri, Nov 18, 2016 at 12:20 PM, vinay patil <vinay18.patil@gmail.com> wrote:
> Hi Max,
>
> When I provided the config file using --yarnship option , it worked. Thank
> you.
>
> However I did not get this : " it's better to add it to the ship files in
> the
> cluster descriptor" Can you please tell me how to do this ?
>
> Regards,
> Vinay Patil
>
> On Fri, Nov 18, 2016 at 4:12 AM, Maximilian Michels [via Apache Flink User
> Mailing List archive.] <[hidden email]> wrote:
>>
>> The JVM only accepts Jar files in the classpath. You will have to load
>> your custom files from the working directory of the node where the lib
>> directory is shipped.
>>
>> By the way, the /lib directory is meant for Jar files. If you want to
>> ship a custom file, it's better to add it to the ship files in the
>> cluster descriptor or use the --yarnship option from the command-line.
>>
>> -Max
>>
>>
>> On Thu, Nov 17, 2016 at 5:37 AM, vinay patil <[hidden email]> wrote:
>>
>> > Hi Robert,
>> >
>> > Sorry for the late response.
>> >
>> > I checked the TM and JM logs but am not able to see that file added in
>> > the
>> > classpath.
>> >
>> > I have deployed Flink on EMR, and placed the yaml file in
>> > /usr/lib/flink/lib
>> > folder.
>> >
>> > What am I doing wrong here ?
>> >
>> > Regards,
>> > Vinay Patil
>> >
>> > On Wed, Oct 26, 2016 at 1:25 PM, rmetzger0 [via Apache Flink User
>> > Mailing
>> > List archive.] <[hidden email]> wrote:
>>
>> >>
>> >> Hi Vinay,
>> >>
>> >> the JobManager and TaskManager logs contain the classpath used when
>> >> starting a container on YARN. Can you check if the yaml file is in the
>> >> classpath?
>> >>
>> >> On Tue, Oct 25, 2016 at 8:28 AM, vinay patil <[hidden email]> wrote:
>> >>>
>> >>> Hi Max,
>> >>>
>> >>> As discussed here , I have put my yaml file in the flink lib
>> >>> directory,
>> >>> but
>> >>> still I am not able to get this file from classpath.
>> >>>
>> >>> I am using Flink 1.1.1 and cfg4j to load the file from classpath.
>> >>>
>> >>> Running the job on YARN in EMR using the below command:
>> >>>
>> >>> ./bin/flink run <jar_name>
>> >>>
>> >>> Can you please let me know what I am doing wrong here.
>> >>>
>> >>> Regards,
>> >>> Vinay Patil
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> >>>
>> >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9702.html
>> >>> Sent from the Apache Flink User Mailing List archive. mailing list
>> >>> archive at Nabble.com.
>> >>
>> >>
>> >>
>> >>
>> >> ________________________________
>> >> If you reply to this email, your message will be added to the
>> >> discussion
>> >> below:
>> >>
>> >>
>> >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9719.html
>> >> To start a new topic under Apache Flink User Mailing List archive.,
>> >> email
>> >> [hidden email]
>> >> To unsubscribe from Apache Flink User Mailing List archive., click
>> >> here.
>> >> NAML
>> >
>> >
>> >
>> > ________________________________
>> > View this message in context: Re: add FLINK_LIB_DIR to classpath on yarn
>> > ->
>> > add properties file to class path on yarn
>> >
>> > Sent from the Apache Flink User Mailing List archive. mailing list
>> > archive
>> > at Nabble.com.
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p10187.html
>> To start a new topic under Apache Flink User Mailing List archive., email
>> [hidden email]
>> To unsubscribe from Apache Flink User Mailing List archive., click here.
>> NAML
>
>
>
> ________________________________
> View this message in context: Re: add FLINK_LIB_DIR to classpath on yarn ->
> add properties file to class path on yarn
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.

Mime
View raw message