flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinay patil <vinay18.pa...@gmail.com>
Subject Re: add FLINK_LIB_DIR to classpath on yarn -> add properties file to class path on yarn
Date Tue, 22 Nov 2016 16:40:46 GMT
OK, got that, thank you for your solution, it worked for me with --yarnship
option

Regards,
Vinay Patil

On Tue, Nov 22, 2016 at 4:44 PM, Maximilian Michels [via Apache Flink User
Mailing List archive.] <ml-node+s2336050n10277h39@n4.nabble.com> wrote:

> Hi Vinay,
>
> I was referring to setting up an deploying a YARN cluster directly
> through a Java program, instead of using the command-line interface.
> When you do that, you typically construct a YarnClusterDescriptor and
> parameterize it, then you call deploy() to deploy the cluster and
> create a YarnClusterClient for interaction.
>
> -Max
>
>
> On Fri, Nov 18, 2016 at 12:20 PM, vinay patil <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10277&i=0>> wrote:
>
> > Hi Max,
> >
> > When I provided the config file using --yarnship option , it worked.
> Thank
> > you.
> >
> > However I did not get this : " it's better to add it to the ship files
> in
> > the
> > cluster descriptor" Can you please tell me how to do this ?
> >
> > Regards,
> > Vinay Patil
> >
> > On Fri, Nov 18, 2016 at 4:12 AM, Maximilian Michels [via Apache Flink
> User
> > Mailing List archive.] <[hidden email]> wrote:
> >>
> >> The JVM only accepts Jar files in the classpath. You will have to load
> >> your custom files from the working directory of the node where the lib
> >> directory is shipped.
> >>
> >> By the way, the /lib directory is meant for Jar files. If you want to
> >> ship a custom file, it's better to add it to the ship files in the
> >> cluster descriptor or use the --yarnship option from the command-line.
> >>
> >> -Max
> >>
> >>
> >> On Thu, Nov 17, 2016 at 5:37 AM, vinay patil <[hidden email]> wrote:
> >>
> >> > Hi Robert,
> >> >
> >> > Sorry for the late response.
> >> >
> >> > I checked the TM and JM logs but am not able to see that file added
> in
> >> > the
> >> > classpath.
> >> >
> >> > I have deployed Flink on EMR, and placed the yaml file in
> >> > /usr/lib/flink/lib
> >> > folder.
> >> >
> >> > What am I doing wrong here ?
> >> >
> >> > Regards,
> >> > Vinay Patil
> >> >
> >> > On Wed, Oct 26, 2016 at 1:25 PM, rmetzger0 [via Apache Flink User
> >> > Mailing
> >> > List archive.] <[hidden email]> wrote:
> >>
> >> >>
> >> >> Hi Vinay,
> >> >>
> >> >> the JobManager and TaskManager logs contain the classpath used when
> >> >> starting a container on YARN. Can you check if the yaml file is in
> the
> >> >> classpath?
> >> >>
> >> >> On Tue, Oct 25, 2016 at 8:28 AM, vinay patil <[hidden email]>
wrote:
> >> >>>
> >> >>> Hi Max,
> >> >>>
> >> >>> As discussed here , I have put my yaml file in the flink lib
> >> >>> directory,
> >> >>> but
> >> >>> still I am not able to get this file from classpath.
> >> >>>
> >> >>> I am using Flink 1.1.1 and cfg4j to load the file from classpath.
> >> >>>
> >> >>> Running the job on YARN in EMR using the below command:
> >> >>>
> >> >>> ./bin/flink run <jar_name>
> >> >>>
> >> >>> Can you please let me know what I am doing wrong here.
> >> >>>
> >> >>> Regards,
> >> >>> Vinay Patil
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> View this message in context:
> >> >>>
> >> >>> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9702.html
> >> >>> Sent from the Apache Flink User Mailing List archive. mailing list
> >> >>> archive at Nabble.com.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ________________________________
> >> >> If you reply to this email, your message will be added to the
> >> >> discussion
> >> >> below:
> >> >>
> >> >>
> >> >> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p9719.html
> >> >> To start a new topic under Apache Flink User Mailing List archive.,
> >> >> email
> >> >> [hidden email]
> >> >> To unsubscribe from Apache Flink User Mailing List archive., click
> >> >> here.
> >> >> NAML
> >> >
> >> >
> >> >
> >> > ________________________________
> >> > View this message in context: Re: add FLINK_LIB_DIR to classpath on
> yarn
> >> > ->
> >> > add properties file to class path on yarn
> >> >
> >> > Sent from the Apache Flink User Mailing List archive. mailing list
> >> > archive
> >> > at Nabble.com.
> >>
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p10187.html
> >> To start a new topic under Apache Flink User Mailing List archive.,
> email
> >> [hidden email]
> >> To unsubscribe from Apache Flink User Mailing List archive., click
> here.
> >> NAML
> >
> >
> >
> > ________________________________
> > View this message in context: Re: add FLINK_LIB_DIR to classpath on yarn
> ->
> > add properties file to class path on yarn
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive
> > at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p10277.html
> To start a new topic under Apache Flink User Mailing List archive., email
> ml-node+s2336050n1h83@n4.nabble.com
> To unsubscribe from Apache Flink User Mailing List archive., click here
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
> .
> NAML
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/add-FLINK-LIB-DIR-to-classpath-on-yarn-tp8097p10279.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Mime
View raw message