From user-return-32666-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Feb 17 16:31:54 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id AD4D7180636 for ; Mon, 17 Feb 2020 17:31:53 +0100 (CET) Received: (qmail 18103 invoked by uid 500); 17 Feb 2020 16:31:52 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 18093 invoked by uid 99); 17 Feb 2020 16:31:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Feb 2020 16:31:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8622DC2D79 for ; Mon, 17 Feb 2020 16:31:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id sCPKgg_SRh9i for ; Mon, 17 Feb 2020 16:31:50 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::241; helo=mail-lj1-x241.google.com; envelope-from=lazy.gopher@gmail.com; receiver= Received: from mail-lj1-x241.google.com (mail-lj1-x241.google.com [IPv6:2a00:1450:4864:20::241]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id AFBC57E133 for ; Mon, 17 Feb 2020 16:31:49 +0000 (UTC) Received: by mail-lj1-x241.google.com with SMTP id r19so19556669ljg.3 for ; Mon, 17 Feb 2020 08:31:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JXawhykyuB/jSaVKdN5aCG7qHgT6KATt80Sm8iBvPG0=; b=jn2tn+jv0nI9azOyPVOJspURDsZCJutr/y0LC+D5NURA1lxOt/SfHeUROC39g6i/SH Qzpu8fhs6TwLJlU74Nh8BnvfgR+dQbfkNquGaUR2I6fvWQFjhte/2EDEwAXcFePu9FlF oeVgZ1Kz1rK3uaoxBxjJAXXe50lmvs/yBkDShIZ8A1lvqN2TKYNu8tWjaL3lAMvPRW/f tsKrwVc8N4xXxdsOMHids1ldTV4haTyfHhmLl5oanwkbUPMs9eCG6IELcBJoih8zG8aV DrHwwpes/HkcD3xg1K3/CJVI75JfkrlKL/x0JaEVNgeif8EfOM+T5D15YEosK5wZazaJ ql+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JXawhykyuB/jSaVKdN5aCG7qHgT6KATt80Sm8iBvPG0=; b=HSz94yn2H/L7wCqExSjSRlw12IQFqEykTSCJIyYhw42ZjBjUMRM/E/c08QQpm+gBxA oDyuCC9kyq9b1G1qDn6cqMY+ydKmwjsBjwRfY1F+2IUXypKy0TwyBKHckGKkpvT9dfPo rhEBSKOtRk5YE2gXbBDrEQxi+Fjs0uog0R6o5TsohY6E45QSpGKcwS/ftanY0fq8xiQN XPeAUHnvamL1ytCs8GDcfCGy1g8ZTYKAbldmbp31X/I6J4l52W3Xn23OVP2rso+xc3Z3 gVpQPR/tZPL2k4exnAC0z5LpaJDOb74Wmp5ws6wMuETTRT7taxDYCnimFI6ueRwnJd32 F/2Q== X-Gm-Message-State: APjAAAWQ71AuvpTXd4Hf2EaqRZ/tLBiFRzXtMqriOPPAmTNaiOKt8qLC HAUaSuMIIRlVlQmz4R0AITdLMgSNIsJVkoVZj5A= X-Google-Smtp-Source: APXvYqxirrjd6kKP4U6EWvDzL2uuKnNvkdo9fCIzOiLpY3UEnophlg5ZiKknsd65DR4UGg5huvDd8DGoJMelID6MyDM= X-Received: by 2002:a2e:b610:: with SMTP id r16mr10818982ljn.33.1581957108893; Mon, 17 Feb 2020 08:31:48 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Maxim Parkachov Date: Mon, 17 Feb 2020 17:31:38 +0100 Message-ID: Subject: Re: [Flink 1.10] Classpath doesn't include custom files in lib/ To: Yang Wang Cc: user Content-Type: multipart/alternative; boundary="0000000000008689bd059ec81751" --0000000000008689bd059ec81751 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Yang, I've just tried your suggestions, but, unfortunately, in yarn per job mode it doesn't work, both commands return null. I double checked that file is shipped to yarn container, but I feel that it happens later in process. At the moment I'm reading file with File interface, instead of getting it as resource, which I do in local mode. Regards, Maxim. On Mon, Feb 17, 2020 at 3:03 PM Yang Wang wrote: > Hi Maxim, > > I have verified that the following two ways could both work. > > getClass().getClassLoader().getResource("lib/job.properties") > getClass().getClassLoader().getResource("job.properties") > > > Best, > Yang > > Maxim Parkachov =E4=BA=8E2020=E5=B9=B42=E6=9C=881= 7=E6=97=A5=E5=91=A8=E4=B8=80 =E4=B8=8B=E5=8D=886:47=E5=86=99=E9=81=93=EF=BC= =9A > >> Hi Yang, >> >> thanks, this explains why classpath behavior changed, but now I struggle >> to >> understand how I could overwrite resource, which is already shipped in >> job jar. >> >> Before I had job.properties files in JAR in under >> resources/lib/job.properties >> for local development and deploying on cluster it was overwritten >> with environment specific settings in lib/job.properties of flink >> distribution. >> Now this doesn't seem to work. I'm using: >> >> getClass.getClassLoader.getResource("lib/job.properties") >> >> to get file. Could it be the problem ? >> >> Thanks, >> Maxim. >> >> On Mon, Feb 17, 2020 at 4:12 AM Yang Wang wrote: >> >>> Hi Maxim Parkachov, >>> >>> The users files also have been shipped to JobManager and TaskManager. >>> However, it >>> is not directly added to the classpath. Instead, the parent directory i= s >>> added to the >>> classpath. This changes are to make resource classloading work. You >>> could check more >>> information here[1]. >>> >>> >>> [1]. https://issues.apache.org/jira/browse/FLINK-13127 >>> >>> >>> Best, >>> Yang >>> >>> Maxim Parkachov =E4=BA=8E2020=E5=B9=B42=E6=9C= =8815=E6=97=A5=E5=91=A8=E5=85=AD =E4=B8=8A=E5=8D=8812:58=E5=86=99=E9=81=93= =EF=BC=9A >>> >>>> Hi everyone, >>>> >>>> I'm trying to run my job with flink 1.10 with YARN cluster per-job >>>> mode. In the previous versions all files in lib/ folder were automatic= ally >>>> included in classpath. Now, with 1.10 I see only *.jar files are inclu= ded >>>> in classpath. but not "other" files. Is this deliberate change or bug = ? >>>> >>>> Generally, what is recommended way to include custom files in classpat= h >>>> and ship it during start to all containers ? >>>> >>>> Thank >>>> >>> --0000000000008689bd059ec81751 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Yang,

I've just tried your sugge= stions, but, unfortunately, in yarn per job mode it doesn't work, both = commands return null.
I double checked that file is shipped to ya= rn container, but I feel that it happens later in process.
At the= moment I'm reading file with File interface, instead of getting it as = resource, which=C2=A0I do in local mode.

Regards,<= /div>
Maxim.


<= div dir=3D"ltr" class=3D"gmail_attr">On Mon, Feb 17, 2020 at 3:03 PM Yang W= ang <danrtsey.wy@gmail.com&= gt; wrote:
Hi Maxim,

I have verified that the following= two ways could both work.

getClass().getClassLoad= er().getResource("lib/job.properties&= quot;)
getClass().getClassLoader().getResour= ce("job.properties")=C2= =A0


Best,
Yang

Maxim= Parkachov <l= azy.gopher@gmail.com> =E4=BA=8E2020=E5=B9=B42=E6=9C=8817=E6=97=A5=E5= =91=A8=E4=B8=80 =E4=B8=8B=E5=8D=886:47=E5=86=99=E9=81=93=EF=BC=9A
=
Hi Yang,=

thanks, this explains why classpath behavior changed, b= ut now I struggle to
understand how I could overwrite resource, w= hich is already shipped=C2=A0in job jar.

Before I = had job.properties files in JAR in under resources/lib/job.properties
=
for local development and deploying on cluster it was overwritten=C2= =A0
with environment specific settings in=C2=A0 lib/job.propertie= s of flink distribution.=C2=A0
Now this doesn't seem to work.= =C2=A0 I'm using:=C2=A0
getClass.getClassLoader.getResource("lib/job.properties")
to= get file. Could it be the problem ?

Thanks,
Maxim.
On Mon, = Feb 17, 2020 at 4:12 AM Yang Wang <danrtsey.wy@gmail.com> wrote:
Hi=C2=A0Maxim P= arkachov,

The users files also have been shipped to JobM= anager and TaskManager. However, it
is not directly added to the = classpath. Instead, the parent directory is added to the
classpat= h. This changes are to=C2=A0make resource classloading work. You could chec= k more
information here[1].


[1].=C2=A0https://issues.apache.org/jira/browse/FLINK-13127


Best,
Yang

Maxim Parkac= hov <lazy.gop= her@gmail.com> =E4=BA=8E2020=E5=B9=B42=E6=9C=8815=E6=97=A5=E5=91=A8= =E5=85=AD =E4=B8=8A=E5=8D=8812:58=E5=86=99=E9=81=93=EF=BC=9A
Hi everyone,<= div>
I'm trying=C2=A0to run my job with flink 1.10 with Y= ARN cluster per-job mode. In the previous versions all files in lib/ folder= were automatically included in classpath. Now, with 1.10 I see only *.jar = files are included in classpath. but not "other" files. Is this d= eliberate change or bug ?

Generally, what is recom= mended way to include custom files in classpath and ship it during start to= all containers ?

Thank
--0000000000008689bd059ec81751--