From user-return-30003-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Sep 24 15:02:11 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EF793180644 for ; Tue, 24 Sep 2019 17:02:10 +0200 (CEST) Received: (qmail 98505 invoked by uid 500); 24 Sep 2019 15:02:09 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 98495 invoked by uid 99); 24 Sep 2019 15:02:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Sep 2019 15:02:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E0944182B1C for ; Tue, 24 Sep 2019 15:02:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.251 X-Spam-Level: ** X-Spam-Status: No, score=2.251 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.25, HTML_FONT_LOW_CONTRAST=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=xeli-eu.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id fwo0l8f6ZyLz for ; Tue, 24 Sep 2019 15:02:06 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::432; helo=mail-wr1-x432.google.com; envelope-from=xeli@xelification.com; receiver= Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 421C47DDAA for ; Tue, 24 Sep 2019 15:02:06 +0000 (UTC) Received: by mail-wr1-x432.google.com with SMTP id o18so2329630wrv.13 for ; Tue, 24 Sep 2019 08:02:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xeli-eu.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=ZrXe2UoutVyX0e4Y78LcVp+ukiWdnlH2hv8MAqnHCmI=; b=qrwpfAUxxin+vYzXXwkrTm3nsgHlGSjRg9oT+sbmBH0ZvJIaeyO36ZPYgK7owXr7MB 67PKZJQI6CamcIAMfWh5cUTbDU1Dk0ziVVCfJ6TnsPEX8xgC9Sr9NUYEpOqWP4LrjO2Q s74NKtQDcE4vUNgG/4t/pmXrkmyCQvs9UikoDX4EIGiBRQKqBRR/5+5njk8RtPVj9oM7 BsdnY2048gunEiZlWREMFk/w0YHtTGYHZ9rhP8GmtvN20M0MtAcyyW2Sf4b3wGpGkxsC HY9xxy+MqQ/Rc1+tOszqNeSrh0fKjyNoTFMbmzhIbbzZyq8G4OiKr9d2aLlJgERTtSvL +XXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ZrXe2UoutVyX0e4Y78LcVp+ukiWdnlH2hv8MAqnHCmI=; b=k5mcy8UVg66ge+Q1hZYLjqlyzW0LhwYdrZ2IoU6BkQ/P8E26kusoKG5J9Kv7wcTg2c l3vrDRQQ7YtC7d15s/K1LobmcRGBQWe8TGRb1AxMyVKlFwURovcZIDRd13KeuNVTMYvI oIEyQMag6ZJ8ANzZnNFs3B74hDPqk4Yrrq76qvS+6SxkccZQf2ZnAV0hmTabsWgqFzUL o5Lngv+W92pW/oIhob1GYwVNPL7L/rUiYj/QXczuPLYywQDmpaRVoae/rWVnB06QZHzn gzwDCbgbh52/cJ+HNOnBg4yevtIh8ruoG4ucTdIHJNicpdPwCctPVwlAu28V9s05lTfa PUiA== X-Gm-Message-State: APjAAAVPtP1o67QNfwYJ5SqvHwlGqBBCzcwmTu6zCbY4exg5d4toiQAm ylSotxEISahAr/nWMVRbbW+ZKZb0mu3tBXQ4qABCn2wXCJ0= X-Google-Smtp-Source: APXvYqyTdYykZ0w+jugoPhB8yhv01kNCABq+LRc7bccSdeuD67NiPJADy/LTuYAMpBT8dmsN8dK/MLmxeMyBAPdvzn4= X-Received: by 2002:adf:f709:: with SMTP id r9mr2761998wrp.228.1569337321330; Tue, 24 Sep 2019 08:02:01 -0700 (PDT) MIME-Version: 1.0 From: Richard Deurwaarder Date: Tue, 24 Sep 2019 17:01:50 +0200 Message-ID: Subject: Setting environment variables of the taskmanagers (yarn) To: user Content-Type: multipart/alternative; boundary="000000000000927f6305934dd1a3" --000000000000927f6305934dd1a3 Content-Type: text/plain; charset="UTF-8" Hello, We have our flink job (1.8.0) running on our hadoop 2.7 cluster with yarn. We would like to add the GCS connector to use GCS rather than HDFS. Following the documentation of the GCS connector[1] we have to specify which credentials we want to use and there are two ways of doing this: * Edit core-site.xml * Set an environment variable: GOOGLE_APPLICATION_CREDENTIALS Because we're on a company shared hadoop cluster we do not want to change the cluster wide core-site.xml. This leaves me with two options: 1. Create a custom core-site.xml and use --yarnship to send it to all the taskmanager contains. If I do this, to what value should I set fs.hdfs.hadoopconf[2] in flink-conf ? 2. The second option would be to set an environment variable, however because the taskmanagers are started via yarn I'm having trouble figuring out how to make sure this environment variable is set for each yarn container / taskmanager. I would appreciate any help you can provide. Thank you, Richard [1] https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcs/INSTALL.md#configure-hadoop [2] https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/config.html#hdfs --000000000000927f6305934dd1a3 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

We have our fli= nk job (1.8.0) running on our hadoop 2.7 cluster with yarn. We would like t= o add the GCS connector to use GCS rather than HDFS.=C2=A0
Follow= ing the documentation of the GCS connector[1] we have to specify which cred= entials we want to use and there are two ways of doing this:
=C2= =A0 * Edit core-site.xml
=C2=A0 * Set an environment variable:=C2= =A0GOOGLE_APPLICATION_CREDENTIALS

Because we're on a company shared= hadoop cluster we do not want to change the cluster wide core-site.xml.=C2= =A0

This leaves me with two options:
1. Create a custom core-site.xml and use --yarnship to send it = to all the taskmanager contains. If I do this, to what=C2=A0value should I = set=C2=A0fs.hdfs.hadoopconf[2] in flin= k-conf ?
2. The second option would be to set an environment vari= able, however because the taskmanagers are started via yarn I'm having = trouble figuring out how to make sure this environment variable is set for = each yarn container / taskmanager.

I would appreci= ate any help=C2=A0you can provide.

--000000000000927f6305934dd1a3--