From user-return-73204-archive-asf-public=cust-asf.ponee.io@spark.apache.org Tue Jan 9 23:13:26 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 735A6180718 for ; Tue, 9 Jan 2018 23:13:26 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 62B11160C2D; Tue, 9 Jan 2018 22:13:26 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 81D84160C17 for ; Tue, 9 Jan 2018 23:13:25 +0100 (CET) Received: (qmail 30089 invoked by uid 500); 9 Jan 2018 22:13:23 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 30079 invoked by uid 99); 9 Jan 2018 22:13:23 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jan 2018 22:13:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C02DB180336 for ; Tue, 9 Jan 2018 22:13:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.148 X-Spam-Level: ** X-Spam-Status: No, score=2.148 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nq8ApHGegaL5 for ; Tue, 9 Jan 2018 22:13:22 +0000 (UTC) Received: from mail-wr0-f179.google.com (mail-wr0-f179.google.com [209.85.128.179]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 2AD015F22E for ; Tue, 9 Jan 2018 22:13:21 +0000 (UTC) Received: by mail-wr0-f179.google.com with SMTP id f8so15643484wre.4 for ; Tue, 09 Jan 2018 14:13:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=qM4NRQy6A2Cg25515ujsPkOr+iT4BzfUCAcdWapqyIg=; b=jPXEPgG+Vn3dRAcaf7mIDbhPaGfOOKVM3fDmLfMAQY6mc7v6nuTgx2TXK0f4Rv8RNA 9y64MUMH2ldnD79cKw9li/UHoO1RA3ZeSIoIOqxWa8PuBcELslF8/v5OooPN/GBnV/zT 2okajcATNq+pI7eECIlgVMFd0c0AsQSwXH6Hhqz2UJCSOiSt1fuXq+2Hg67Rt0Xx0zTG UmO4xbCUY0EIHIXrrYSbGgQCPm/4u36EcnSbWcWWpyMdw7gryUzWqbjpsCWCXW3JhMpa VYl9JfaVCRZ7c4fulxe1pkvPllC3Ue7P7LNtpdWdMOJ6kDGcstbcaj46r0sE678fKGc3 sRVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=qM4NRQy6A2Cg25515ujsPkOr+iT4BzfUCAcdWapqyIg=; b=FlC8PE9UP5U52yMa8yRI69pLg+TEXvHHh3WlhwMVsMHv9g120dCDB4ZnPBQHcG5/Ra yuCLhdDx1HmeIeLBqolhd/cSevaDHS2Ts6+5F+71ChxQj/rtYPeqCoFiBbc7nDIGEuhM lSaqqZpZdYT5PcuM9UJ5aZr1FGOLRKalE4STk5NO6loeZY+p8U1Sj8PR5+HQjrf6+6Gy a6zV+UELsgdfbxdBe5cs7uHPleZfLp/hRqdWsd/Avs0O/wFCgfR7KzH82AlGpnL+kHt8 d4WoiUpPkfaI2FJ3iv08hWJbXnfPRfBvihYmis1c86Ak0KlWeoIoob3/bL3jdsIVh+qS XwIQ== X-Gm-Message-State: AKGB3mIPoL2Uze7STHXJwwqOwSNZ/0ixI60eDiGRbett0rWY0E25SOwX oHPwtQSgaW6i1s3h/22b0b6o26DCyL6zKD+7KPU04w== X-Google-Smtp-Source: ACJfBouGd6Lkklm3QwbuSq8sU2HFmIidUtDMNIhcQfGPwDmf9wl/+5HdpLnnXdV0IEQifNQL1zsshgJRkCtt53AHZ+8= X-Received: by 10.223.198.84 with SMTP id u20mr14123028wrg.16.1515535999994; Tue, 09 Jan 2018 14:13:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.223.190.18 with HTTP; Tue, 9 Jan 2018 14:13:19 -0800 (PST) From: Jhon Anderson Cardenas Diaz Date: Tue, 9 Jan 2018 17:13:19 -0500 Message-ID: Subject: Spark UI stdout/stderr links point to executors internal address To: user@spark.apache.org Content-Type: multipart/alternative; boundary="089e0820f324ec8ff105625f385c" --089e0820f324ec8ff105625f385c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable *Environment:* AWS EMR, yarn cluster. *Description:* On Spark ui, in Environment and Executors tabs, the links of stdout and stderr point to the internal address of the executors. This would imply to expose the executors so that links can be accessed. Shouldn't those links be pointed to master then handled internally serving the master as a proxy for these files instead of exposing the internal machines? I have tried setting SPARK_PUBLIC_DNS and SPARK_LOCAL_IP variables so they contain the master ip address. I also tried with this properties: spark.yarn.appMasterEnv.SPARK_LOCAL_IP and spark.yarn.appMasterEnv.SPARK_PUBLIC_DNS but it does not seem to work. Any suggestion? =E2=80=8B --089e0820f324ec8ff105625f385c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Environment:

AWS EMR, yarn clu= ster.

=C2=A0

Description:

On Spark ui, in= Environment and Executors tabs, the links of stdout and stderr point to th= e internal address of the executors. This would imply to expose the executo= rs so that links can be accessed. Shouldn't those links be pointed to m= aster then handled internally serving the master as a proxy for these files= instead of exposing the internal machines?

=C2=A0

I have tried setting SPARK_PUBL= IC_DNS and SPARK_LOCAL_IP variables so they contain the master ip address. = I also tried with this properties: spark.yarn.appMasterEnv.SPARK_LOCAL_IP a= nd spark.yarn.appMasterEnv.SPARK_PUBLIC_DNS but it does not seem to work.


Any= suggestion?
=E2=80=8B

--089e0820f324ec8ff105625f385c--