From user-return-25811-archive-asf-public=cust-asf.ponee.io@hive.apache.org Thu Feb 8 15:28:31 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 7B52718064F for ; Thu, 8 Feb 2018 15:28:31 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 6B031160C4A; Thu, 8 Feb 2018 14:28:31 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8A635160C3D for ; Thu, 8 Feb 2018 15:28:30 +0100 (CET) Received: (qmail 87139 invoked by uid 500); 8 Feb 2018 14:28:29 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 87129 invoked by uid 99); 8 Feb 2018 14:28:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Feb 2018 14:28:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A491DC0186 for ; Thu, 8 Feb 2018 14:28:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.988 X-Spam-Level: X-Spam-Status: No, score=0.988 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=-1, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=pbinc.onmicrosoft.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id X1HIjDC6I2vd for ; Thu, 8 Feb 2018 14:28:26 +0000 (UTC) Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0099.outbound.protection.outlook.com [104.47.42.99]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 38E0F5F11F for ; Thu, 8 Feb 2018 14:28:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pbinc.onmicrosoft.com; s=selector1-pb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=vIRBU+RqreaK5VzFbExyN50v1yDHl6b3L7Q2N/QnXFg=; b=u9iF4ZqbNWwn4sX4j/SDH2j5r+FjJHcLkW0ScTkA/GBOdHPkZbDms7syUCTRaBYdYryCwvU8HrLZ529a0z1mLwEnZUTwZpmhcIE6bm0dV/+UPn5fuu0IObbBB5A3mjcqxLGvahc6GC0l2/F1V6XDgMxvVyOP/g7JnYkDmLVOXMA= Received: from DM5P128MB0025.NAMP128.PROD.OUTLOOK.COM (129.75.86.152) by DM5P128MB0028.NAMP128.PROD.OUTLOOK.COM (129.75.86.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.464.14; Thu, 8 Feb 2018 14:28:17 +0000 Received: from DM5P128MB0025.NAMP128.PROD.OUTLOOK.COM ([129.75.86.152]) by DM5P128MB0025.NAMP128.PROD.OUTLOOK.COM ([129.75.86.152]) with mapi id 15.20.0464.016; Thu, 8 Feb 2018 14:28:17 +0000 From: Ray Navarette To: "user@hive.apache.org" Subject: Resources/Distributed Cache on Spark Thread-Topic: Resources/Distributed Cache on Spark Thread-Index: AdOg6JLyWF/ys3+6TG6YMhvNxUJY9g== Date: Thu, 8 Feb 2018 14:28:17 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Ray.Navarette@pb.com; x-originating-ip: [24.194.192.4] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5P128MB0028;7:7eHtIk/DJDXu4rUFLosd2XWUP+Qe4y/D9sXaUd0GnEqFxIYqcScAmeHCUgqRUlmoQD3eU6FfY8IU1HZJBd5OBAQf675jS1aK/TLYHKtnoU+F47CNLm8r038MjpwuAqqpP2KmLjgpM5wEsbTgsWgpzMvOafWY7SA/aHZKbbu7CYni+laFwCsiHn5i5m0K77dvJ67IqgeIwLbo9hZVkSkOZpqFhm24zyOX6htMhs4lLwrhxeoqeClnsEKTmj4KqFW9;20:hV3XrAIIb32AbSZxJtsT/soClAL7FPhkqoaBqu326YIaLsNwvnIh5QB4nmAOO+FyayjLpZtEFSil22/5J1ouPUQ4vKvcuTP2A9FTCnZmUl/xumCOYVvqZkUzGV63srXp3N5mYFA5yApegd1BtVQtuO3TGjfDVGPT+kAl4QT/CNYNL4jURGS6Pskc60YbxCcyg64zBPNlX5oEMyGWzjlrmjwYGrwdbRxKCIvpw70uHvXrna1fUxnlRaKTDS3397fRzpmwzolbD824huBrnP0Oq91XczxzOQwxIPbBCV0JeA718mvt5Ynz7eCZLXdwuUXkU3y9D6BYGTtcwDrYDyNJ8TP+tqsqniKc92kd32GPBdZUT05fqbRM/wkih3h3LV666ecV7lqx7+XwFNp0rPhK0eWYutLsVsQ0siZdLZZBtM4RDT289/9e2aCHhTyXOTwDgrEuPeyuCopeHe74CybY2SyIQlFF20zWfLlniZ6yVNEVGzSj2HTThyyEO6+WhVWm x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 5ce7650b-b78f-4390-1c9d-08d56f003263 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(7020095)(4652020)(48565401081)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603307)(7153060)(7193020);SRVR:DM5P128MB0028; x-ms-traffictypediagnostic: DM5P128MB0028: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(21748063052155); x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(6040501)(2401047)(5005006)(8121501046)(3002001)(3231101)(2400082)(944501161)(10201501046)(93006095)(93001095)(6055026)(6041288)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(20161123562045)(6072148)(201708071742011);SRVR:DM5P128MB0028;BCL:0;PCL:0;RULEID:;SRVR:DM5P128MB0028; x-forefront-prvs: 0577AD41D6 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39860400002)(396003)(346002)(376002)(39380400002)(366004)(199004)(189003)(9326002)(3480700004)(102836004)(59450400001)(2501003)(74316002)(68736007)(3660700001)(33656002)(7736002)(3280700002)(54896002)(86362001)(1730700003)(81166006)(55016002)(2351001)(5640700003)(6436002)(8936002)(66066001)(6306002)(956003)(97736004)(9686003)(25786009)(6916009)(790700001)(2900100001)(81156014)(6506007)(5660300001)(6116002)(26005)(106356001)(8676002)(53936002)(3846002)(316002)(99286004)(478600001)(186003)(105586002)(2906002)(7696005)(72206003);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5P128MB0028;H:DM5P128MB0025.NAMP128.PROD.OUTLOOK.COM;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; received-spf: None (protection.outlook.com: pb.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: Q/mUVzxfGZqYc115HH9MvLFT3fnUahTne00XPkuVArPAJwo2h8GHbfWfAm0jmNNroKWtK+zWPfcatAC19TnPtw== spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: multipart/alternative; boundary="_000_DM5P128MB002534ABDED83739188CCC6889F30DM5P128MB0025NAMP_" MIME-Version: 1.0 X-OriginatorOrg: pb.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5ce7650b-b78f-4390-1c9d-08d56f003263 X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Feb 2018 14:28:17.8207 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8a4925a9-fd8e-4866-b31c-f719fb05dce6 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5P128MB0028 --_000_DM5P128MB002534ABDED83739188CCC6889F30DM5P128MB0025NAMP_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, I'm hoping to find some information about using "ADD FILES " when usi= ng the spark execution engine. I've seen some jira tickets reference this = functionality, but little else. We have written some custom UDFs which req= uire some external resources. When using the MR execution engine, we can r= eference the file paths using a relative path and they are properly distrib= uted and resolved. When I try to do the same under spark engine, I receive= an error saying the file is unavailable. Does "ADD FILES " work on spark, and if so, how should I properly ref= erence those files in order to read them in the executors? Thanks much for your help, Ray --_000_DM5P128MB002534ABDED83739188CCC6889F30DM5P128MB0025NAMP_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

I’m hoping to find some information about usin= g “ADD FILES <PATH>” when using the spark execution engin= e.  I’ve seen some jira tickets reference this functionality, bu= t little else.  We have written some custom UDFs which require some ex= ternal resources.  When using the MR execution engine, we can reference the = file paths using a relative path and they are properly distributed and reso= lved.  When I try to do the same under spark engine, I receive an erro= r saying the file is unavailable.

 

Does “ADD FILES <PATH>” work on sp= ark, and if so, how should I properly reference those files in order to rea= d them in the executors?

 

Thanks much for your help,

Ray

--_000_DM5P128MB002534ABDED83739188CCC6889F30DM5P128MB0025NAMP_--