Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2274E17A80 for ; Wed, 25 Mar 2015 23:44:49 +0000 (UTC) Received: (qmail 62246 invoked by uid 500); 25 Mar 2015 23:44:46 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 62166 invoked by uid 500); 25 Mar 2015 23:44:46 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 62156 invoked by uid 99); 25 Mar 2015 23:44:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2015 23:44:46 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zsxwing@gmail.com designates 209.85.213.169 as permitted sender) Received: from [209.85.213.169] (HELO mail-ig0-f169.google.com) (209.85.213.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Mar 2015 23:44:40 +0000 Received: by ignm3 with SMTP id m3so3965292ign.0 for ; Wed, 25 Mar 2015 16:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=dQzirAmMQuoDMMSpHAIYR8ekMuCoGbif5sEpELFbpyM=; b=TELII6e8iCJ79NgST1lS0DAakyoB8kZhaCElcaYNtyhDiJz2uELR5CtMgjUEE1Uiz8 RB3QwGitqFGFpKpsDWZ+ueM30rA3gG/N/WhGA9pxrzf36VmbYrVLdBZ4F4LKUxunMEF7 oE82VLurwENmPIVh2WpV8Bm3qTvA8jUi085ORtQHqNI1Yeq8S2GVH9Z6eqvLGcvq9LEZ 0wKL17vYZU0BWBAmfG9WF73HzrhjpFhBiO3JpvkwW9cSKwdXVSYsnQ+enkGb0k2rhyy5 0FJp44DEb2VaR5N6lUDaakC75Pz7bMfYgYpqN3Eum8tgpW2NFgpHGbhp82LFsMU7jiRD Y7Ug== MIME-Version: 1.0 X-Received: by 10.50.143.42 with SMTP id sb10mr32654585igb.49.1427327060342; Wed, 25 Mar 2015 16:44:20 -0700 (PDT) Received: by 10.107.164.153 with HTTP; Wed, 25 Mar 2015 16:44:20 -0700 (PDT) In-Reply-To: References: Date: Thu, 26 Mar 2015 07:44:20 +0800 Message-ID: Subject: Re: How to specify the port for AM Actor ... From: Shixiong Zhu To: Manoj Samel Cc: "user@spark.apache.org" Content-Type: multipart/alternative; boundary=001a1134b3ce68fcc60512257a3f X-Virus-Checked: Checked by ClamAV on apache.org --001a1134b3ce68fcc60512257a3f Content-Type: text/plain; charset=UTF-8 There is no configuration for it now. Best Regards, Shixiong Zhu 2015-03-26 7:13 GMT+08:00 Manoj Samel : > There may be firewall rules limiting the ports between host running spark > and the hadoop cluster. In that case, not all ports are allowed. > > Can it be a range of ports that can be specified ? > > On Wed, Mar 25, 2015 at 4:06 PM, Shixiong Zhu wrote: > >> It's a random port to avoid port conflicts, since multiple AMs can run in >> the same machine. Why do you need a fixed port? >> >> Best Regards, >> Shixiong Zhu >> >> 2015-03-26 6:49 GMT+08:00 Manoj Samel : >> >>> Spark 1.3, Hadoop 2.5, Kerbeors >>> >>> When running spark-shell in yarn client mode, it shows following message >>> with a random port every time (44071 in example below). Is there a way to >>> specify that port to a specific port ? It does not seem to be part of ports >>> specified in http://spark.apache.org/docs/latest/configuration.html >>> spark.xxx.port ... >>> >>> Thanks, >>> >>> 15/03/25 22:27:10 INFO Client: Application report for >>> application_1427316153428_0014 (state: ACCEPTED) >>> 15/03/25 22:27:10 INFO YarnClientSchedulerBackend: ApplicationMaster >>> registered as Actor[akka.tcp://sparkYarnAM@xyz >>> :44071/user/YarnAM#-1989273896] >>> >> >> > --001a1134b3ce68fcc60512257a3f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
There is no configuration for it now.

Best Regards,

Shixiong Zhu

2015-03-26 7:13 GMT+08:00 Manoj Samel <manojsameltech@gmail.com>:
There may be firewall rules limiting the ports between = host running spark and the hadoop cluster. In that case, not all ports are = allowed.

Can it be a range of ports that can be specifie= d ?

On Wed, Mar 25, 2015 at 4:06 PM, Shix= iong Zhu <zsxwing@gmail.com> wrote:
It's a random port to avoid port conflicts,= since multiple AMs can run in the same machine. Why do you need a fixed po= rt?

Best Regards,

Shixiong Zhu

2015-03-26 6:49 GMT+08:00 Manoj Samel <manojsameltech@gmail.com>:
Spark 1.3, Hadoop 2.5, Kerbeors

When running spark-shell in yarn client mode, it shows following mes= sage with a random port every time (44071 in example below). Is there a way= to specify that port to a specific port ? It does not seem to be part of p= orts specified in=C2=A0http://spark.apache.org/docs/latest/config= uration.html spark.xxx.port ...

Thanks,
<= div>
15/03/25 22:27:10 INFO Client: Application report for ap= plication_1427316153428_0014 (state: ACCEPTED)
15/03/25 22:27:10 = INFO YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka= .tcp://sparkYarnAM@xyz:44071/user/YarnAM#-1989273896]



--001a1134b3ce68fcc60512257a3f--