From hadoop-dev-return-22629-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org Thu Nov 15 00:35:07 2007 Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 50292 invoked from network); 15 Nov 2007 00:35:06 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 15 Nov 2007 00:35:06 -0000 Received: (qmail 42335 invoked by uid 500); 15 Nov 2007 00:34:53 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 42290 invoked by uid 500); 15 Nov 2007 00:34:52 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 42281 invoked by uid 99); 15 Nov 2007 00:34:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Nov 2007 16:34:52 -0800 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Nov 2007 00:34:50 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 98385714245 for ; Wed, 14 Nov 2007 16:34:43 -0800 (PST) Message-ID: <32590099.1195086883619.JavaMail.jira@brutus> Date: Wed, 14 Nov 2007 16:34:43 -0800 (PST) From: "Allen Wittenauer (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Commented: (HADOOP-2185) Server ports: to roll or not to roll. In-Reply-To: <31601905.1194664550727.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542640 ] Allen Wittenauer commented on HADOOP-2185: ------------------------------------------ The primary issue is that hadoop isn't predictable. The randomization of ports (including picking purely ephemeral ports) is that it is impossible to: 1) Make sure Hadoop doesn't sit on a port that another application may be using unless that application is already running. 2) Create firewall rules that prevents connection to Hadoop services. 3) Create QoS (Quality of Service) settings such that HDFS has a higher/lower priority vs. some other service. I really want predictability. I want to be able to say that my JobTracker is always using port a,b,c, my namenode is always using ports d,e,f, my datanode is always using port g,h,i, etc. If that port is in use, then it should be perfectly accepting to have that process fail. Predictability == management when we're talking about large scale administration. As Arun pointed out, if someone really doesn't care what ports these services run on, then using 0 should be a reliable equivalent to using port rolling. As to the names, I'd prefer 'hdfs' over 'dfs' if only because a lot of the people I talk to always follow up with "Why are you using Microsoft's DFS?". *sigh* I prefer 'http' over 'info' if only because most people are more likely to recognize that a web interface is sitting on that port and that it might require extra care. [The (ab)use of that port by Hadoop is another issue... ;) ] One concern I have is what happens if I have multiple interfaces (nics). How does it work if I only want to bind to one interface or if I want to bind to all of them or I want to bind to different ports on those different interfaces? > Server ports: to roll or not to roll. > ------------------------------------- > > Key: HADOOP-2185 > URL: https://issues.apache.org/jira/browse/HADOOP-2185 > Project: Hadoop > Issue Type: Improvement > Components: conf, dfs, mapred > Affects Versions: 0.15.0 > Reporter: Konstantin Shvachko > Fix For: 0.16.0 > > > Looked at the issues related to port rolling. My impression is that port rolling is required only for the unit tests to run. > Even the name-node port should roll there, which we don't have now, in order to be able to start 2 cluster for testing say dist cp. > For real clusters on the contrary port rolling is not desired and some times even prohibited. > So we should have a way of to ban port rolling. My proposition is to > # use ephemeral port 0 if port rolling is desired > # if a specific port is specified then port rolling should not happen at all, meaning that a > server is either able or not able to start on that particular port. > The desired port is specified via configuration parameters. > - Name-node: fs.default.name = host:port > - Data-node: dfs.datanode.port > - Job-tracker: mapred.job.tracker = host:port > - Task-tracker: mapred.task.tracker.report.bindAddress = host > Task-tracker currently does not have an option to specify port, it always uses the ephemeral port 0, > and therefore I propose to add one. > - Secondary node does not need a port to listen on. > For info servers we have two sets of config variables *.info.bindAddress and *.info.port > except for the task tracker, which calls them *.http.bindAddress and *.http.port instead of "info". > With respect to the info servers I propose to completely eliminate the port parameters, and form > *.info.bindAddress = host:port > Info servers should do the same thing, namely start or fail on the specified port if it is not 0, > and start on any free port if it is ephemeral. > For the task-tracker I would rename tasktracker.http.bindAddress to mapred.task.tracker.info.bindAddress > For the data-node the info dfs.datanode.info.bindAddress should be included into the default config. > Is there a reason why it is not there? > This is the summary of proposed changes: > || Server || current name = value || proposed name = value || > | NameNode | fs.default.name = host:port | same | > | | dfs.info.bindAddress = host | dfs.info.bindAddress = host:port | > | DataNode | dfs.datanode.port = port | same | > | | dfs.datanode.info.bindAddress = host | dfs.datanode.info.bindAddress = host:port | > | | dfs.datanode.info.port = port | eliminate | > | JobTracker| mapred.job.tracker = host:port | same | > | | mapred.task.tracker.info.bindAddress = host | mapred.task.tracker.info.bindAddress = host:port | > | | mapred.task.tracker.info.port = port | eliminate | > | TaskTracker| mapred.task.tracker.report.bindAddress = host | mapred.task.tracker.report.bindAddress = host:port | > | | tasktracker.http.bindAddress = host | mapred.task.tracker.info.bindAddress = host:port | > | | tasktracker.http.port = port | eliminate | > | SecondaryNameNode | dfs.secondary.info.bindAddress = host | dfs.secondary.info.bindAddress = host:port | > | | dfs.secondary.info.port = port | eliminate | > Do we also want to set some uniform naming convention for the configuration variables? > Like having hdfs instead of dfs, or info instead of http, or systematically using either datanode > or data.node would make that look better in my opinion. > So these are all +*api*+ changes. I would +*really*+ like some feedback on this, especially from > people who deal with configuration issues on practice. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.