Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@giraph.apache.org
Date: Thu, 30 Jun 2016 21:23:10 +0000 (UTC)
From: "Sergey Edunov (JIRA)" <jira@apache.org>
To: giraph-dev@incubator.apache.org
Message-ID: <JIRA.12702596.1395309504000.8110.1467321790350@Atlassian.JIRA>
In-Reply-To: <JIRA.12702596.1395309504000@Atlassian.JIRA>
References: <JIRA.12702596.1395309504000@Atlassian.JIRA> <JIRA.12702596.1395309504789@arcas>
Subject: [jira] [Commented] (GIRAPH-882) List of zookeeper connection
 strings is trimmed by Hadoop counters.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 30 Jun 2016 21:23:12 -0000


    [ https://issues.apache.org/jira/browse/GIRAPH-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357880#comment-15357880 ] 

Sergey Edunov commented on GIRAPH-882:
--------------------------------------

Well, we don't need to worry about multiple zookeepers anymore, because we can only have one, and single host name can not exceed 255 characters (RFC 1035). On many systems this limit is actually lower. You can try: 
$getconf HOST_NAME_MAX
64   - this is on my linux box
255 - this is on my mac 

Also, note if you want to properly take care of a single host name being that long, you need to fix much more than what has been fixed so far. See for example 

ZooKeeperManager#createCandidateStamp(), it has this line:

    Path myCandidacyPath = new Path(
        taskDirectory, myHostname +
        HOSTNAME_TASK_SEPARATOR + taskPartition);


or 
ZooKeeperManager#onlineZooKeeperServer(), it has this line:
      Path myReadyPath = new Path(
          serverDirectory, myHostname +
          HOSTNAME_TASK_SEPARATOR + taskPartition +
          HOSTNAME_TASK_SEPARATOR + zkBasePort);

we use host names pretty much everywhere, if this is what you want to fix when much more needs to be done. 


> List of zookeeper connection strings is trimmed by Hadoop counters.
> -------------------------------------------------------------------
>
>                 Key: GIRAPH-882
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-882
>             Project: Giraph
>          Issue Type: Bug
>          Components: zookeeper
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Lukas Nalezenec
>         Attachments: GIRAPH-882-rev2.patch, GIRAPH-882-rev3.patch, GIRAPH-882.patch, testrun.log
>
>
> We are running job with quorum of 3 zookeepers. Each serves has got long name (turing452.fi.callan.de:22181). Connection strings are stored to Hadoop Counters (for example: turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turing488.fi.callan.de:22181) but since name of counter is limited to ~63 character the connection string is trimmed (turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turin).
> 14/03/18 23:44:41 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
> 14/03/18 23:44:41 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turin sessionTimeout=60000 
> Exception in thread "main" java.net.UnknownHostException: turin
> 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> 	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
> 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
> 	at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
> 	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
> 	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
> 	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:60)
> 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
> 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
> 	at org.apache.giraph.zk.ZooKeeperExt.<init>(ZooKeeperExt.java:114)
> 	at org.apache.giraph.job.JobProgressTracker.<init>(JobProgressTracker.java:69)
> 	at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:255)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)