giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Edunov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-882) List of zookeeper connection strings is trimmed by Hadoop counters.
Date Thu, 30 Jun 2016 21:23:10 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357880#comment-15357880
] 

Sergey Edunov commented on GIRAPH-882:
--------------------------------------

Well, we don't need to worry about multiple zookeepers anymore, because we can only have one,
and single host name can not exceed 255 characters (RFC 1035). On many systems this limit
is actually lower. You can try: 
$getconf HOST_NAME_MAX
64   - this is on my linux box
255 - this is on my mac 

Also, note if you want to properly take care of a single host name being that long, you need
to fix much more than what has been fixed so far. See for example 

ZooKeeperManager#createCandidateStamp(), it has this line:

    Path myCandidacyPath = new Path(
        taskDirectory, myHostname +
        HOSTNAME_TASK_SEPARATOR + taskPartition);


or 
ZooKeeperManager#onlineZooKeeperServer(), it has this line:
      Path myReadyPath = new Path(
          serverDirectory, myHostname +
          HOSTNAME_TASK_SEPARATOR + taskPartition +
          HOSTNAME_TASK_SEPARATOR + zkBasePort);

we use host names pretty much everywhere, if this is what you want to fix when much more needs
to be done. 




> List of zookeeper connection strings is trimmed by Hadoop counters.
> -------------------------------------------------------------------
>
>                 Key: GIRAPH-882
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-882
>             Project: Giraph
>          Issue Type: Bug
>          Components: zookeeper
>    Affects Versions: 1.1.0, 1.2.0
>            Reporter: Lukas Nalezenec
>         Attachments: GIRAPH-882-rev2.patch, GIRAPH-882-rev3.patch, GIRAPH-882.patch,
testrun.log
>
>
> We are running job with quorum of 3 zookeepers. Each serves has got long name (turing452.fi.callan.de:22181).
Connection strings are stored to Hadoop Counters (for example: turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turing488.fi.callan.de:22181)
but since name of counter is limited to ~63 character the connection string is trimmed (turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turin).
> 14/03/18 23:44:41 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
> 14/03/18 23:44:41 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turin
sessionTimeout=60000 
> Exception in thread "main" java.net.UnknownHostException: turin
> 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> 	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
> 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
> 	at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
> 	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
> 	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
> 	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:60)
> 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
> 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
> 	at org.apache.giraph.zk.ZooKeeperExt.<init>(ZooKeeperExt.java:114)
> 	at org.apache.giraph.job.JobProgressTracker.<init>(JobProgressTracker.java:69)
> 	at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:255)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message