giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lee Dongjin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-882) List of zookeeper connection strings is trimmed by Hadoop counters.
Date Wed, 14 Jan 2015 15:19:35 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277034#comment-14277034
] 

Lee Dongjin commented on GIRAPH-882:
------------------------------------

For this bug, two facts are related: First, the Zookeeper server list is stored temporarily
in hdfs, in an empty file named with "zkServerList_HOST1 PORT1 HOST2 PORT2 ...". However,
there is a length limitation on hdfs' filename: if it exceeds 255 characters, the tail of
the filename can be trimmed. Second, In order to display the job summary on console, Giraph
stores the Zookeeper server list into GiraphConstants.ZOOKEEPER_SERVER_PORT_COUNTER_GROUP
counter group. But it also has length limitation (by default, 128) - by this reason, it can
be also trimmed. (In short, the bug description is a little bit inaccurate.)

Needless to say, the first one is the heart of this problem. This patch fixes that problem,
by storing given Zookeeper server list in hdfs file's contents, instead of the filename. I
validated this patch using 5 Zookeeper servers, each of whose name was 53 characters. That
is:

the-answer-to-life-the-universe-and-everything-1:2181
the-answer-to-life-the-universe-and-everything-2:2181
the-answer-to-life-the-universe-and-everything-3:2181
the-answer-to-life-the-universe-and-everything-4:2181
the-answer-to-life-the-universe-and-everything-5:2181

With commas, the whole length of Zookeeper server list is 53 * 5 + 4 = 269 > 255.

> List of zookeeper connection strings is trimmed by Hadoop counters.
> -------------------------------------------------------------------
>
>                 Key: GIRAPH-882
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-882
>             Project: Giraph
>          Issue Type: Bug
>          Components: zookeeper
>    Affects Versions: 1.1.0
>            Reporter: Lukas Nalezenec
>
> We are running job with quorum of 3 zookeepers. Each serves has got long name (turing452.fi.callan.de:22181).
Connection strings are stored to Hadoop Counters (for example: turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turing488.fi.callan.de:22181)
but since name of counter is limited to ~63 character the connection string is trimmed (turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turin).
> 14/03/18 23:44:41 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
> 14/03/18 23:44:41 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=turing452.fi.callan.de:22181,turing124.fi.callan.de:22181,turin
sessionTimeout=60000 
> Exception in thread "main" java.net.UnknownHostException: turin
> 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
> 	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
> 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
> 	at java.net.InetAddress.getAllByName0(InetAddress.java:1246)
> 	at java.net.InetAddress.getAllByName(InetAddress.java:1162)
> 	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
> 	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:60)
> 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
> 	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
> 	at org.apache.giraph.zk.ZooKeeperExt.<init>(ZooKeeperExt.java:114)
> 	at org.apache.giraph.job.JobProgressTracker.<init>(JobProgressTracker.java:69)
> 	at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:255)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message