Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E63B4106AB for ; Mon, 1 Jul 2013 18:20:00 +0000 (UTC) Received: (qmail 77927 invoked by uid 500); 1 Jul 2013 18:19:54 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 77791 invoked by uid 500); 1 Jul 2013 18:19:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 77782 invoked by uid 99); 1 Jul 2013 18:19:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jul 2013 18:19:47 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ojoshi@hortonworks.com designates 209.85.215.47 as permitted sender) Received: from [209.85.215.47] (HELO mail-la0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jul 2013 18:19:41 +0000 Received: by mail-la0-f47.google.com with SMTP id fe20so4714862lab.20 for ; Mon, 01 Jul 2013 11:19:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=0O0Q2rLJAWvWSHqOSMvIup01cObFiEd02FHtxbaoHeM=; b=Jmjq6Pf+rm/7g5MP/X3cPnFgqS3Rswjv3wdwD6RiNFHrTI3/LsaN/pivXRiSvXZnLv a5T9kneEJzrGEduzjxXgy9/JKVnjOsg6p7bjNThAf0af1pg1PrVSNJkeRgH7nTG03lP9 D4h41fKAq50rvTbmxfQD6PWxi/bnsHnEfPpElw2PjsoYJKOGgRjzxxFhZjz4O5XrfyU6 lMjyvYDD3Z0HQvWYVUhBqE0pce9CgwBHlN8ivXEVS5vPjCKGPHMTMUaz5WJvkcV4gkYN 41R4w/N3+19KWENiP3KLi2CctBDvsVI5qYL3ksbxPPSy4b+/+Tl3k1hRHxTxPJMmmo96 JyIQ== MIME-Version: 1.0 X-Received: by 10.152.25.169 with SMTP id d9mr12492488lag.63.1372702760141; Mon, 01 Jul 2013 11:19:20 -0700 (PDT) Received: by 10.112.26.50 with HTTP; Mon, 1 Jul 2013 11:19:20 -0700 (PDT) In-Reply-To: References: <06006DDA5A27D541991944AC4117E7A96E1C9888@szxeml560-mbx.china.huawei.com> Date: Mon, 1 Jul 2013 11:19:20 -0700 Message-ID: Subject: Re: Yarn HDFS and Yarn Exceptions when processing "larger" datasets. From: Omkar Joshi To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0158b20266969404e07744cf X-Gm-Message-State: ALoCoQl2UBC+wWmn6eBxJGlE1ydtpuUTc+So7QbUPQiAIezSXSZlVtpLCDRehkYNyi1FcdF0ZbYM X-Virus-Checked: Checked by ClamAV on apache.org --089e0158b20266969404e07744cf Content-Type: text/plain; charset=ISO-8859-1 Hi, As I don't know your complete AM code and how your containers are communicating with each other...Certain things which might help you in debugging.... where you are starting your RM (is it really running on 8030???? are you sure there is no previously started RM still running there?) Also in yarn-site.xml can you try changing RM address to something like "localhost:" and configure maximum client thread size for handling AM requests? only your AM is expected to communicate with RM on AM-RM protocol.. by any chance in your code; are containers directly communicating with RM on AM-RM protocol?? The address of the scheduler interface. yarn.resourcemanager.scheduler.address ${yarn.resourcemanager.hostname}:8030 Number of threads to handle scheduler interface. yarn.resourcemanager.scheduler.client.thread-count 50 Thanks, Omkar Joshi *Hortonworks Inc.* On Fri, Jun 28, 2013 at 5:35 AM, blah blah wrote: > Hi > > Sorry to reply so late. I don't have the data you requested (sorry I have > no time, my deadline is within 3 days). However I have observed that this > issue occurs not only for the "larger" datasets (6.8MB), but for all > datasets and all jobs in general. However for smaller datasets (1MB) the AM > does not throw the Exception, only containers throw exceptions (same as in > previous e-mail). When these exception are throws my code (AM and > containers) does not perform any operations on HDFS, they only perform > in-memory computation and communication. Also I have observed that these > exception occur at "random", I couldn't observe any pattern. I can execute > job successfully, then resubmit the job repeating the experiment and these > exceptions occur (no change was made to src code, input dataset,or > execution/input parameters). > > As for the high network usage, as I said I don't have the data. But YARN > is running on nodes which are exclusive for my experiments no other > software runs on these nodes (only OS and YARN). Besides I don't think that > 20 containers working on 1MB dataset (total) can be called high network > usage. > > regards > tmp > > > > 2013/6/26 Devaraj k > >> Hi,**** >> >> ** ** >> >> Could you check the network usage in the cluster when this problem >> occurs? Probably it is causing due to high network usage. **** >> >> ** ** >> >> Thanks**** >> >> Devaraj k**** >> >> ** ** >> >> *From:* blah blah [mailto:tmp5330@gmail.com] >> *Sent:* 26 June 2013 05:39 >> *To:* user@hadoop.apache.org >> *Subject:* Yarn HDFS and Yarn Exceptions when processing "larger" >> datasets.**** >> >> ** ** >> >> Hi All**** >> >> First let me excuse for the poor thread title but I have no idea how to >> express the problem in one sentence. **** >> >> I have implemented new Application Master with the use of Yarn. I am >> using old Yarn development version. Revision 1437315, from 2013-01-23 >> (SNAPSHOT 3.0.0). I can not update to current trunk version, as prototype >> deadline is soon, and I don't have time to include Yarn API changes.**** >> >> Currently I execute experiments in pseudo-distributed mode, I use guava >> version 14.0-rc1. I have a problem with Yarn's and HDFS Exceptions for >> "larger" datasets. My AM works fine and I can execute it without a problem >> for a debug dataset (1MB size). But when I increase the size of input to >> 6.8 MB, I am getting the following exceptions:**** >> >> AM_Exceptions_Stack >> >> Exception in thread "Thread-3" >> java.lang.reflect.UndeclaredThrowableException >> at >> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135) >> at >> org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.allocate(AMRMProtocolPBClientImpl.java:77) >> at >> org.apache.hadoop.yarn.client.AMRMClientImpl.allocate(AMRMClientImpl.java:194) >> at >> org.tudelft.ludograph.app.AppMasterContainerRequester.sendContainerAskToRM(AppMasterContainerRequester.java:219) >> at >> org.tudelft.ludograph.app.AppMasterContainerRequester.run(AppMasterContainerRequester.java:315) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: com.google.protobuf.ServiceException: java.io.IOException: >> Failed on local exception: java.io.IOException: Response is null.; Host >> Details : local host is: "linux-ljc5.site/127.0.0.1"; destination host >> is: "0.0.0.0":8030; >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212) >> at $Proxy10.allocate(Unknown Source) >> at >> org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.allocate(AMRMProtocolPBClientImpl.java:75) >> ... 4 more >> Caused by: java.io.IOException: Failed on local exception: >> java.io.IOException: Response is null.; Host Details : local host is: >> "linux-ljc5.site/127.0.0.1"; destination host is: "0.0.0.0":8030; >> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:760) >> at org.apache.hadoop.ipc.Client.call(Client.java:1240) >> at >> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) >> ... 6 more >> Caused by: java.io.IOException: Response is null. >> at >> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:950) >> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)**** >> >> Container_Exception >> >> Exception in thread "org.apache.hadoop.hdfs.SocketCache@6da0d866" >> java.lang.NoSuchMethodError: >> com.google.common.collect.LinkedListMultimap.values()Ljava/util/List; >> at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:257) >> at org.apache.hadoop.hdfs.SocketCache.access$100(SocketCache.java:45) >> at org.apache.hadoop.hdfs.SocketCache$1.run(SocketCache.java:126) >> at java.lang.Thread.run(Thread.java:662) >> >> **** >> >> As I said this problem does not occur for the 1MB input. For the 6MB >> input nothing is changed except the input dataset. Now a little bit of what >> am I doing, to give you the context of the problem. My AM starts N (debug >> 4) containers and each container reads its input data part. When this >> process is finished I am exchanging parts of input between containers >> (exchanging IDs of input structures, to provide means for communication >> between data structures). During the process of exchanging IDs these >> exceptions occur. I start Netty Server/Client on each container and I use >> ports 12000-12099 as mean of communicating these IDs. **** >> >> Any help will be greatly appreciated. Sorry for any typos and if the >> explanation is not clear just ask for any details you are interested in. >> Currently it is after 2 AM I hope this will be a valid excuse.**** >> >> regards**** >> >> tmp**** >> > > --089e0158b20266969404e07744cf Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

As I don't know your comp= lete AM code and how your containers are communicating with each other...Ce= rtain things which might help you in debugging.... where you are starting y= our RM (is it really running on 8030???? are you sure there is no previousl= y started RM still running there?) Also in yarn-site.xml can you try changi= ng RM address to something like "localhost:<free-port-but-not-defau= lt>" and configure maximum client thread size for handling AM reque= sts? only your AM is expected to communicate with RM on AM-RM protocol.. by= any chance in your code; are containers directly communicating with RM on = AM-RM protocol??

=A0=A0<pro= perty>

=A0 =A0 <descript= ion>The address of the scheduler interfac= e.</description>

=A0 =A0 <name>yarn.resourcemanager.scheduler.address</name>

=A0 =A0 <value>${yarn.resourcemanager.hostname}:8030</value>

=A0 </prop= erty>


=A0 <prope= rty>

=A0 =A0 <descript= ion>Number of threads to handle scheduler= interface.</description= >

=A0 =A0 <name>yarn.resourcemanager.scheduler.client.thread= -count</name>

=A0 =A0 <v= alue>50= </value>

=A0 </prop= erty>



Thanks,
Omkar Joshi


On Fri, Jun 28, 2013 at 5:35 AM, blah bl= ah <tmp5330@gmail.com> wrote:
Hi

Sorry to reply so late. I don= 9;t have the data you requested (sorry I have no time, my deadline is withi= n 3 days). However I have observed that this issue occurs not only for the = "larger" datasets (6.8MB), but for all datasets and all jobs in g= eneral. However for smaller datasets (1MB) the AM does not throw the Except= ion, only containers throw exceptions (same as in previous e-mail). When th= ese exception are throws my code (AM and containers) does not perform any o= perations on HDFS, they only perform in-memory computation and communicatio= n. Also I have observed that these exception occur at "random", I= couldn't observe any pattern. I can execute job successfully, then res= ubmit the job repeating the experiment and these exceptions occur (no chang= e was made to src code, input dataset,or execution/input parameters).

As for the high network usage, as I said I don't have the dat= a. But YARN is running on nodes which are exclusive for my experiments no o= ther software runs on these nodes (only OS and YARN). Besides I don't t= hink that 20 containers working on 1MB dataset (total) can be called high n= etwork usage.

regards
tmp



2013/6/26 Devaraj k <devaraj.k@huawei.com>

Hi,

=A0

=A0=A0 Could you check the network usage in the clust= er when this problem occurs? Probably it is causing due to high network usa= ge.

=A0

Thanks

Devaraj k

=A0

From: blah blah [mailto:tmp5330@gmail.com]
Sent: 26 June 2013 05:39
To: user= @hadoop.apache.org
Subject: Yarn HDFS and Yarn Exceptions when processing "larger&= quot; datasets.

=A0

Hi All

First let me excuse for the poor= thread title but I have no idea how to express the problem in one sentence= .

I have implemented new Applicati= on Master with the use of Yarn. I am using old Yarn development version. Re= vision 1437315, from 2013-01-23 (SNAPSHOT 3.0.0). I can not update to curre= nt trunk version, as prototype deadline is soon, and I don't have time to include Yarn API changes.

Currently I execute experiments = in pseudo-distributed mode, I use guava version 14.0-rc1. I have a problem = with Yarn's and HDFS Exceptions for "larger" datasets. My AM = works fine and I can execute it without a problem for a debug dataset (1MB size). But when I increase the size of = input to 6.8 MB, I am getting the following exceptions:

AM_Exceptions_Stack

Exception in thread "Thread-3" java.lang.reflect.UndeclaredThrowa= bleException
=A0=A0=A0 at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionP= BImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
=A0=A0=A0 at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClient= Impl.allocate(AMRMProtocolPBClientImpl.java:77)
=A0=A0=A0 at org.apache.hadoop.yarn.client.AMRMClientImpl.allocate(AMRMClie= ntImpl.java:194)
=A0=A0=A0 at org.tudelft.ludograph.app.AppMasterContainerRequester.sendCont= ainerAskToRM(AppMasterContainerRequester.java:219)
=A0=A0=A0 at org.tudelft.ludograph.app.AppMasterContainerRequester.run(AppM= asterContainerRequester.java:315)
=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Faile= d on local exception: java.io.IOException: Response is null.; Host Details = : local host is: "linux-ljc5.site/127.0.0.1"; destination host is: "0.0.0.0":8= 030;
=A0=A0=A0 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Protobu= fRpcEngine.java:212)
=A0=A0=A0 at $Proxy10.allocate(Unknown Source)
=A0=A0=A0 at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClient= Impl.allocate(AMRMProtocolPBClientImpl.java:75)
=A0=A0=A0 ... 4 more
Caused by: java.io.IOException: Failed on local exception: java.io.IOExcept= ion: Response is null.; Host Details : local host is: "linux-ljc5.site= /127.0.0.1"; destin= ation host is: "0.0.0.0":8030;
=A0=A0=A0 at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:760= )
=A0=A0=A0 at org.apache.hadoop.ipc.Client.call(Client.java:1240)
=A0=A0=A0 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Protobu= fRpcEngine.java:202)
=A0=A0=A0 ... 6 more
Caused by: java.io.IOException: Response is null.
=A0=A0=A0 at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Cli= ent.java:950)
=A0=A0=A0 at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)

Container_Exception

Exception in thread "org.apache.hadoop.hdfs.SocketCache@6da0d866= " java.lang.NoSuchMethodError: com.google.common.collect.LinkedLis= tMultimap.values()Ljava/util/List;
=A0=A0=A0 at org.apache.hadoop.hdfs.SocketCache.clear(SocketCache.java:257)=
=A0=A0=A0 at org.apache.hadoop.hdfs.SocketCache.access$100(SocketCache.java= :45)
=A0=A0=A0 at org.apache.hadoop.hdfs.SocketCache$1.run(SocketCache.java:126)=
=A0=A0=A0 at java.lang.Thread.run(Thread.java:662)

As I said this problem does not = occur for the 1MB input. For the 6MB input nothing is changed except the in= put dataset. Now a little bit of what am I doing, to give you the context o= f the problem. My AM starts N (debug 4) containers and each container reads its input data part. When = this process is finished I am exchanging parts of input between containers = (exchanging IDs of input structures, to provide means for communication bet= ween data structures). During the process of exchanging IDs these exceptions occur. I start Netty Server/Cli= ent on each container and I use ports 12000-12099 as mean of communicating = these IDs.

Any help will be greatly appreci= ated. Sorry for any typos and if the explanation is not clear just ask for = any details you are interested in. Currently it is after 2 AM I hope this w= ill be a valid excuse.

regards

tmp



--089e0158b20266969404e07744cf--