incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Zheng <bearzheng2...@gmail.com>
Subject Re: Deploy S4 App with Yarn -- Memory Problem
Date Fri, 16 Nov 2012 02:23:18 GMT
Hi,

I got another problem when running the Twitter Trend Example in cluster
mode.
I can deploy the twitter-counter and twitter-adapter.
For adapter, there are 2 containers running, one is the AppMaster and the
other is the adapter application.
For counter, there are also 2 containers running, one is the AppMaster and
the other is the counter application.
There should be one more container running for counter application, yes? I
check the logs of the AppMaster of twitter-counter.
Two containers for counter application were set up, but one of them was
completed. I don't know why.
Is this problem related to memory allocation?

10:10:55.925 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Initializing ApplicationMaster with args [--container_memory, 10,
--num_containers, 2, --priority, 0, -c, counter, -zk,
testing.machine1:2181]
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Application master for app, appId=1, clustertimestamp=1353031752144,
attemptId=1
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Application master args = []
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg /
Amount of memory in MB to be requested to host the S4 node /
container_memory / 10
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg /
No. of containers on which the S4 node needs to be hosted /
num_containers / 2
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg /
Application Priority. Default 0 / priority / 0
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg /
The cluster (or application) name / c / counter
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - arg /
Zookeeper connection string / zk / testing.machine1:2181
10:10:56.062 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Application master args = [[ option: container_memory  [ARG] :: Amount
of memory in MB to be requested to host the S4 node ], [ option:
num_containers  [ARG] :: No. of containers on which the S4 node needs
to be hosted ], [ option: priority  [ARG] :: Application Priority.
Default 0 ], [ option: c cluster  [ARG] :: The cluster (or
application) name ], [ option: zk  [ARG] :: Zookeeper connection
string ]]
10:10:56.111 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Starting ApplicationMaster
10:10:56.209 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Connecting to ResourceManager at testing.machine1/192.168.0.100:8030
10:10:56.672 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Min
mem capability of resources in this cluster 512
10:10:56.673 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Max
mem capability of resources in this cluster 3072
10:10:56.673 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Container memory specified below min threshold of cluster. Using min
value., specified=10, min=512
10:10:56.673 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=0, appDone=false, total=2,
requested=0, completed=0, failed=0, currentAllocated=0
10:10:57.682 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=2
10:10:57.684 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=1, releasedSet=0,
progress=0.0
10:10:57.684 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Requested container ask: {Priority: {Priority: 0}, Capability: memory:
512}
10:10:57.698 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:10:57.698 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 5632
10:10:57.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:10:57.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=0, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=0
10:10:57.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=1, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=0
10:10:58.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:10:58.699 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:10:58.715 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=2
10:10:58.717 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Launching S4 node on a new container.,
containerId=container_1353031752144_0001_01_000002,
containerNode=testing.machine1:40558,
containerNodeURI=testing.machine1:8042, containerStateNEW,
containerResourceMemory512
10:10:58.719 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Launching S4 node on a new container.,
containerId=container_1353031752144_0001_01_000003,
containerNode=testing.machine2:44677,
containerNodeURI=testing.machine2:8042, containerStateNEW,
containerResourceMemory512
10:10:58.719 [Thread-2] DEBUG o.a.s.tools.yarn.S4ApplicationMaster -
Connecting to ContainerManager for
containerid=container_1353031752144_0001_01_000002
10:10:58.719 [Thread-2] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Connecting to ContainerManager at testing.machine1:40558
10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=1, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:10:58.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=2, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:10:58.726 [Thread-3] DEBUG o.a.s.tools.yarn.S4ApplicationMaster -
Connecting to ContainerManager for
containerid=container_1353031752144_0001_01_000003
10:10:58.726 [Thread-3] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Connecting to ContainerManager at testing.machine2:44677
10:10:58.735 [Thread-3] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Setting up container launch container for
containerid=container_1353031752144_0001_01_000003
10:10:58.735 [Thread-2] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Setting up container launch container for
containerid=container_1353031752144_0001_01_000002
10:10:58.740 [Thread-2] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Using default user name root
10:10:58.740 [Thread-3] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Using default user name root
10:10:59.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:10:59.726 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=2, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:10:59.728 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=3, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:00.729 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:00.729 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=3, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:00.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=4, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:01.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:01.731 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=4, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:01.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=5, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:02.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:02.733 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=5, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:02.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=6, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:03.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:03.736 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:03.738 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=6, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:03.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=7, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:04.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:04.739 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=7, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:04.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=8, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:05.742 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:05.743 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=8, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:05.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=9, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:06.745 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:06.746 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=9, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:06.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=10, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:07.748 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:07.749 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:07.753 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=10, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:07.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=11, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:08.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:08.754 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:08.756 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=11, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:08.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=12, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:09.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:09.757 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:09.764 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=12, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:09.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=13, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:10.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:10.765 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=13, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:10.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=14, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:11.768 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:11.769 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:11.772 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=14, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:11.773 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=15, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:12.773 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:12.774 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:12.776 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=15, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:12.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=16, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:13.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:13.777 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:13.779 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=16, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:13.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=17, appDone=false, total=2,
requested=2, completed=0, failed=0, currentAllocated=2
10:11:14.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:14.780 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.0
10:11:14.782 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:14.782 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=1
10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
container status for containerID=
container_1353031752144_0001_01_000003, state=COMPLETE, exitStatus=0,
diagnostics=
10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Container completed successfully.,
containerId=container_1353031752144_0001_01_000003
10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=17, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:14.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=18, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:15.783 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:15.784 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.5
10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=18, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:15.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=19, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:16.786 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:16.787 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.5
10:11:16.789 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=19, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:16.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=20, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:17.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:17.790 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.5
10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=20, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:17.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=21, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:18.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Asking RM for containers, askCount=0
10:11:18.793 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Sending request to RM for containers, requestedSet=0, releasedSet=0,
progress=0.5
10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, allocatedCnt=0
10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current available resources in the cluster memory: 4608
10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster - Got
response from RM for container ask, completedCnt=0
10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=21, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2
10:11:18.796 [main] INFO  o.a.s.tools.yarn.S4ApplicationMaster -
Current application state: loop=22, appDone=false, total=2,
requested=2, completed=1, failed=0, currentAllocated=2



On Fri, Nov 16, 2012 at 2:32 AM, Matthieu Morel <matthieu.morel@gmail.com>wrote:

> Hi,
>
> What specific parameter are you trying to set in the yarn
> configuration? It looks like you have a max of 128MB per container.
>
> What seems to be failing is the application master. It uses (by
> default) 128MB for heap + default stack size, but that would be over
> 128MB.
>
> Of course, we still have to properly handle all  parameters available in
> the yarn command, including memory settings, but from what I see the
> issue you are facing is related to the configuration of your Yarn
> cluster. You can get more information from other log files (apart from
> the client console logs), maybe that can help as well. But somewhere
> you seem to be capping the container memory to an insufficient value.
>
> Regards,
>
> Matthieu
>
> On Thu, Nov 15, 2012 at 9:12 AM, Frank Zheng <bearzheng2011@gmail.com>
> wrote:
> > Hi All,
> >
> > I am confused about the memory configuration.
> > In the S4ApplicationMaster.java, it says:
> >         // A resource ask has to be at least the minimum of the
> capability
> > of the cluster, the value has to be
> >         // a multiple of the min value and cannot exceed the max.
> >         // If it is not an exact multiple of min, the RM will allocate to
> > the nearest multiple of min
> > So I set the minimum memory as 128 in the yarn-site.xml
> > But when I deployed the Twitter Counter application, I got this error.
> >
> > 17:04:25.558 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
> > application report from ASM for, appId=1, clientToken=null,
> appDiagnostics=,
> > appMasterHost=, appQueue=default, appMasterRpcPort=0,
> > appStartTime=1352970249485, yarnAppState=RUNNING,
> > distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
> > 17:04:26.560 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
> > application report from ASM for, appId=1, clientToken=null,
> appDiagnostics=,
> > appMasterHost=, appQueue=default, appMasterRpcPort=0,
> > appStartTime=1352970249485, yarnAppState=RUNNING,
> > distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
> > 17:04:27.563 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
> > application report from ASM for, appId=1, clientToken=null,
> appDiagnostics=,
> > appMasterHost=, appQueue=default, appMasterRpcPort=0,
> > appStartTime=1352970249485, yarnAppState=RUNNING,
> > distributedFinalState=UNDEFINED, appTrackingUrl=, appUser=root
> > 17:04:28.569 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient - Got
> > application report from ASM for, appId=1, clientToken=null,
> > appDiagnostics=Application application_1352970173321_0001 failed 1 times
> due
> > to AM Container for appattempt_1352970173321_0001_000001 exited with
> > exitCode: 143 due to: Container
> > [pid=12119,containerID=container_1352970173321_0001_01_000001] is running
> > beyond physical memory limits. Current usage: 138.2mb of 128.0mb physical
> > memory used; 1.1gb of 5.2gb virtual memory used. Killing container.
> > Dump of the process-tree for container_1352970173321_0001_01_000001 :
> >     |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
> >     |- 12213 12119 12119 12119 (java) 285 11 1085763584 35085 java
> -Xmx128m
> > org.apache.s4.tools.yarn.S4ApplicationMaster --container_memory 10
> > --num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
> >     |- 12119 9509 12119 12119 (bash) 4 3 108703744 304 /bin/bash -c java
> > -Xmx128m org.apache.s4.tools.yarn.S4ApplicationMaster --container_memory
> 10
> > --num_containers 3 --priority 0 -c counter -zk testing.machine1:2181
> >
> 1>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stdout
> >
> 2>/home/hadoop/data/log/application_1352970173321_0001/container_1352970173321_0001_01_000001/AppMaster.stderr
> >
> >
> > .Failing this attempt.. Failing the application., appMasterHost=,
> > appQueue=default, appMasterRpcPort=0, appStartTime=1352970249485,
> > yarnAppState=FAILED, distributedFinalState=FAILED, appTrackingUrl=,
> > appUser=root
> > 17:04:28.569 [main] INFO  o.apache.s4.tools.yarn.S4YarnClient -
> Application
> > did not finish. YarnState=FAILED, DSFinalStatus=FAILED. Breaking
> monitoring
> > loop
> > 17:04:28.569 [main] ERROR o.apache.s4.tools.yarn.S4YarnClient -
> Application
> > failed to complete successfully
> >
> >
> > Should the S4AppMaster use the multiple of minimum memory automatically?
> Why
> > the memory of container is only 128 MB?
> >
> >
> > Sincerely,
> > Zheng Yu
> > Mobile:  (852) 60670059
> > Email:    bearzheng2011@gmail.com
> >
> >
> >
>



-- 
Sincerely,
Zheng Yu
Mobile:  (852) 60670059
Email:    bearzheng2011@gmail.com

Mime
View raw message