hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yunjiong zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6285) Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
Date Sat, 04 Mar 2017 19:38:45 GMT

    [ https://issues.apache.org/jira/browse/YARN-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895837#comment-15895837
] 

yunjiong zhao commented on YARN-6285:
-------------------------------------

I checked the returned result by rest API, the major size was ResourceRequest.
If in https://issues.apache.org/jira/browse/YARN-6280, change the default behavior to not
show ResourceRequest will help.
However it's not enough, I have 2 reasons:
1. Slowness in getApplications, below stack trace files shows it spend at least 2.25 seconds
in getApplications. 
{code}
 grep -A20 " #7876 daemon " 829
"363440407@qtp-1966670937-117" #7876 daemon prio=5 os_prio=0 tid=0x00007f12093a2800 nid=0x1c46
runnable [0x00007f05344b8000]
   java.lang.Thread.State: RUNNABLE
	at java.util.regex.Matcher.search(Matcher.java:1248)
	at java.util.regex.Matcher.find(Matcher.java:637)
	at java.util.regex.Matcher.replaceAll(Matcher.java:951)
	at java.lang.String.replace(String.java:2240)
	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1559)
	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:89)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:86)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock.fetchData(AppsBlock.java:84)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:101)
	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
{code}

{code}
 grep -A20 " #7876 daemon " 838
"363440407@qtp-1966670937-117" #7876 daemon prio=5 os_prio=0 tid=0x00007f12093a2800 nid=0x1c46
runnable [0x00007f05344b8000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.hash(HashMap.java:338)
	at java.util.HashMap.putMapEntries(HashMap.java:514)
	at java.util.HashMap.putAll(HashMap.java:784)
	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1466)
	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:89)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:86)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock.fetchData(AppsBlock.java:84)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:101)
	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
	at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
	at org.apache.hadoop.yarn.webapp.view.HtmlBlock$Block.subView(HtmlBlock.java:43)
{code}

{code}
grep -A20 " #7876 daemon " 839
"363440407@qtp-1966670937-117" #7876 daemon prio=5 os_prio=0 tid=0x00007f12093a2800 nid=0x1c46
runnable [0x00007f05344b8000]
   java.lang.Thread.State: RUNNABLE
	at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
	at java.util.LinkedHashSet.<init>(LinkedHashSet.java:169)
	at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1569)
	- locked <0x00007f06bb34c0b8> (a org.apache.hadoop.security.UserGroupInformation)
	at org.apache.hadoop.security.authorize.AccessControlList.isUserInList(AccessControlList.java:234)
	at org.apache.hadoop.security.authorize.AccessControlList.isUserAllowed(AccessControlList.java:244)
	at org.apache.hadoop.yarn.security.AdminACLsManager.isAdmin(AdminACLsManager.java:109)
	at org.apache.hadoop.yarn.server.security.ApplicationACLsManager.checkAccess(ApplicationACLsManager.java:134)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkAccess(ClientRMService.java:280)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:759)
	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:89)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:86)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock.fetchData(AppsBlock.java:84)
	at org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:101)
	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
{code}

{code}
stat 829
  File: `829'
  Size: 279084    	Blocks: 552        IO Block: 4096   regular file
Device: 801h/2049d	Inode: 1707225     Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  680/  hadoop)   Gid: (  680/  hadoop)
Access: 2017-03-04 11:58:19.006442408 -0700
Modify: 2017-02-28 21:13:59.879427685 -0700
Change: 2017-02-28 21:13:59.879427685 -0700

stat 839
  File: `839'
  Size: 285122    	Blocks: 560        IO Block: 4096   regular file
Device: 801h/2049d	Inode: 1707235     Links: 1
Access: (0664/-rw-rw-r--)  Uid: (  680/  hadoop)   Gid: (  680/  hadoop)
Access: 2017-03-04 11:58:19.061464815 -0700
Modify: 2017-02-28 21:14:02.147536200 -0700
Change: 2017-02-28 21:14:02.147536200 -0700
{code}
2. getApplications calls not from Rest API caused lots of GC issue as below GC log and stack
trace shows.
{code}
2017-03-03T01:52:25.851-0700: 50959.694: [GC pause (G1 Evacuation Pause) (young), 0.4000028
secs]
2017-03-03T01:52:34.631-0700: 50968.474: [GC pause (G1 Evacuation Pause) (young), 0.3301571
secs]
2017-03-03T01:52:43.229-0700: 50977.071: [GC pause (G1 Evacuation Pause) (young), 0.3162636
secs]
2017-03-03T01:52:52.298-0700: 50986.141: [GC pause (G1 Evacuation Pause) (young), 0.5773488
secs]
2017-03-03T01:53:01.742-0700: 50995.585: [GC pause (G1 Evacuation Pause) (young), 0.6228938
secs]
2017-03-03T01:53:11.042-0700: 51004.885: [GC pause (G1 Evacuation Pause) (young), 0.5198130
secs]
2017-03-03T01:53:20.631-0700: 51014.474: [GC pause (G1 Evacuation Pause) (young), 1.0241827
secs]
2017-03-03T01:53:29.838-0700: 51023.680: [GC pause (G1 Evacuation Pause) (young), 1.1056881
secs]
2017-03-03T01:53:39.386-0700: 51033.228: [GC pause (G1 Evacuation Pause) (young), 1.2069758
secs]
2017-03-03T01:53:46.515-0700: 51040.358: [GC pause (G1 Evacuation Pause) (young), 1.2806813
secs]
2017-03-03T01:53:52.743-0700: 51046.586: [GC pause (G1 Evacuation Pause) (young), 1.3660887
secs]
2017-03-03T01:53:59.569-0700: 51053.412: [GC pause (G1 Evacuation Pause) (young), 1.7237208
secs]
2017-03-03T01:54:07.152-0700: 51060.995: [GC pause (G1 Evacuation Pause) (young), 2.2575946
secs]
2017-03-03T01:54:14.715-0700: 51068.558: [GC pause (G1 Evacuation Pause) (young), 2.1216675
secs]
2017-03-03T01:54:21.944-0700: 51075.787: [GC pause (G1 Evacuation Pause) (young), 2.2341855
secs]
2017-03-03T01:54:29.803-0700: 51083.646: [GC pause (G1 Evacuation Pause) (young), 2.1069973
secs]
2017-03-03T01:54:37.782-0700: 51091.624: [GC pause (G1 Evacuation Pause) (young), 2.2493109
secs]
2017-03-03T01:54:46.309-0700: 51100.151: [GC pause (G1 Evacuation Pause) (young), 2.4593256
secs]
2017-03-03T01:54:55.607-0700: 51109.450: [GC pause (G1 Evacuation Pause) (young), 2.7311687
secs]
2017-03-03T01:55:04.556-0700: 51118.399: [GC pause (G1 Evacuation Pause) (young), 3.3981589
secs]
2017-03-03T01:55:13.898-0700: 51127.741: [GC pause (G1 Evacuation Pause) (young), 3.8448210
secs]
2017-03-03T01:55:23.763-0700: 51137.605: [GC pause (G1 Evacuation Pause) (young), 4.1505401
secs]
2017-03-03T01:55:32.903-0700: 51146.746: [GC pause (G1 Evacuation Pause) (young), 3.9167032
secs]
2017-03-03T01:55:43.052-0700: 51156.895: [GC pause (G1 Evacuation Pause) (young), 4.5987073
secs]
2017-03-03T01:55:53.258-0700: 51167.101: [GC pause (G1 Evacuation Pause) (young), 4.9785920
secs]
2017-03-03T01:56:03.378-0700: 51177.220: [GC pause (G1 Evacuation Pause) (young), 4.5123468
secs]
2017-03-03T01:56:13.127-0700: 51186.970: [GC pause (G1 Evacuation Pause) (young), 4.3864774
secs]
2017-03-03T01:56:23.580-0700: 51197.423: [GC pause (G1 Evacuation Pause) (young), 4.2933490
secs]
2017-03-03T01:56:33.396-0700: 51207.238: [GC pause (G1 Evacuation Pause) (young), 4.1503730
secs]
2017-03-03T01:56:43.735-0700: 51217.578: [GC pause (G1 Evacuation Pause) (young), 3.9017458
secs]
2017-03-03T01:56:53.717-0700: 51227.559: [GC pause (G1 Evacuation Pause) (young), 4.4856238
secs]
2017-03-03T01:57:03.763-0700: 51237.606: [GC pause (G1 Evacuation Pause) (young), 4.1391234
secs]
2017-03-03T01:57:13.382-0700: 51247.225: [GC pause (G1 Evacuation Pause) (young), 3.7430989
secs]
2017-03-03T01:57:23.240-0700: 51257.083: [GC pause (G1 Evacuation Pause) (young), 3.7170368
secs]
2017-03-03T01:57:32.502-0700: 51266.345: [GC pause (G1 Evacuation Pause) (young), 4.2083496
secs]
2017-03-03T01:57:42.325-0700: 51276.167: [GC pause (G1 Evacuation Pause) (young), 4.4536413
secs]
2017-03-03T01:57:52.080-0700: 51285.923: [GC pause (G1 Evacuation Pause) (young), 4.4747331
secs]
2017-03-03T01:58:02.412-0700: 51296.255: [GC pause (G1 Evacuation Pause) (young), 4.4471336
secs]
2017-03-03T01:58:12.625-0700: 51306.468: [GC pause (G1 Evacuation Pause) (young), 4.7383735
secs]
2017-03-03T01:58:23.470-0700: 51317.312: [GC pause (G1 Evacuation Pause) (young), 4.4978315
secs]
2017-03-03T01:58:33.816-0700: 51327.659: [GC pause (G1 Evacuation Pause) (young), 3.8896589
secs]
2017-03-03T01:58:43.078-0700: 51336.921: [GC pause (G1 Evacuation Pause) (young), 3.5723613
secs]
2017-03-03T01:58:52.025-0700: 51345.868: [GC pause (G1 Evacuation Pause) (young), 3.1778834
secs]
2017-03-03T01:59:01.337-0700: 51355.180: [GC pause (G1 Evacuation Pause) (young), 2.9721626
secs]
2017-03-03T01:59:09.891-0700: 51363.733: [GC pause (G1 Evacuation Pause) (young), 2.9824072
secs]
2017-03-03T01:59:18.260-0700: 51372.103: [GC pause (G1 Evacuation Pause) (young), 2.5214157
secs]
2017-03-03T01:59:26.149-0700: 51379.992: [GC pause (G1 Evacuation Pause) (young), 2.0152994
secs]
2017-03-03T01:59:33.786-0700: 51387.629: [GC pause (G1 Evacuation Pause) (young), 2.0538482
secs]
2017-03-03T01:59:41.507-0700: 51395.350: [GC pause (G1 Evacuation Pause) (young), 1.4754542
secs]
2017-03-03T01:59:49.089-0700: 51402.932: [GC pause (G1 Evacuation Pause) (young), 1.8150508
secs]
2017-03-03T01:59:56.538-0700: 51410.380: [GC pause (G1 Evacuation Pause) (young), 1.8172462
secs]
2017-03-03T02:00:04.261-0700: 51418.103: [GC pause (G1 Evacuation Pause) (young), 2.2711223
secs]
2017-03-03T02:00:12.197-0700: 51426.040: [GC pause (G1 Evacuation Pause) (young), 2.5642459
secs]
2017-03-03T02:00:23.308-0700: 51437.151: [GC pause (G1 Evacuation Pause) (young), 2.4554458
secs]
2017-03-03T02:00:34.748-0700: 51448.591: [GC pause (G1 Evacuation Pause) (young), 2.2141849
secs]
2017-03-03T02:00:45.771-0700: 51459.614: [GC pause (G1 Evacuation Pause) (young), 2.7949601
secs]
2017-03-03T02:00:54.892-0700: 51468.735: [GC pause (G1 Evacuation Pause) (young), 3.3745082
secs]
2017-03-03T02:01:04.026-0700: 51477.868: [GC pause (G1 Evacuation Pause) (young), 2.9395345
secs]
2017-03-03T02:01:12.708-0700: 51486.550: [GC pause (G1 Evacuation Pause) (young), 3.1936103
secs]
2017-03-03T02:01:21.180-0700: 51495.023: [GC pause (G1 Evacuation Pause) (young), 2.6952087
secs]
2017-03-03T02:01:29.839-0700: 51503.682: [GC pause (G1 Evacuation Pause) (young), 2.5272125
secs]
2017-03-03T02:01:37.805-0700: 51511.648: [GC pause (G1 Evacuation Pause) (young), 2.5906181
secs]
2017-03-03T02:01:46.330-0700: 51520.172: [GC pause (G1 Evacuation Pause) (young), 2.8425709
secs]
2017-03-03T02:01:54.931-0700: 51528.773: [GC pause (G1 Evacuation Pause) (young), 3.1661278
secs]
2017-03-03T02:02:04.177-0700: 51538.020: [GC pause (G1 Evacuation Pause) (young), 3.7464941
secs]
2017-03-03T02:02:13.918-0700: 51547.760: [GC pause (G1 Evacuation Pause) (young), 4.3365882
secs]
{code}

{code}
#grep getApplications * -B15 -A10
jstack.2017-03-03T01:54:01-	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:135)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T01:54:01-
jstack.2017-03-03T01:54:01-"IPC Server handler 8 on 8032" #261 daemon prio=5 os_prio=0 tid=0x00007f31845a3000
nid=0x586c runnable [0x00007f245afa2000]
jstack.2017-03-03T01:54:01-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T01:54:01-	at java.util.regex.Pattern.compile(Pattern.java:1692)
jstack.2017-03-03T01:54:01-	at java.util.regex.Pattern.<init>(Pattern.java:1351)
jstack.2017-03-03T01:54:01-	at java.util.regex.Pattern.compile(Pattern.java:1054)
jstack.2017-03-03T01:54:01-	at java.lang.String.replace(String.java:2239)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1559)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T01:54:01:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T01:54:01:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T01:54:01:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T01:54:01-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T01:54:01-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T01:54:01-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T01:54:01-
--
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T01:55:02-
jstack.2017-03-03T01:55:02-"IPC Server handler 26 on 8032" #279 daemon prio=5 os_prio=0 tid=0x00007f3184315000
nid=0x587e runnable [0x00007f2459e91000]
jstack.2017-03-03T01:55:02-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T01:55:02-	at java.util.regex.Pattern$BnM.optimize(Pattern.java:5408)
jstack.2017-03-03T01:55:02-	at java.util.regex.Pattern.compile(Pattern.java:1709)
jstack.2017-03-03T01:55:02-	at java.util.regex.Pattern.<init>(Pattern.java:1351)
jstack.2017-03-03T01:55:02-	at java.util.regex.Pattern.compile(Pattern.java:1054)
jstack.2017-03-03T01:55:02-	at java.lang.String.replace(String.java:2239)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1469)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T01:55:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T01:55:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T01:55:02:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T01:55:02-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T01:55:02-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T01:55:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T01:55:02-
--
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T01:56:02-
jstack.2017-03-03T01:56:02-"IPC Server handler 0 on 8032" #252 daemon prio=5 os_prio=0 tid=0x00007f318431d800
nid=0x585f runnable [0x00007f245b8ab000]
jstack.2017-03-03T01:56:02-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T01:56:02-	at java.util.regex.Pattern$BnM.optimize(Pattern.java:5442)
jstack.2017-03-03T01:56:02-	at java.util.regex.Pattern.compile(Pattern.java:1709)
jstack.2017-03-03T01:56:02-	at java.util.regex.Pattern.<init>(Pattern.java:1351)
jstack.2017-03-03T01:56:02-	at java.util.regex.Pattern.compile(Pattern.java:1054)
jstack.2017-03-03T01:56:02-	at java.lang.String.replace(String.java:2239)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1471)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T01:56:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T01:56:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T01:56:02:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T01:56:02-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T01:56:02-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T01:56:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T01:56:02-
--
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:135)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T01:58:01-
jstack.2017-03-03T01:58:01-"IPC Server handler 9 on 8032" #262 daemon prio=5 os_prio=0 tid=0x00007f31845a4800
nid=0x586d runnable [0x00007f245aea1000]
jstack.2017-03-03T01:58:01-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T01:58:01-	at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)
jstack.2017-03-03T01:58:01-	at java.lang.StringBuffer.<init>(StringBuffer.java:116)
jstack.2017-03-03T01:58:01-	at java.util.regex.Matcher.replaceAll(Matcher.java:953)
jstack.2017-03-03T01:58:01-	at java.lang.String.replace(String.java:2240)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1473)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T01:58:01:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T01:58:01:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T01:58:01:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T01:58:01-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T01:58:01-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T01:58:01-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T01:58:01-
--
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T02:00:01-
jstack.2017-03-03T02:00:01-"IPC Server handler 24 on 8032" #277 daemon prio=5 os_prio=0 tid=0x00007f3184b7b800
nid=0x587c runnable [0x00007f245a093000]
jstack.2017-03-03T02:00:01-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T02:00:01-	at java.util.regex.Pattern$BnM.match(Pattern.java:5464)
jstack.2017-03-03T02:00:01-	at java.util.regex.Matcher.search(Matcher.java:1248)
jstack.2017-03-03T02:00:01-	at java.util.regex.Matcher.find(Matcher.java:637)
jstack.2017-03-03T02:00:01-	at java.util.regex.Matcher.replaceAll(Matcher.java:956)
jstack.2017-03-03T02:00:01-	at java.lang.String.replace(String.java:2240)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1473)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T02:00:01:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T02:00:01:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T02:00:01:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T02:00:01-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T02:00:01-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T02:00:01-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T02:00:01-
--
jstack.2017-03-03T02:02:02-	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread.run(ApplicationMasterLauncher.java:119)
jstack.2017-03-03T02:02:02-
jstack.2017-03-03T02:02:02-"IPC Server handler 49 on 8032" #302 daemon prio=5 os_prio=0 tid=0x00007f3184bb6000
nid=0x5895 runnable [0x00007f245877a000]
jstack.2017-03-03T02:02:02-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T02:02:02-	at java.util.regex.Pattern.compile(Pattern.java:1686)
jstack.2017-03-03T02:02:02-	at java.util.regex.Pattern.<init>(Pattern.java:1351)
jstack.2017-03-03T02:02:02-	at java.util.regex.Pattern.compile(Pattern.java:1054)
jstack.2017-03-03T02:02:02-	at java.lang.String.replace(String.java:2239)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1471)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T02:02:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T02:02:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T02:02:02:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T02:02:02-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T02:02:02-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T02:02:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T02:02:02-
--
jstack.2017-03-03T02:03:04-   java.lang.Thread.State: TIMED_WAITING (parking)
jstack.2017-03-03T02:03:04-	at sun.misc.Unsafe.park(Native Method)
jstack.2017-03-03T02:03:04-	- parking to wait for  <0x00007f24fc076b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
jstack.2017-03-03T02:03:04-	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
jstack.2017-03-03T02:03:04-	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
jstack.2017-03-03T02:03:04-	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:135)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T02:03:04-
jstack.2017-03-03T02:03:04-"IPC Server handler 10 on 8032" #263 daemon prio=5 os_prio=0 tid=0x00007f31845a6000
nid=0x586e waiting for monitor entry [0x00007f245ada0000]
jstack.2017-03-03T02:03:04-   java.lang.Thread.State: BLOCKED (on object monitor)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAccess(CapacityScheduler.java:1636)
jstack.2017-03-03T02:03:04-	- waiting to lock <0x00007f24fc5c5220> (a org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.checkAccess(QueueACLsManager.java:49)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkAccess(ClientRMService.java:282)
jstack.2017-03-03T02:03:04:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:759)
jstack.2017-03-03T02:03:04:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T02:03:04:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T02:03:04-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T02:03:04-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T02:03:04-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T02:03:04-
--
jstack.2017-03-03T02:06:02-	at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:728)
jstack.2017-03-03T02:06:02-	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
jstack.2017-03-03T02:06:02-
jstack.2017-03-03T02:06:02-"52032475@qtp-1265053098-825" #45152 daemon prio=5 os_prio=0 tid=0x00007f24159fd000
nid=0x2dc4 runnable [0x00007f244e1e3000]
jstack.2017-03-03T02:06:02-   java.lang.Thread.State: RUNNABLE
jstack.2017-03-03T02:06:02-	at java.util.regex.Pattern.compile(Pattern.java:1709)
jstack.2017-03-03T02:06:02-	at java.util.regex.Pattern.<init>(Pattern.java:1351)
jstack.2017-03-03T02:06:02-	at java.util.regex.Pattern.compile(Pattern.java:1054)
jstack.2017-03-03T02:06:02-	at java.lang.String.replace(String.java:2239)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertFromProtoFormat(ProtoUtils.java:270)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.convertFromProtoFormat(LogAggregationReportPBImpl.java:158)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.LogAggregationReportPBImpl.getLogAggregationStatus(LogAggregationReportPBImpl.java:142)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationReportsForApp(RMAppImpl.java:1473)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getLogAggregationStatusForAppReport(RMAppImpl.java:1549)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:631)
jstack.2017-03-03T02:06:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:814)
jstack.2017-03-03T02:06:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:89)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:86)
jstack.2017-03-03T02:06:02-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T02:06:02-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.webapp.AppsBlock.fetchData(AppsBlock.java:84)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:101)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
--
jstack.2017-03-03T02:06:02-   java.lang.Thread.State: TIMED_WAITING (parking)
jstack.2017-03-03T02:06:02-	at sun.misc.Unsafe.park(Native Method)
jstack.2017-03-03T02:06:02-	- parking to wait for  <0x00007f24fc076b60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
jstack.2017-03-03T02:06:02-	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
jstack.2017-03-03T02:06:02-	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
jstack.2017-03-03T02:06:02-	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:135)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2172)
jstack.2017-03-03T02:06:02-
jstack.2017-03-03T02:06:02-"IPC Server handler 29 on 8032" #282 daemon prio=5 os_prio=0 tid=0x00007f318458c800
nid=0x5881 waiting for monitor entry [0x00007f2459b8e000]
jstack.2017-03-03T02:06:02-   java.lang.Thread.State: BLOCKED (on object monitor)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAccess(CapacityScheduler.java:1636)
jstack.2017-03-03T02:06:02-	- waiting to lock <0x00007f24fc5c5220> (a org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.checkAccess(QueueACLsManager.java:49)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkAccess(ClientRMService.java:282)
jstack.2017-03-03T02:06:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:759)
jstack.2017-03-03T02:06:02:	at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:681)
jstack.2017-03-03T02:06:02:	at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
jstack.2017-03-03T02:06:02-	at java.security.AccessController.doPrivileged(Native Method)
jstack.2017-03-03T02:06:02-	at javax.security.auth.Subject.doAs(Subject.java:422)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
jstack.2017-03-03T02:06:02-	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
jstack.2017-03-03T02:06:02-
--
{code}

> Add option to set max limit on ResourceManager for ApplicationClientProtocol.getApplications
> --------------------------------------------------------------------------------------------
>
>                 Key: YARN-6285
>                 URL: https://issues.apache.org/jira/browse/YARN-6285
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: yunjiong zhao
>            Assignee: yunjiong zhao
>         Attachments: YARN-6285.001.patch, YARN-6285.002.patch, YARN-6285.003.patch
>
>
> When users called ApplicationClientProtocol.getApplications, it will return lots of data,
and generate lots of garbage on ResourceManager which caused long time GC.
> For example, on one of our RM, when called rest API " http://<rm http address:port>/ws/v1/cluster/apps"
it can return 150MB data which have 944 applications.
> getApplications have limit parameter, but some user might not set it, and then the limit
will be Long.MAX_VALUE.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message