hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5548) Use MockRMMemoryStateStore to reduce test failures
Date Thu, 24 Nov 2016 18:16:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15693926#comment-15693926
] 

Varun Saxena edited comment on YARN-5548 at 11/24/16 6:16 PM:
--------------------------------------------------------------

bq. Few test that are using to control specific events stored in RMState stores.
This is somewhat surprising. Why are we even using state store for non RM restart scenarios.
Yes there will be some test cases where we are testing state store itself where change wont
be required. But other than these 2 cases, passing state store to MockRM may not be required.
Will have to be looked at test by test.
This prompted me to have a cursory glance at the patch. 
For instance, TestCapacityScheduler has some, if not all tests which do not even require a
state store to be passed to MockRM. We can remove such references where state store is not
required. Few of these tests which looked like not needing a state store, I tried and they
succeed even if I do not pass the state store to MockRM.

I think in general though wherever state store is required in a test which uses MockRM, we
can use MockRMMemoryStateStore as MockRMMemoryStateStore bypasses dispatcher, it may be better
for most tests than putting events to dispatcher and letting other thread process it. May
theoretically reduce test time a bit. Except ofcourse in tests where multiple threads are
trying to access the state store. Thoughts ?


was (Author: varun_saxena):
bq. Few test that are using to control specific events stored in RMState stores.
This is somewhat surprising. Why are we even using state store for non RM restart scenarios.
Yes there will be some test cases where we are testing state store itself where change wont
be required. But other than that it may not be required. Will have to be looked at test by
test.
This prompted me to have a cursory glance at the patch. 
For instance, TestCapacityScheduler has some, if not all tests which do not even require a
state store to be passed to MockRM. We can remove such references where state store is not
required. Few of these tests which looked like not needing a state store, I tried and they
succeed even if I do not pass the state store to MockRM.

I think in general though wherever state store is required in a test which uses MockRM, we
can use MockRMMemoryStateStore as MockRMMemoryStateStore bypasses dispatcher, it may be better
for most tests than putting events to dispatcher and letting other thread process it. May
theoretically reduce test time a bit. Except ofcourse in tests where multiple threads are
trying to access the state store. Thoughts ?

> Use MockRMMemoryStateStore to reduce test failures
> --------------------------------------------------
>
>                 Key: YARN-5548
>                 URL: https://issues.apache.org/jira/browse/YARN-5548
>             Project: Hadoop YARN
>          Issue Type: Test
>            Reporter: Bibin A Chundatt
>            Assignee: Bibin A Chundatt
>              Labels: oct16-easy, test
>         Attachments: YARN-5548.0001.patch, YARN-5548.0002.patch, YARN-5548.0003.patch,
YARN-5548.0004.patch, YARN-5548.0005.patch, YARN-5548.0006.patch, YARN-5548.0007.patch
>
>
> https://builds.apache.org/job/PreCommit-YARN-Build/12850/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestRMRestart/testFinishedAppRemovalAfterRMRestart/
> {noformat}
> Error Message
> Stacktrace
> java.lang.AssertionError: expected null, but was:<submit_time: 1471885197416 application_submission_context
{ application_id { id: 1 cluster_timestamp: 1471885197388 } application_name: "" queue: "default"
priority { priority: 0 } am_container_spec { } cancel_tokens_when_complete: true maxAppAttempts:
2 resource { memory: 1024 virtual_cores: 1 } applicationType: "YARN" keep_containers_across_application_attempts:
false attempt_failures_validity_interval: 0 am_container_resource_request { priority { priority:
0 } resource_name: "*" capability { memory: 1024 virtual_cores: 1 } num_containers: 0 relax_locality:
true node_label_expression: "" execution_type_request { execution_type: GUARANTEED enforce_execution_type:
false } } } user: "jenkins" start_time: 1471885197417 application_state: RMAPP_FINISHED finish_time:
1471885197478>
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.failNotNull(Assert.java:664)
> 	at org.junit.Assert.assertNull(Assert.java:646)
> 	at org.junit.Assert.assertNull(Assert.java:656)
> 	at org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testFinishedAppRemovalAfterRMRestart(TestRMRestart.java:1656)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message