hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10830) FsDatasetImpl#removeVolumes() crashes with IllegalMonitorStateException when vol being removed is in use
Date Sat, 03 Sep 2016 12:08:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460980#comment-15460980
] 

Hadoop QA commented on HDFS-10830:
----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 20s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 29s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 50s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 33s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 27s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  1s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 32s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  6s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 40s{color} |
{color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 14s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 23s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 51s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 51s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  1m 26s{color}
| {color:orange} root: The patch generated 1 new + 114 unchanged - 0 fixed = 115 total (was
114) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 53s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 30s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 28s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 49s{color} | {color:red}
hadoop-common-project_hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was
0) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 29s{color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m  2s{color} | {color:red}
hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 24s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 28s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10830 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12826968/HDFS-10830.02.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 2bb80c25c605 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 07650bc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16629/artifact/patchprocess/diff-checkstyle-root.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-HDFS-Build/16629/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16629/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16629/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16629/testReport/ |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16629/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> FsDatasetImpl#removeVolumes() crashes with IllegalMonitorStateException when vol being
removed is in use
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10830
>                 URL: https://issues.apache.org/jira/browse/HDFS-10830
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.0.0-alpha1
>            Reporter: Manoj Govindassamy
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-10830.01.patch, HDFS-10830.02.patch
>
>
> {{FsDatasetImpl#removeVolumes()}} operation crashes abruptly with IllegalMonitorStateException
whenever the volume being removed is in use concurrently.
> Looks like {{removeVolumes()}} is waiting on a monitor object "this" (that is FsDatasetImpl)
which it has never locked, leading to  IllegalMonitorStateException. This monitor wait happens
only the volume being removed is in use (referencecount > 0). The thread performing this
remove volume operation thus crashes abruptly and block invalidations for the remove volumes
are totally skipped. 
> {code:title=FsDatasetImpl.java|borderStyle=solid}
> @Override
> public void removeVolumes(Set<File> volumesToRemove, boolean clearFailure) {
> ..
> ..
> try (AutoCloseableLock lock = datasetLock.acquire()) {   <== LOCK acquire datasetLock
> for (int idx = 0; idx < dataStorage.getNumStorageDirs(); idx++) {
>   .. .. ..
>   asyncDiskService.removeVolume(sd.getCurrentDir());     <== volume SD1 remove
>   volumes.removeVolume(absRoot, clearFailure);
>   volumes.waitVolumeRemoved(5000, this);                 <== WAIT on "this" ?? But,
we haven't locked it yet.
>                                                              This will cause IllegalMonitorStateException
>                                                              and crash getBlockReports()/FBR
thread!
>   for (String bpid : volumeMap.getBlockPoolList()) {
>     List<ReplicaInfo> blocks = new ArrayList<>();
>     for (Iterator<ReplicaInfo> it = volumeMap.replicas(bpid).iterator();
>          it.hasNext(); ) {
>         .. .. .. 
>         it.remove();                                     <== volumeMap removal
>       }
>     blkToInvalidate.put(bpid, blocks);
>   }
>  .. ..
> }                                                        <== LOCK release datasetLock
  
> // Call this outside the lock.
> for (Map.Entry<String, List<ReplicaInfo>> entry :
> blkToInvalidate.entrySet()) {
>  ..
>  for (ReplicaInfo block : blocks) {
>   invalidate(bpid, block);                               <== Notify NN of Block removal
>  }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message