hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10691) FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException
Date Tue, 26 Jul 2016 10:46:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393615#comment-15393615
] 

Hadoop QA commented on HDFS-10691:
----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 17s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  0s{color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m  0s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 46s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 26s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 52s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 12s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 47s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 54s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 46s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 44s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 44s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 23s{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 16 unchanged
- 1 fixed = 18 total (was 17) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 49s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 10s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 50s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 51s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 57m 23s{color} | {color:green}
hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 20s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 45s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12820130/HDFS-10691.001.patch
|
| JIRA Issue | HDFS-10691 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 93bf7afe5832 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7cac765 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16188/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16188/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16188/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-10691
>                 URL: https://issues.apache.org/jira/browse/HDFS-10691
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-10691.001.patch
>
>
> I use hdfs oiv -p FileDistribution command to do a file analyse. But the {{ArrayIndexOutOfBoundsException}}
happened and lead the process terminated. The stack infos:
> {code}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 103
> 	at org.apache.hadoop.hdfs.tools.offlineImageViewer.FileDistributionCalculator.run(FileDistributionCalculator.java:243)
> 	at org.apache.hadoop.hdfs.tools.offlineImageViewer.FileDistributionCalculator.visit(FileDistributionCalculator.java:176)
> 	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.run(OfflineImageViewerPB.java:176)
> 	at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPB.main(OfflineImageViewerPB.java:129)
> {code}
> I looked into the code and I found the exception was threw in increasing count of {{distribution}}.
And the reason for the exception is that the bucket number was more than the distribution's
length.
> Here are my steps:
> 1).The input command params:
> {code}
> hdfs oiv -p FileDistribution -maxSize 104857600 -step 1024000
> {code}
> The {{numIntervals}} in code should be 104857600/1024000 =102(real value:102.4), so the
{{distribution}}'s length should be {{numIntervals}} + 1 = 103.
> 2).The {{ArrayIndexOutOfBoundsException}} will happens when the file size is in range
(maxSize-steps, maxSize]. For example, if the size of one file is 104800000, and it's in range
of size as mentioned before. And the bucket number is calculated as 104800000/1024000=102.3,
then in code we do the {{Math.ceil}} of this, so the final value should be 103. But the {{distribution}}'s
length is also 103, it means the index is from 0 to 102. So the {{ArrayIndexOutOfBoundsException}}
happens.
> In a word, the exception will happens when {{maxSize}} can not be divided by {{step}}
and meanwhile the size of file is in range (maxSize-step, maxSize]. The related logic should
be changed from 
> {code}
> int bucket = fileSize > maxSize ? distribution.length - 1 : (int) Math
>             .ceil((double)fileSize / steps);
> {code}
> to 
> {code}
>         int bucket =
>             fileSize >= maxSize || (fileSize + steps) > maxSize ? distribution.length
- 1
>                 : (int) Math.ceil((double) fileSize / steps);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message