hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wujinhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15919) AliyunOSS: Enable Yarn to use OSS
Date Sun, 18 Nov 2018 08:16:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690828#comment-16690828
] 

wujinhu commented on HADOOP-15919:
----------------------------------

Thanks [~cheersyang] for your suggestions. I have fixed the issues you have mentioned.

 

---------------------------------------------------------

{color:#f79232}+_TestOSSFileContext.java_+{color}

{color:#f79232}+_This is an empty class._+{color}

Yes, the class extends TestFileContext and will run tests in its parent.

---------------------------------------------------------

{color:#f79232}+_AliyunOSSFileSystemStore_+{color}

{color:#f79232}+_Not sure why you need to remove_+{color}

{color:#f79232}+_statistics.incrementBytesWritten(file.length());_+{color}

Because this patch added tests defined in FCStatisticsBaseTest, and I found this is a minor
bug. So I fixed it and avoid duplicated calculation of bytesWritten metric.
{code:java}
diff --git a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
index 9c4435c11f3..3def6a7bed7 100644
--- a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
+++ b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
@@ -136,7 +136,7 @@ public FSDataOutputStream create(Path path, FsPermission permission,
 key,
 uploadPartSize,
 new SemaphoredDelegatingExecutor(boundedThreadPool,
- blockOutputActiveBlocks, true)), (Statistics)(null));
+ blockOutputActiveBlocks, true)), statistics);
 }

 LOG.debug(result.getETag());
 statistics.incrementWriteOps(1);
- statistics.incrementBytesWritten(file.length());
{code}
Below is the unit test result in my mac env.
{code:java}
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.688 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.027 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.464 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.308 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.214 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations
[WARNING] Tests run: 68, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 74.481 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.018 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 27.65 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.293 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.672 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.827 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.349 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.18 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.013 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.597 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 9.673 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
[INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.472 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen
[INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.858 s - in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS
[INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.462 s - in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.208 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 267.698 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.685 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 267, Failures: 0, Errors: 0, Skipped: 7
{code}

> AliyunOSS: Enable Yarn to use OSS
> ---------------------------------
>
>                 Key: HADOOP-15919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15919
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/oss
>    Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3
>            Reporter: wujinhu
>            Assignee: wujinhu
>            Priority: Major
>         Attachments: HADOOP-15919.001.patch, HADOOP-15919.002.patch, HADOOP-15919.003.patch,
HADOOP-15919.004.patch
>
>
> Uses DelegateToFileSystem to expose AliyunOSSFileSystem as an AbstractFileSystem



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message