hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13560) S3ABlockOutputStream to support huge (many GB) file writes
Date Sat, 24 Sep 2016 15:17:20 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15519178#comment-15519178
] 

Steve Loughran commented on HADOOP-13560:
-----------------------------------------

last patch 7129f against s3a ireland *with block output enabled on all S3a tests by default*

 Failure in distcp
{code}
-------------------------------------------------------------------------------Tests run:
4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 110.45 sec <<< FAILURE! - in
org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCplargeFilesFromRemote(org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp)
 Time elapsed: 38.58 sec  <<< FAILURE! java.lang.AssertionError: wrong length DeprecatedRawLocalFileStatus{path=file:/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test/data/ITestS3AContractDistCp/largeFilesFromRemote/outputDir/inputDir/file1;
isDirectory=false; length=1048576; replication=1; blocksize=33554432; modification_time=1474617816000;
access_time=1474617816000; owner=; group=; permission=rw-rw-rw-; isSymlink=false} expected:<11534336>
but was:<1048576>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:555)
        at org.apache.hadoop.fs.contract.ContractTestUtils.verifyFileContents(ContractTestUtils.java:202)
        at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.largeFiles(AbstractContractDistCpTest.java:174)
        at org.apache.hadoop.tools.contract.AbstractContractDistCpTest.largeFilesFromRemote(AbstractContractDistCpTest.java:108)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
        at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
target/failsafe-reports/org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp.txt (END)
{code}

This is interesting; it is not occuring when run standalone. Race condition or consistency
problem surfacing? Or did this patch break upload when block output is enabled everywhere.
I may try to add some checks before the discp D/L to verify the source files are of the expected
files, so guarantee it's not a consistency.

> S3ABlockOutputStream to support huge (many GB) file writes
> ----------------------------------------------------------
>
>                 Key: HADOOP-13560
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13560
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-13560-branch-2-001.patch, HADOOP-13560-branch-2-002.patch,
HADOOP-13560-branch-2-003.patch, HADOOP-13560-branch-2-004.patch
>
>
> An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights that metadata
isn't copied on large copies.
> 1. Add a test to do that large copy/rname and verify that the copy really works
> 2. Verify that metadata makes it over.
> Verifying large file rename is important on its own, as it is needed for very large commit
operations for committers using rename



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message