hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker
Date Fri, 10 Nov 2017 21:10:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248034#comment-16248034

Steve Loughran commented on HADOOP-15003:

Aarons test failure has forced me to look at all the abort code in the committers. So I have.

* Found bug in the existing S3AFileSystem.listPendingUploads(path), in that as path never
had a trailing "/", it'd list pending uploads in any directory which began with the same prefix,
e.g. /work/job1 and work/job12. The final committer cleanup() logic would therefore abort
adjacent work. Fixed. 
* turned off that bulk cleanup to see what would happen, helped explore that list and delete
pending code, which ultimately proved insufficent (see below). listPendingUploads-> abort
is in fact the only way to do it reliably.
* logging improved @debug level, with a bit more on impoort to deal with the use case "things
aren't working in production, here are the logs, VM is gone"

Two little issues related to cleanup

Magic App attempt abort would list all in app attempt dest/__magic/$jobid_$attemptid/{*.pendingset,
*.pending} and clean up
But this doesn't cleanup on any failed attempt, which will have a different attemptID, so
not be found. 

For the magic committer, switched to a much simpler cleanup of
* list all uploads pending under $dest, abort
* magic: delete __magic/*
* staging: abort wrapped job
* staging: delete local staging dirs for entire job

That is: no list/read/cancel  of pending job & task events.  Expensive *and not complete*.

This bulk abort of all pending uploads of a  dest handles: failed tasks, failed attempts,
failure of incomplete jobs  from other processes/work which was never aborted. (e.g: a previous
query  somewhere failed leaving pending work). 
This was already being done after all the "structured"  aborts; here I remove that work and
only do the bulk list & rm for a  simpler life; made sure all of that excutes in parallel
for fastest cleanup.

This significantly simplifies the abort, commit failure and helper classes (no need to suppress
and reduces the diff between the staging & magic committers (so more has been pulled up
to the base class)

That is: this is the first patch for a while which cuts out production code in any measurable

The staging committer *currently* does the same, but it retains the listing and abort of all
outstanding requests first. Why? I've kept it there to handle the use case of >1 partitioned
commit running simultaneously, where you don't want to abort all the outstanding stuff. That's
not fully rounded off in this patch because it's still doing the bulk cleanup: I want to see
what Ryan thinks is best here. I think I'm going to opt for: list then optional bulk abort.
That means the default doesn't leave pending objects around. That makes the list operation
superfluous, of course, but (a) it's going against HDFS so is less expensive and (b) means
that all tests cover the codepath, rather than having a condition which has weak coverage.
That is: I don't think it's that harmful: there's the cost of an HDFS list(recursive=true),
an open + parse of every read; For the magic committer, in constrast, its more expensive (the
LIST, the GET) and it doesn't support that notion of partitioned work: you can't get >1
job writing to the same directory tree.

h3. Docs: 

* mention examining the _SUCCESS file in troubleshooting section
* defaults of the various options

I'm wrapping up some details on cleanup in the committer arch doc, not included in this patch
as it is late on a friday evening.

h3. Misc

* Invoke adds new method {{ignoreIOExceptions(Log, text, path, lambda-operation)}} which runs
the operation & logs & info if there is a problem. This replaces a lot of the try/catch(IOE
ignored) sequences in cleanup. What it makes easy is to isolate every single operation this
way, so if one fails, the next step runs. This makes cleanup potentially more rigorous.
* S3AUtils {{list(fs, path, recursive, filter)}}, replicates the classic {{FileSystem.list(path,
filter)}} but works on the higher performance recursive listFileStatus operation, and returns
a list of those. Some bits of code taking {{List<FileStatus>}} have had to be changed
to {{List<? extends FileStatus>}} because despite Java type erasure means this is the
same at runtime, it still complains needlessly in the compile.

h3. Testing

* Huge magic commit test is called ITestS3AHugeMagicCommits; guarantees excluded with the
normal rules. This is important as the test_010_ test case, which creates the file, skips
the operation if it runs in parallel...which is going to fail the test with *the exact stack
trace which aaron saw*. That is, I think the test failure really could have been a false alarm.
But it forced me to look at cleanup, which is a good thing.
* New test to create two parallel jobs and work each one simultaneously. Found the listMultipartUploads-prefix

* The mock tests got very confused; the mock AmazonS3Client now has to track outstanding MPU
requests so that listMultipartUploads() can return the full list of active requests, and abortMultipartUpload()
raises 404 if there isn't one outstanding. All the little mock answers are now java-8 lambda
expressions to make a little bit smaller. Reinstated a comment out bit of {{assertValidUpload}}
which checked expected tag list vs. actual; I must have commented it out a while back.

h3. Test run

Test runs: Endpoint S3 ireland with ddb on

One transient failure where my explicitly enabled throttling failed too often for the test;
Cranked back the normal throttle there from 50% to 25%.

Tests run: 16, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 101.795 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.commit.ITestCommitOperations
testCommitEmptyFile(org.apache.hadoop.fs.s3a.commit.ITestCommitOperations)  Time elapsed:
12.755 sec  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: Completing multipart commit on fork-0007/test/DELAY_LISTING_ME/testCommitEmptyFile/empty-commit.txt:
com.amazonaws.AmazonServiceException: throttled count = 3 (Service: null; Status Code: 503;
Error Code: null; Request ID: null):null: throttled count = 3 (Service: null; Status Code:
503; Error Code: null; Request ID: null)
	at org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.maybeFail(InconsistentAmazonS3Client.java:611)
	at org.apache.hadoop.fs.s3a.InconsistentAmazonS3Client.completeMultipartUpload(InconsistentAmazonS3Client.java:537)
	at org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$finalizeMultipartUpload$1(WriteOperationHelper.java:226)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
	at org.apache.hadoop.fs.s3a.WriteOperationHelper.finalizeMultipartUpload(WriteOperationHelper.java:219)
	at org.apache.hadoop.fs.s3a.WriteOperationHelper.completeMPUwithRetries(WriteOperationHelper.java:264)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.innerCommit(CommitOperations.java:166)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commit(CommitOperations.java:138)
	at org.apache.hadoop.fs.s3a.commit.CommitOperations.commitOrFail(CommitOperations.java:121)
	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.commit(ITestCommitOperations.java:325)
	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.commit(ITestCommitOperations.java:302)
	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.createCommitAndVerify(ITestCommitOperations.java:284)
	at org.apache.hadoop.fs.s3a.commit.ITestCommitOperations.testCommitEmptyFile(ITestCommitOperations.java:202)

> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>                 Key: HADOOP-15003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15003
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch, HADOOP-13786-043.patch,
HADOOP-13786-044.patch, HADOOP-13786-045.patch, HADOOP-13786-046.patch, HADOOP-13786-047.patch
> This is a Yetus only JIRA created to have Yetus review the HADOOP-13786/HADOOP-14971
patch as a .patch file, as the review PR [https://github.com/apache/hadoop/pull/282] is stopping
this happening in HADOOP-14971.
> Reviews should go into the PR/other task

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message