hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Demoor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
Date Wed, 15 Feb 2017 16:55:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15868162#comment-15868162

Thomas Demoor commented on HADOOP-14028:

Not sure yet, looking at the aws sdk code but I'm still confused.

This is the comment for readlimit in {{com.amazonaws.RequestClientOptions}}
     * Used to enable mark-and-reset for
     * non-mark-and-resettable non-file input stream for up to 128K memory
     * buffering by default. Add 1 to get around an implementation quirk of
     * BufferedInputStream.
     * Retries after reading {@link #DEFAULT_STREAM_BUFFER_SIZE} bytes would
     * fail to reset the underlying input stream as the mark position would
     * have been invalidated.
The upload / partupload code gets this from a legacy system property and puts it in awsreq
variable but the awsreq variable doesn't seem to be used anymore.  
Code is here {{com/amazonaws/services/s3/AmazonS3Client.java:1627}} and here 
{{com/amazonaws/services/s3/AmazonS3Client.java:3130}} (sdk 1.11.86)

So don't think it's set for our inputstreams. I guess we currently use the default 128K+1byte?

> S3A block output streams don't delete temporary files in multipart uploads
> --------------------------------------------------------------------------
>                 Key: HADOOP-14028
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14028
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.8.0
>         Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2
>            Reporter: Seth Fitzsimmons
>            Assignee: Steve Loughran
>            Priority: Critical
>         Attachments: HADOOP-14028-branch-2-001.patch, HADOOP-14028-branch-2.8-002.patch,
HADOOP-14028-branch-2.8-003.patch, HADOOP-14028-branch-2.8-004.patch, HADOOP-14028-branch-2.8-005.patch
> I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I was looking
for after running into the same OOM problems) and don't see it cleaning up the disk-cached
> I'm generating a ~50GB file on an instance with ~6GB free when the process starts. My
expectation is that local copies of the blocks would be deleted after those parts finish uploading,
but I'm seeing more than 15 blocks in /tmp (and none of them have been deleted thus far).
> I see that DiskBlock deletes temporary files when closed, but is it closed after individual
blocks have finished uploading or when the entire file has been fully written to the FS (full
upload completed, including all parts)?
> As a temporary workaround to avoid running out of space, I'm listing files, sorting by
atime, and deleting anything older than the first 20: `ls -ut | tail -n +21 | xargs rm`
> Steve Loughran says:
> > They should be deleted as soon as the upload completes; the close() call that the
AWS httpclient makes on the input stream triggers the deletion. Though there aren't tests
for it, as I recall.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message