hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-997) Implement S3 retry mechanism for failed block transfers
Date Mon, 12 Feb 2007 21:55:06 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated HADOOP-997:

    Attachment: HADOOP-997.patch

Here's a patch for review.

It actually has a general retry mechanism, a la HADOOP-601. (There is no annotations support,
but it wouldn't be too hard to extend RetryProxy to have a method that introspects the runtime
method annotations of a passed in object and constructs a proxy based on them.) I've put it
in org.apache.hadoop.io.retry, but I'm not sure about this as it isn't IO specific. The idea
is that this mechanism can be reused for HDFS, MapRed and IPC retries.

On the S3 front, I have increased the block size to 64 MB (since we don't have to worry about
the whole block being buffered in memory any longer), and I've removed the notes about JetS3t
buffering, since this shouldn't need to be changed by users now.

I haven't tried it with any big files yet...

> Implement S3 retry mechanism for failed block transfers
> -------------------------------------------------------
>                 Key: HADOOP-997
>                 URL: https://issues.apache.org/jira/browse/HADOOP-997
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.11.0
>            Reporter: Tom White
>         Assigned To: Tom White
>         Attachments: HADOOP-997.patch
> HADOOP-882 improves S3FileSystem so that when certain communications problems with S3
occur the operation is retried. However, the retry mechanism cannot handle a block transfer
failure, since blocks may be very large and we don't want to buffer them in memory. This improvement
is to write a wrapper (using java.lang.reflect.Proxy if possible - see discussion in HADOOP-882)
that can retry block transfers.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message