hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13560) S3ABlockOutputStream to support huge (many GB) file writes
Date Wed, 19 Oct 2016 10:43:59 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-13560:
------------------------------------
    Description: 
There's two output stream mechanisms in Hadooop 2.7.x, neither of which handle massive multi-GB
files that well.

"classic": buffer everything to HDD until to the close() operation; time to close becomes
O(data); as is available disk space. Fails to exploit exploit idle bandwidth, and on EC2 VMs
with not much HDD capacity (especially completing with HDFS storage), can fill up the disk.

{{S3AFastOutputStream}} uploads data in partition-sized blocks, buffering via byte arrays.
Avoids disk problems and as it writes as soon as the first partition is ready, close() time
is O(outstanding-data). However: needs tuning to reduce amount of data buffered. Get it wrong,
and the first clue you get may be that the process goes OOM or is killed by YARN. Which is
a shame, as get it right and operations which generates lots of data, complete much faster,
including distcp.

This patch proposes a new output stream, a successor to both, {{S3ABlockOutputStream}}.

# uses block upload model of S3AFastOutputStream
# supports buffering via: HDD, heap and (recycled) byte buffer, offering a choice between
memory and HDD use. HDD: no OOM problems on small JVMs/need to tune.
# Uses the fast output stream mechanism of limiting queue size for data to upload. Even when
buffering via HDD, you may need to limit that use.
# lots of instrumentation to see what's being written.
# good defaults out the box (e.g buffer to HDD, partition size to strike a good balance of
early upload and scaleability)
# robust against transient failures. The AWS SDK retries a PUT on failure; the entire block
may need to be replayed, so HDD input cannot be buffered via {{java.io.BufferedInputStream}}.
It has also surfaced in testing that if the final commit of a multipart option fails, it isn't
retried —at least in the current SDK in use. Do that ourselves.
# use roundrobin directory allocation, for most effective disk use
# take an AWS SDK {{com.amazonaws.event.ProgressListener}} for progress callbacks, giving
more detail on the operation. (It actually takes a {{org.apache.hadoop.util.Progressable}},
but if that also implements the AWS interface, that is used instead.

All of this to come with scale tests

* generate large files using all buffer mechanisms
* Do a large copy/rname and verify that the copy really works, including metadata
* be configurable with sizes up to muti-GB, which also means that the test timeouts need to
be configurable to match the time it can take.
* As they are slow, make them optional, using the {{-Dscale}} switch to enable.

Verifying large file rename is important on its own, as it is needed for very large commit
operations for committers using rename

The goal here is to implement a single, object stream which can be used for all outputs, tuneable
as to whether to use disk or memory, and on queue sizes, but otherwise be all that's needed.
We can do future development on this, remove its predecessor {{S3AFastOutputStream}}, so keeping
docs and testing down, and leave the original {{S3AOutputStream}} alone for regression testing/fallback.

  was:
An AWS SDK [issue|https://github.com/aws/aws-sdk-java/issues/367] highlights that metadata
isn't copied on large copies.

1. Add a test to do that large copy/rname and verify that the copy really works
2. Verify that metadata makes it over.

Verifying large file rename is important on its own, as it is needed for very large commit
operations for committers using rename


> S3ABlockOutputStream to support huge (many GB) file writes
> ----------------------------------------------------------
>
>                 Key: HADOOP-13560
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13560
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.9.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 2.8.0, 3.0.0-alpha2
>
>         Attachments: HADOOP-13560-branch-2-001.patch, HADOOP-13560-branch-2-002.patch,
HADOOP-13560-branch-2-003.patch, HADOOP-13560-branch-2-004.patch
>
>
> There's two output stream mechanisms in Hadooop 2.7.x, neither of which handle massive
multi-GB files that well.
> "classic": buffer everything to HDD until to the close() operation; time to close becomes
O(data); as is available disk space. Fails to exploit exploit idle bandwidth, and on EC2 VMs
with not much HDD capacity (especially completing with HDFS storage), can fill up the disk.
> {{S3AFastOutputStream}} uploads data in partition-sized blocks, buffering via byte arrays.
Avoids disk problems and as it writes as soon as the first partition is ready, close() time
is O(outstanding-data). However: needs tuning to reduce amount of data buffered. Get it wrong,
and the first clue you get may be that the process goes OOM or is killed by YARN. Which is
a shame, as get it right and operations which generates lots of data, complete much faster,
including distcp.
> This patch proposes a new output stream, a successor to both, {{S3ABlockOutputStream}}.
> # uses block upload model of S3AFastOutputStream
> # supports buffering via: HDD, heap and (recycled) byte buffer, offering a choice between
memory and HDD use. HDD: no OOM problems on small JVMs/need to tune.
> # Uses the fast output stream mechanism of limiting queue size for data to upload. Even
when buffering via HDD, you may need to limit that use.
> # lots of instrumentation to see what's being written.
> # good defaults out the box (e.g buffer to HDD, partition size to strike a good balance
of early upload and scaleability)
> # robust against transient failures. The AWS SDK retries a PUT on failure; the entire
block may need to be replayed, so HDD input cannot be buffered via {{java.io.BufferedInputStream}}.
It has also surfaced in testing that if the final commit of a multipart option fails, it isn't
retried —at least in the current SDK in use. Do that ourselves.
> # use roundrobin directory allocation, for most effective disk use
> # take an AWS SDK {{com.amazonaws.event.ProgressListener}} for progress callbacks, giving
more detail on the operation. (It actually takes a {{org.apache.hadoop.util.Progressable}},
but if that also implements the AWS interface, that is used instead.
> All of this to come with scale tests
> * generate large files using all buffer mechanisms
> * Do a large copy/rname and verify that the copy really works, including metadata
> * be configurable with sizes up to muti-GB, which also means that the test timeouts need
to be configurable to match the time it can take.
> * As they are slow, make them optional, using the {{-Dscale}} switch to enable.
> Verifying large file rename is important on its own, as it is needed for very large commit
operations for committers using rename
> The goal here is to implement a single, object stream which can be used for all outputs,
tuneable as to whether to use disk or memory, and on queue sizes, but otherwise be all that's
needed. We can do future development on this, remove its predecessor {{S3AFastOutputStream}},
so keeping docs and testing down, and leave the original {{S3AOutputStream}} alone for regression
testing/fallback.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message