hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11446) S3AOutputStream should use shared thread pool to avoid OutOfMemoryError
Date Sat, 03 Jan 2015 20:01:35 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ted Yu updated HADOOP-11446:
----------------------------
    Release Note: 
The following parameters are introduced in this JIRA:
fs.s3a.threads.max    the maximum number of threads to allow in the pool used by TransferManager
fs.s3a.threads.core    the number of threads to keep in the pool used by TransferManager
fs.s3a.threads.keepalivetime  when the number of threads is greater than the core, this is
the maximum time that excess idle threads will wait for new tasks before terminating
fs.s3a.max.total.tasks    the maximum number of tasks that the LinkedBlockingQueue can hold

> S3AOutputStream should use shared thread pool to avoid OutOfMemoryError
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-11446
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11446
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>         Attachments: hadoop-11446-001.patch, hadoop-11446-002.patch, hadoop-11446-003.patch
>
>
> Here is part of the output including the OOME when hbase snapshot is exported to s3a
(nofile ulimit was increased to 102400):
> {code}
> 2014-12-19 13:15:03,895 INFO  [main] s3a.S3AFileSystem: OutputStream for key 'FastQueryPOC/2014-12-11/EVENT1-IDX-snapshot/.hbase-snapshot/.tmp/EVENT1_IDX_snapshot_2012_12_11/
   650a5678810fbdaa91809668d11ccf09/.regioninfo' closed. Now beginning upload
> 2014-12-19 13:15:03,895 INFO  [main] s3a.S3AFileSystem: Minimum upload part size: 16777216
threshold2147483647
> Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:713)
>         at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
>         at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
>         at com.amazonaws.services.s3.transfer.internal.UploadMonitor.<init>(UploadMonitor.java:129)
>         at com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:449)
>         at com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:382)
>         at org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:127)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:54)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
>         at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:791)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:882)
>         at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:886)
> {code}
> In S3AOutputStream#close():
> {code}
>       TransferManager transfers = new TransferManager(client);
> {code}
> This results in each TransferManager creating its own thread pool, leading to the OOME.
> One solution is to pass shared thread pool to TransferManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message