hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@apache.org
Subject [hadoop] branch trunk updated: HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
Date Tue, 04 Jun 2019 15:48:14 GMT
This is an automated email from the ASF dual-hosted git repository.

todd pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
     new ea3b0a1  HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream
is causing lots of heap allocation in HBase when using short-circut read
ea3b0a1 is described below

commit ea3b0a184405c3feca024a560807ea215b6858b9
Author: huzheng <openinx@gmail.com>
AuthorDate: Tue Jun 4 10:54:46 2019 +0800

    HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing
lots of heap allocation in HBase when using short-circut read
---
 .../java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java    | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java
index 8e592f4..88b1686 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java
@@ -86,6 +86,7 @@ import org.slf4j.LoggerFactory;
 @InterfaceAudience.Private
 public class BlockReaderFactory implements ShortCircuitReplicaCreator {
   static final Logger LOG = LoggerFactory.getLogger(BlockReaderFactory.class);
+  private static final int SMALL_BUFFER_SIZE = 512;
 
   public static class FailureInjector {
     public void injectRequestFileDescriptorsFailure() throws IOException {
@@ -582,7 +583,7 @@ public class BlockReaderFactory implements ShortCircuitReplicaCreator
{
           Slot slot) throws IOException {
     ShortCircuitCache cache = clientContext.getShortCircuitCache();
     final DataOutputStream out =
-        new DataOutputStream(new BufferedOutputStream(peer.getOutputStream()));
+        new DataOutputStream(new BufferedOutputStream(peer.getOutputStream(), SMALL_BUFFER_SIZE));
     SlotId slotId = slot == null ? null : slot.getSlotId();
     new Sender(out).requestShortCircuitFds(block, token, slotId, 1,
         failureInjector.getSupportsReceiptVerification());


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message