hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Rodionov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10194) FSDataOutputStream.write() allocates new byte buffer on each operation
Date Tue, 22 Mar 2016 22:01:25 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207395#comment-15207395
] 

Vladimir Rodionov commented on HDFS-10194:
------------------------------------------

I will try to explain how this affects HBase.

HBase well-known issue is bad behavior under compaction stress. When HBase Compactor writes
new file it uses, of course DFS (HDFS) write API. On every 1MB written to hdfs, HBase RS JVM
allocates 1MB buffer in Eden space. If HBase writes 100MB sec - its 100MB sec in Eden space.
Young GC get triggered more frequently, which results in false object promotion to tenured
space, which results eventually in long full GC pauses, which, in turn, sometimes results
in RS crash. 

This is for CMS. 

> FSDataOutputStream.write() allocates new byte buffer on each operation
> ----------------------------------------------------------------------
>
>                 Key: HDFS-10194
>                 URL: https://issues.apache.org/jira/browse/HDFS-10194
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.7.1
>            Reporter: Vladimir Rodionov
>
> This is the code:
> {code}
>  private DFSPacket createPacket(int packetSize, int chunksPerPkt, long offsetInBlock,
long seqno, boolean lastPacketInBlock) throws InterruptedIOException {
>      final byte[] buf;
>      final int bufferSize = PacketHeader.PKT_MAX_HEADER_LEN +   packetSize;
>  
>      try {
>        buf = byteArrayManager.newByteArray(bufferSize);
>      } catch (InterruptedException ie) {
>        final InterruptedIOException iioe = new InterruptedIOException(
>            "seqno=" + seqno);
>        iioe.initCause(ie);
>        throw iioe;
>      }
>  
>      return new DFSPacket(buf, chunksPerPkt, offsetInBlock, seqno,
>                           getChecksumSize(), lastPacketInBlock);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message