hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Kang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6672) BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
Date Fri, 02 Apr 2010 14:26:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852833#action_12852833
] 

Xiao Kang commented on HADOOP-6672:
-----------------------------------

Sun JDK's implementation of writeInt() writeLong() is not the same in class DataOutputStream.
writeInt() wirtes one byte four times whereas wirteLong() writes a eight bytes buffer only
one time. The underlying org.apache.hadoop.mapred.MapTask$MapOutputBuffer.write() spends too
much time to get lock. So it's much efficient to implement  writeInt() as writeLong() to decrease
the invocation counts of MapOutputBuffer.write().

    public final void writeInt(int v) throws IOException {
        out.write((v >>> 24) & 0xFF);
        out.write((v >>> 16) & 0xFF);
        out.write((v >>>  8) & 0xFF);
        out.write((v >>>  0) & 0xFF);
        incCount(4);
    }

    private byte writeBuffer[] = new byte[8];

    public final void writeLong(long v) throws IOException {
        writeBuffer[0] = (byte)(v >>> 56);
        writeBuffer[1] = (byte)(v >>> 48);
        writeBuffer[2] = (byte)(v >>> 40);
        writeBuffer[3] = (byte)(v >>> 32);
        writeBuffer[4] = (byte)(v >>> 24);
        writeBuffer[5] = (byte)(v >>> 16);
        writeBuffer[6] = (byte)(v >>>  8);
        writeBuffer[7] = (byte)(v >>>  0);
        out.write(writeBuffer, 0, 8);
	incCount(8);
    }

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may
be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message