Return-Path: 
 <hdfs-issues-return-60754-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F2CBF85F
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Thu, 18 Apr 2013 18:48:15 +0000 (UTC)
Received: (qmail 63143 invoked by uid 500); 18 Apr 2013 18:48:15 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 63093 invoked by uid 500); 18 Apr 2013 18:48:15 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 63027 invoked by uid 99); 18 Apr 2013 18:48:14 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Apr 2013 18:48:14 +0000
Date: Thu, 18 Apr 2013 18:48:14 +0000 (UTC)
From: "Colin Patrick McCabe (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12643159.1366274836351.188044.1366310894863@arcas>
In-Reply-To: <JIRA.12643159.1366274836351@arcas>
References: <JIRA.12643159.1366274836351@arcas>
Subject: [jira] [Commented] (HDFS-4710) Turning off HDFS short-circuit
 checksums unexpectedly slows down Hive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635514#comment-13635514 ] 

Colin Patrick McCabe commented on HDFS-4710:
--------------------------------------------

If we honor {{dfs.client.read.shortcircuit.buffer.size}}, the client can decide whether to use a bounce buffer or not, and if so, how big.
                
> Turning off HDFS short-circuit checksums unexpectedly slows down Hive
> ---------------------------------------------------------------------
>
>                 Key: HDFS-4710
>                 URL: https://issues.apache.org/jira/browse/HDFS-4710
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.0.4-alpha
>         Environment: Centos (EC2) + short-circuit reads on
>            Reporter: Gopal V
>            Priority: Minor
>              Labels: perfomance
>
> When short-circuit reads are on, HDFS client slows down when checksums are turned off.
> With checksums on, the query takes 45.341 seconds and with it turned off, it takes 56.345 seconds. This is slower than the speeds observed when short-circuiting is turned off.
> The issue seems to be that FSDataInputStream.readByte() calls are directly transferred to the disk fd when the checksums are turned off.
> Even though all the columns are integers, the data being read will be read via DataInputStream which does
> {code}
> public final int readInt() throws IOException {
>         int ch1 = in.read();
>         int ch2 = in.read();
>         int ch3 = in.read();
>         int ch4 = in.read();
> {code}
> To confirm, an strace of the Yarn container shows
> {code}
> 26690 read(154, "B", 1)                 = 1
> 26690 read(154, "\250", 1)              = 1
> 26690 read(154, ".", 1)                 = 1
> 26690 read(154, "\24", 1)               = 1
> {code}
> To emulate this without the entirety of Hive code, I have written a simpler test app 
> https://github.com/t3rmin4t0r/shortcircuit-reader
> The jar will read a file in -bs <n> sized buffers. Running it with 1 byte blocks gives similar results to the Hive test run.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira