Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 48135197DE for ; Sun, 24 Apr 2016 11:12:13 +0000 (UTC) Received: (qmail 49874 invoked by uid 500); 24 Apr 2016 11:12:13 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 49818 invoked by uid 500); 24 Apr 2016 11:12:13 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 49798 invoked by uid 99); 24 Apr 2016 11:12:12 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Apr 2016 11:12:12 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id CC5302C1F54 for ; Sun, 24 Apr 2016 11:12:12 +0000 (UTC) Date: Sun, 24 Apr 2016 11:12:12 +0000 (UTC) From: "Guram Savinov (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-8814) BlockSender.sendChunks() exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-8814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255559#comment-15255559 ] Guram Savinov commented on HDFS-8814: ------------------------------------- I have the same IOException in unit-tests with miniDFS cluster > BlockSender.sendChunks() exception > ---------------------------------- > > Key: HDFS-8814 > URL: https://issues.apache.org/jira/browse/HDFS-8814 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Affects Versions: 2.6.0, 2.7.1 > Environment: OS: CentOS Linux release 7.1.1503 (Core) > Kernel: 3.10.0-229.1.2.el7.x86_64 > Reporter: Marius > > Hi > I was running some streaming jobs with avro files from my hadoop cluster. They performed poorly so i checked the logs of my datanodes and found this: > http://pastebin.com/DXKJJ55z > The cluster is running on CentOS machines: > CentOS Linux release 7.1.1503 (Core) > This is the Kernel: > 3.10.0-229.1.2.el7.x86_64 > No one on the userlist replied and i could not find anything helpful on the internet despite disk failure which is unlikely to cause this because here are several machines and its not very likely that all of their disks fail at the same time. > This error is not reported on the console when running a job and the error occurs from time to time and then dissapears and comes back again. > The block size of the cluster is the default value. > This is my command: > hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. > 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat org.apache.avro.mapred.AvroAsTextInputFormat > Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)