hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marius (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8814) BlockSender.sendChunks() exception
Date Thu, 23 Jul 2015 17:19:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639159#comment-14639159
] 

Marius commented on HDFS-8814:
------------------------------

Thanks, i found that this issue appears for longer txt files, too. These are my settings in
hdfs-site.xml:
One the SingleNode:
http://pastebin.com/WNc93zWW

One the Multinode:
http://pastebin.com/faHYKLFv

Note that the address on the multinode was added later in order to check if there are some
port issues, however this was the only port i tested.

Marius

> BlockSender.sendChunks() exception
> ----------------------------------
>
>                 Key: HDFS-8814
>                 URL: https://issues.apache.org/jira/browse/HDFS-8814
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: HDFS
>    Affects Versions: 2.6.0, 2.7.1
>         Environment: OS: CentOS Linux release 7.1.1503 (Core) 
> Kernel: 3.10.0-229.1.2.el7.x86_64
>            Reporter: Marius
>
> Hi
> I was running some streaming jobs with avro files from my hadoop cluster. They performed
poorly so i checked the logs of my datanodes and found this:
> http://pastebin.com/DXKJJ55z
> The cluster is running on CentOS machines:
> CentOS Linux release 7.1.1503 (Core) 
> This is the Kernel:
> 3.10.0-229.1.2.el7.x86_64
> No one on the userlist replied and i could not find anything helpful on the internet
despite disk failure which is unlikely to cause this because here are several machines and
its not very likely that all of their disks fail at the same time.
> This error is not reported on the console when running a job and the error occurs from
time to time and then dissapears and comes back again.
> The block size of the cluster is the default value.
> This is my command:
> hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1.
> 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar
-input /Y/Y1.avro -output /htest/output -mapper mapper.py -reducer reducer.py -inputformat
org.apache.avro.mapred.AvroAsTextInputFormat
> Marius



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message