hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 茅旭峰 <m9s...@gmail.com>
Subject Re: I got errors from hdfs about DataStreamer Exceptions.
Date Sat, 16 Apr 2011 15:12:31 GMT
I double checked the cluster, all of the disk have plenty of free space.
I could not put any data into the cluster. The cluster summary shows

==
Cluster Summary***1987209 files and directories, 1993327 blocks = 3980536
total. Heap Size is 4.2 GB / 5.56 GB (75%)
*
Configured Capacity : 120.88 TB DFS Used : 49.55 TB Non DFS Used :
6.12 TBDFS Remaining:65.21 TBDFS Used%:40.99 %DFS Remaining%:53.95 %
Live
Nodes <http://cdn-hadoop-01:50070/dfsnodelist.jsp?whatNodes=LIVE> : 14 Dead
Nodes <http://cdn-hadoop-01:50070/dfsnodelist.jsp?whatNodes=DEAD> : 0
==

and hadoop fsck / shows good
==
Status: HEALTHY
 Total size:    17900052452875 B
 Total dirs:    35613
 Total files:   1951594 (Files currently being written: 1)
 Total blocks (validated):      1992835 (avg. block size 8982204 B)
 Minimally replicated blocks:   1992835 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       5 (2.5089885E-4 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0000422
 Corrupt blocks:                0
 Missing replicas:              21 (3.5125346E-4 %)
 Number of data-nodes:          14
 Number of racks:               1


The filesystem under path '/' is HEALTHY

==

Any suggestions? Thanks!

On Sat, Apr 16, 2011 at 9:48 PM, Sonal Goyal <sonalgoyal4@gmail.com> wrote:

> Hi,
>
> What is the cluster free disk space and current usage?
>
> Thanks and Regards,
> Sonal
> <https://github.com/sonalgoyal/hiho>Hadoop ETL and Data Integration<https://github.com/sonalgoyal/hiho>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
>
> On Sat, Apr 16, 2011 at 9:18 AM, 茅旭峰 <m9suns@gmail.com> wrote:
>
>> Hi Jun Young,
>>
>> It looks like I'm also running into the same issue you've got.
>> Have you found more info? I googled
>> 'java.io.IOException: Unable to create new block.', some guys
>> suggested that it might be running out of fd, but it's not my case.
>> I verified the fds in use by lsof and ls /proc/pid/fd.
>> Do you have any hints?
>>
>> Thanks!
>>
>> Mao Xu-Feng
>>
>> On Fri, Feb 18, 2011 at 10:51 AM, Jun Young Kim <juneng603@gmail.com>
>> wrote:
>>
>> > hi, harsh.
>> > you're always giving a response very quickly. ;)
>> >
>> > I am using a version 0.21.0 now.
>> > before asking about this problem, I've checked already file system
>> healthy.
>> >
>> > $> hadoop fsck /
>> > .....
>> > .....
>> > Status: HEALTHY
>> >  Total size:    24231595038 B
>> >  Total dirs:    43818
>> >  Total files:   41193 (Files currently being written: 2178)
>> >  Total blocks (validated):      40941 (avg. block size 591866 B) (Total
>> > open file blocks (not validated): 224)
>> >  Minimally replicated blocks:   40941 (100.0 %)
>> >  Over-replicated blocks:        1 (0.0024425392 %)
>> >  Under-replicated blocks:       2 (0.0048850784 %)
>> >  Mis-replicated blocks:         0 (0.0 %)
>> >  Default replication factor:    2
>> >  Average block replication:     2.1106226
>> >  Corrupt blocks:                0
>> >  Missing replicas:              4 (0.00462904 %)
>> >  Number of data-nodes:          8
>> >  Number of racks:               1
>> >
>> > The filesystem under path '/' is HEALTHY
>> >
>> > additionally, I found a little different error. here it is.
>> >
>> > java.io.IOException: Bad connect ack with firstBadLink as
>> > 10.25.241.107:50010 at
>> >
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:889)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:820)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427)
>> >
>> >
>> > here is my execution environment.
>> >
>> > average job count : 20
>> > max map capacity : 128
>> > max reduce capacity : 128
>> > avg/slot per node : 32
>> >
>> > avg input file size per job : 200M ~ 1G
>> >
>> >
>> > thanks.
>> >
>> > Junyoung Kim (juneng603@gmail.com)
>> >
>> >
>> > On 02/18/2011 11:43 AM, Harsh J wrote:
>> >
>> >> You may want to check your HDFS health stat via 'fsck'
>> >> (http://namenode/fsck or `hadoop fsck`). There may be a few corrupt
>> >> files or bad DNs.
>> >>
>> >> Would also be good to know what exact version of Hadoop you're running.
>> >>
>> >> On Fri, Feb 18, 2011 at 7:59 AM, Jun Young Kim<juneng603@gmail.com>
>> >>  wrote:
>> >>
>> >>> hi, all.
>> >>>
>> >>> I got errors from hdfs.
>> >>>
>> >>> 2011-02-18 11:21:29[WARN ][DFSOutputStream.java]run()(519) :
>> DataStreamer
>> >>> Exception: java.io.IOException: Unable to create new block.
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:832)
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427)
>> >>>
>> >>> 2011-02-18 11:21:29[WARN
>> >>> ][DFSOutputStream.java]setupPipelineForAppendOrRecovery()(730) : Could
>> >>> not
>> >>> get block locations. Source file
>> >>> "/user/test/51/output/ehshop00newsvc-r-00000" - Aborting...
>> >>> 2011-02-18 11:21:29[WARN ][Child.java]main()(234) : Exception running
>> >>> child
>> >>> : java.io.EOFException
>> >>>        at java.io.DataInputStream.readShort(DataInputStream.java:298)
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Status.read(DataTransferProtocol.java:113)
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:881)
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:820)
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427)
>> >>>
>> >>> 2011-02-18 11:21:29[INFO ][Task.java]taskCleanup()(996) : Runnning
>> >>> cleanup
>> >>> for the task
>> >>>
>> >>>
>> >>>
>> >>> I think this one is also not different error.
>> >>>
>> >>> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
>> >>> blk_-2325764274016776017_8292 file=/user/test/51/input/kids.txt
>> >>>
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:559)
>> >>>
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:367)
>> >>>
>> >>>        at
>> >>> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:514)
>> >>>
>> >>>        at java.io.DataInputStream.read(DataInputStream.java:83)
>> >>>
>> >>>        at
>> org.apache.hadoop.util.LineReader.readLine(LineReader.java:138)
>> >>>
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:149)
>> >>>
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:465)
>> >>>
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>> >>>
>> >>>        at
>> >>>
>> >>>
>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:90)
>> >>>
>> >>>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>> >>>
>> >>>
>> >>> -->  I've checked the file '/user/test/51/input/kids.txt ', but,
there
>> is
>> >>> not
>> >>> strange ones. this file is healthy.
>> >>>
>> >>> Does anybody know about this error?
>> >>> How could I fix this one?
>> >>>
>> >>> thanks.
>> >>>
>> >>> --
>> >>> Junyoung Kim (juneng603@gmail.com)
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message