hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1189) Still seeing some unexpected 'No space left on device' exceptions
Date Mon, 02 Apr 2007 20:31:32 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Raghu Angadi updated HADOOP-1189:

    Attachment: HADOOP-1189.patch

Attached patch prints a warning and throws the IOException received.

The new log entry looks like this:

2007-04-02 12:59:15,940 WARN org.apache.hadoop.dfs.DataNode: No space left on device while
writing blk_8638782110649810591 (length: 67108864) to /export/crawlspace/rangadi/tmp/ramfs
(Cur available space : 20554389)
2007-04-02 12:59:15,943 ERROR org.apache.hadoop.dfs.DataNode: DataXCeiver java.io.IOException:
No space left on device
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:260)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:837)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:603)
        at java.lang.Thread.run(Thread.java:619)

> Still seeing some unexpected 'No space left on device' exceptions
> -----------------------------------------------------------------
>                 Key: HADOOP-1189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1189
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.12.2
>            Reporter: Raghu Angadi
>         Assigned To: Raghu Angadi
>             Fix For: 0.13.0
>         Attachments: HADOOP-1189.patch
> One of the datanodes has one full partition (disk) out of four. Expected behaviour is
that datanode should skip this partition and use only the other three. HADOOP-990 fixed some
bugs related to this. It seems to work ok but some exceptions are still seeping through. In
one case there 33 of these out 1200+ blocks written to this node. Not sure what caused this.
I will submit a patch to the prints a more useful message throw the original exception.
> Two unlikely reasons I can think of are 2% reserve space (8GB in this case) is not enough
or client some how still says block size is zero in some cases. Better error message should
help here.
> If you see small number of these exceptions compared to number of blocks written, for
now you don't need change anything.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message