hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1296) Bulk upload can overwhelm a datanode filling its disk because HDFS deletes lag
Date Fri, 26 Jun 2009 22:51:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12724754#action_12724754
] 

stack commented on HBASE-1296:
------------------------------

This happened to me about two weeks ago. Namenode was uncontactable -- so it appeared -- and
it was all deleting all the time.

> Bulk upload can overwhelm a datanode filling its disk because HDFS deletes lag
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-1296
>                 URL: https://issues.apache.org/jira/browse/HBASE-1296
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> From a private mail with Jérôme Thièvre:
> "Two times I got the same kind of problems described by schubert zhang in the thread
*HDFS unbalance issue. (HBase over HDFS). *The first time, I found one of my regionserver
 with a full disk, whereas the others were at 10% of their capacity. I saw a lot of move file
command at hadoop level from the full regionserver to others, but after check on the hdfs,
I can see that hadoop copies the file but fails to delete it.  As the regionserver disk was
full, it didn't work anymore, even after hadoop and hbase restart. I had to delete all the
data.
> "The second time this problem occurs I stopped the row insertion process before the disk
was full. After hadoop and hbase restart, hadoop has deleted the files and the system was
operational.
> It seems that in some cases where the system is heavily loaded with continuous writes
and compactions, hadoop can't remove files."

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message