hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: cant remove files from tmp
Date Mon, 06 Jun 2011 20:43:27 GMT
Hi Prem,

My guess is that your Linux filesystem on this partition is corrupt. Check
dmesg for output indicating fs-level errors.

-Todd

On Mon, Jun 6, 2011 at 1:23 PM, Jain, Prem <Premanshu.Jain@netapp.com>wrote:

> Mapuser or hdfs user didn't seem to help, so I switched to root:
>
> [root@hadoop20 mapred]# ls -la /part/data
> total 0
> drwx------ 3 hdfs   hadoop 16 Jun  6 10:22 .
> drwxrwxrwx 4 hdfs   hadoop 47 May 26 18:36 ..
> drwxr-xr-x 4 mapred mapred 35 May 26 21:02 tmp
> [root@hadoop20 mapred]#
>
> [root@hadoop20 mapred]# pwd
>
> /part/data/tmp/distcache/642114211252449475_2038269146_799583695/hmaster/user/mapred
> [root@hadoop20 mapred]# ls -la
> total 0
> drwxr-xr-x 3 mapred mapred 22 Jun  6 12:46 .
> drwxr-xr-x 3 mapred mapred 19 May 26 21:17 ..
> ?--------- ? ?      ?       ?            ? input-dir
>
>
>
> -----Original Message-----
> From: Marcos Ortiz [mailto:mlortiz@uci.cu]
> Sent: Monday, June 06, 2011 1:17 PM
> To: hdfs-user@hadoop.apache.org
> Cc: Jain, Prem
> Subject: Re: cant remove files from tmp
>
> * Why are using he root user for these operations?
> * Which are your permisions on your data directory? (ls -la /part/data)?
>
> Regards
>
> El 6/6/2011 3:41 PM, Jain, Prem escribió:
> > I have a wrecked datanode which is giving me hard time restarting. It
> > keeps complaining of Datanode dead, pid file exists.  I already tried
> > deleting the files but seems like the files are corrupted and don't
> > allow me delete.
> >
> > ____________________________________________________________________
> >
> > Here is the log:
> > ____________________________________________________________________
> >
> > /************************************************************
> > STARTUP_MSG: Starting DataNode
> > STARTUP_MSG:   host = hadoop20/192.168.1.190
> > STARTUP_MSG:   args = []
> > STARTUP_MSG:   version = 0.20.2-cdh3u0
> > STARTUP_MSG:   build =  -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
> > compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
> > ************************************************************/
> > 2011-06-06 09:11:01,232 INFO
> > org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
> > already set up for Hadoop, not re-installing.
> > 2011-06-06 09:11:01,369 ERROR
> > org.apache.hadoop.hdfs.server.datanode.DataNode:
> > org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access
> > `/part/data/tmp/distcache/642114211252449475_2038269146_79
> > 9583695/hmaster/user/mapred/input-dir': No such file or directory
> > du: cannot read directory
> > `/part/data/tmp/mapred/jobcache/job_201105261845_0005': Permission
> > denied
> >
> >
> > _________________________________
> > Here is the file I can't delete
> > _________________________________
> > [root@hadoop20 distcache]# pwd
> > /part/data/tmp/distcache
> > [root@hadoop20 distcache]# ls -la
> > total 0
> > drwxr-xr-x 3 mapred mapred 52 May 26 21:36 .
> > drwxr-xr-x 4 mapred mapred 35 May 26 21:02 ..
> > drwxr-xr-x 3 mapred mapred 20 May 26 21:17
> > 642114211252449475_2038269146_799583695
> > [root@hadoop20 distcache]# cd *
> > [root@hadoop20 642114211252449475_2038269146_799583695]# ls -la
> > total 0
> > drwxr-xr-x 3 mapred mapred 20 May 26 21:17 .
> > drwxr-xr-x 3 mapred mapred 52 May 26 21:36 ..
> > drwxr-xr-x 3 mapred mapred 17 May 26 21:17 hmaster
> > [root@hadoop20 642114211252449475_2038269146_799583695]# cd h*
> > [root@hadoop20 hmaster]# ls
> > user
> > [root@hadoop20 hmaster]# cd *
> > [root@hadoop20 user]# ls -la
> > total 0
> > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 .
> > drwxr-xr-x 3 mapred mapred 17 May 26 21:17 ..
> > drwxr-xr-x 3 mapred mapred 22 May 26 21:17 mapred
> > [root@hadoop20 user]# cd m*
> > [root@hadoop20 mapred]# ls -la
> > total 0
> > drwxr-xr-x 3 mapred mapred 22 May 26 21:17 .
> > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 ..
> > ?--------- ? ?      ?       ?            ? input-dir
> > [root@hadoop20 mapred]# rm input-dir
> > rm: cannot lstat `input-dir': No such file or directory
> > [root@hadoop20 mapred]# touch *
> > [root@hadoop20 mapred]# ls
> > input-dir  input-dir
> > [root@hadoop20 mapred]# rm *
> > rm: remove regular empty file `input-dir'? y
> > rm: cannot lstat `input-dir': No such file or directory
> > [root@hadoop20 mapred]# ls -la
> > total 0
> > drwxr-xr-x 3 mapred mapred 22 Jun  6 12:45 .
> > drwxr-xr-x 3 mapred mapred 19 May 26 21:17 ..
> > ?--------- ? ?      ?       ?            ? input-dir
> > [root@hadoop20 mapred]#
> >
>
> --
> Marcos Luís Ortíz Valmaseda
>  Software Engineer (UCI)
>  http://marcosluis2186.posterous.com
>  http://twitter.com/marcosluis2186
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message