hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Uma Maheswara Rao G 72686 <mahesw...@huawei.com>
Subject Re: FileSystem closed
Date Thu, 29 Sep 2011 17:18:19 GMT
FileSystem objects will be cached in jvm.
When it tries to get the FS object by using Filesystem.get(..) ( sequence file internally
will use it), it will return same fs object if scheme and authority is same for the uri.

 fs cache key's equals implementation is below

 static boolean isEqual(Object a, Object b) {
        return a == b || (a != null && a.equals(b));        
      }

      /** {@inheritDoc} */
      public boolean equals(Object obj) {
        if (obj == this) {
          return true;
        }
        if (obj != null && obj instanceof Key) {
          Key that = (Key)obj;
          return isEqual(this.scheme, that.scheme)
                 && isEqual(this.authority, that.authority)
                 && isEqual(this.ugi, that.ugi)
                 && (this.unique == that.unique);
        }
        return false;        
      }


I think, here some your files uri and schems are same and got the same fs object. When it
closes first one, diffenitely other will get this exception.

Regards,
Uma

----- Original Message -----
From: Joey Echeverria <joey@cloudera.com>
Date: Thursday, September 29, 2011 10:34 pm
Subject: Re: FileSystem closed
To: common-user@hadoop.apache.org

> Do you close your FileSystem instances at all? IIRC, the FileSystem
> instance you use is a singleton and if you close it once, it's closed
> for everybody. My guess is you close it in your cleanup method and you
> have JVM reuse turned on.
> 
> -Joey
> 
> On Thu, Sep 29, 2011 at 12:49 PM, Mark question 
> <markq2011@gmail.com> wrote:
> > Hello,
> >
> >  I'm running 100 mappers sequentially on a single machine, where 
> each> mapper opens 100 files at the beginning then read one by one 
> sequentially> and closes after each one is done. After executing 6 
> mappers, the 7th gives
> > this error:
> >
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>   
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> > 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>
   at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>   
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> > 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>
   at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>   
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
> >    at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
> >    at java.security.AccessController.doPrivileged(Native Method)
> >    at javax.security.auth.Subject.doAs(Subject.java:396)
> >    at
> > 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)>
   at org.apache.hadoop.mapred.Child.main(Child.java:211)
> > java.io.IOException: Filesystem closed
> >    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:297)
> >    at 
> org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:426)>    at java.io.FilterInputStream.close(FilterInputStream.java:155)
> >    at
> > 
> org.apache.hadoop.io.SequenceFile$Reader.close(SequenceFile.java:1653)>    at Mapper_Reader20HM4.CleanUp(Mapper_Reader20HM4.java:124)
> >    at BFMapper20HM9.close(BFMapper20HM9.java:264)
> >    at BFMapRunner20HM9.run(BFMapRunner20HM9.java:95)
> >    at 
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:397)>
> >
> > Can anybody give me a hint of what that could be?
> >
> > Thank you,
> > Mark
> >
> 
> 
> 
> -- 
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
> 

Mime
View raw message