hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Re: LeaseExpiredException : Lease mismatch in Hadoop mapReduce| How to solve?
Date Fri, 15 Nov 2013 08:48:31 GMT
u r most welcome :)


On Fri, Nov 15, 2013 at 12:46 PM, chandu banavaram <
chandu.banavaram@gmail.com> wrote:

> thanks
>
>
> On Thu, Nov 14, 2013 at 10:18 PM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:
>
>> @chandu banavaram:
>> This exception usually happens if hdfs is trying to write into a file
>> which is no more in hdfs..
>>
>> I think in my case certain files are not created in my hdfs.it failed to
>> create due to some permissions.
>>
>> I am trying out.
>>
>>
>> On Wed, Nov 13, 2013 at 9:25 AM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:
>>
>>> :) Ok
>>> Why u also experienced the same?
>>>
>>>
>>> On Tue, Nov 12, 2013 at 5:14 PM, chandu banavaram <
>>> chandu.banavaram@gmail.com> wrote:
>>>
>>>> plz send the answer to me  for this query
>>>>
>>>>
>>>> On Tue, Nov 12, 2013 at 2:52 AM, unmesha sreeveni <
>>>> unmeshabiju@gmail.com> wrote:
>>>>
>>>>> While running job with 90 Mb file i am getting LeaseExpiredException
>>>>>
>>>>> 13/11/12 15:46:41 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>> parsing the arguments. Applications should implement Tool for the same.
>>>>> 13/11/12 15:46:42 INFO input.FileInputFormat: Total input paths to
>>>>> process : 1
>>>>> 13/11/12 15:46:43 INFO mapred.JobClient: Running job:
>>>>> job_201310301645_25033
>>>>>  13/11/12 15:46:44 INFO mapred.JobClient:  map 0% reduce 0%
>>>>> 13/11/12 15:46:56 INFO mapred.JobClient: Task Id :
>>>>> attempt_201310301645_25033_m_000000_0, Status : FAILED
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
>>>>> Lease mismatch on /user/hdfs/in/map owned by
>>>>> DFSClient_NONMAPREDUCE_-1622335545_1 but is accessed by
>>>>> DFSClient_NONMAPREDUCE_-1561990512_1
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2459)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2437)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2503)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2480)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:556)
>>>>> at
>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:337)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44958)
>>>>>  at org.
>>>>> attempt_201310301645_25033_m_000000_0: SLF4J: Class path contains
>>>>> multiple SLF4J bindings.
>>>>> attempt_201310301645_25033_m_000000_0: SLF4J: Found binding in
>>>>> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>> attempt_201310301645_25033_m_000000_0: SLF4J: Found binding in
>>>>> [jar:file:/tmp/hadoop-mapred/mapred/local/taskTracker/hdfs/jobcache/job_201310301645_25033/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>> attempt_201310301645_25033_m_000000_0: SLF4J: See
>>>>> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>>>>>  13/11/12 15:47:02 INFO mapred.JobClient: Task Id :
>>>>> attempt_201310301645_25033_m_000000_1, Status : FAILED
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
>>>>> Lease mismatch on /user/hdfs/in/map owned by
>>>>> DFSClient_NONMAPREDUCE_-1622335545_1 but is accessed by
>>>>> DFSClient_NONMAPREDUCE_-1662926329_1
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2459)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2262)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2175)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:299)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44954)
>>>>>  at
>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java
>>>>> attempt_201310301645_25033_m_000000_1: SLF4J: Class path contains
>>>>> multiple SLF4J bindings.
>>>>> attempt_201310301645_25033_m_000000_1: SLF4J: Found binding in
>>>>> [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>> attempt_201310301645_25033_m_000000_1: SLF4J: Found binding in
>>>>> [jar:file:/tmp/hadoop-mapred/mapred/local/taskTracker/hdfs/jobcache/job_201310301645_25033/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>>>>> attempt_201310301645_25033_m_000000_1: SLF4J: See
>>>>> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>>>>> attempt_201310301645_25033_m_000000_1: log4j:WARN No appenders could
>>>>> be found for logger (org.apache.hadoop.hdfs.DFSClient).
>>>>> attempt_201310301645_25033_m_000000_1: log4j:WARN Please initialize
>>>>> the log4j system properly.
>>>>> attempt_201310301645_25033_m_000000_1: log4j:WARN See
>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
>>>>> 13/11/12 15:47:10 INFO mapred.JobClient: Task Id :
>>>>> attempt_201310301645_25033_m_000001_0, Status : FAILED
>>>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
>>>>> No lease on /user/hdfs/in/map: File is not open for writing. Holder
>>>>> DFSClient_NONMAPREDUCE_-1622335545_1 does not have any open files.
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2452)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2437)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:2503)
>>>>> at
>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2480)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:556)
>>>>> at
>>>>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:337)
>>>>>  at
>>>>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44958)
>>>>>
>>>>> Why is it so?
>>>>> My mapper code is
>>>>>
>>>>> public void map(Object key, Text value, Context context)
>>>>> throws IOException, InterruptedException {
>>>>>  Configuration conf = new Configuration();
>>>>> FileSystem fs = FileSystem.get(conf);
>>>>>
>>>>>  Path inputfile = new Path("in/map");
>>>>>  BufferedWriter getdatabuffer = new BufferedWriter(new
>>>>> OutputStreamWriter(fs.create(inputfile)));
>>>>>    getdatabuffer.write(value.toString());
>>>>>    getdatabuffer.close();
>>>>> Path Attribute = new Path("in/Attribute");
>>>>>     int row =0;
>>>>>         BufferedReader read = new BufferedReader(new
>>>>> InputStreamReader(fs.open(inputfile)));
>>>>>         String str = null;
>>>>>         while((str = read.readLine())!=null){
>>>>>
>>>>>         row++; //total row count
>>>>>         StringTokenizer st =new StringTokenizer(str," ");
>>>>>         col = st.countTokens();
>>>>>       }
>>>>>         read.close();
>>>>> ...........
>>>>> ...........
>>>>> .............
>>>>> ............
>>>>> Further computation is based on the above "map" file.
>>>>>
>>>>> Why this happens?
>>>>> I think it is unable to write into in/map for several times.
>>>>> How to get rid of this?
>>>>> *Any Suggestions?*
>>>>>
>>>>> --
>>>>> *Thanks & Regards*
>>>>>
>>>>> Unmesha Sreeveni U.B
>>>>>
>>>>> *Junior Developer*
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Thanks & Regards*
>>>
>>> Unmesha Sreeveni U.B
>>>
>>> *Junior Developer*
>>>
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>


-- 
*Thanks & Regards*

Unmesha Sreeveni U.B

*Junior Developer*

Mime
View raw message