hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Log rolling failed with ioe
Date Fri, 04 Mar 2011 18:56:30 GMT
Those logs are _really_ hard to follow since the master one has a
bunch of stuff edited out... I see the master was already doing what
appears to be log splitting before the beginning of that log and then
who knows what happened in the master after the region server got the
exception because that part is also missing.

Anyways it looks like you're just suffering from GC pauses, there
should be something explicit about it in the region server log after
the SEVERE exception.

Learn more about GC pauses with this presentation:
http://tiny.cloudera.com/wrflf

J-D

On Thu, Mar 3, 2011 at 7:07 PM, Murali Krishna. P
<muralikpbhat@yahoo.com> wrote:
> Master: http://pastebin.com/kv3Guz71
> RS: http://pastebin.com/9EixG9GB
>
>  Thanks,
> Murali Krishna
>
>
>
>
> ________________________________
> From: Jean-Daniel Cryans <jdcryans@apache.org>
> To: user@hbase.apache.org
> Sent: Fri, 4 March, 2011 12:23:06 AM
> Subject: Re: Log rolling failed with ioe
>
> The mailing list strips the attachements, can you pastebin them or
> host them on a web server?
>
> Thx,
>
> J-D
>
> On Thu, Mar 3, 2011 at 10:49 AM, Murali Krishna. P
> <muralikpbhat@yahoo.com> wrote:
>> No exceptions before that, sending few lines before them in the log file.
>> Also there was a similar retry error in master log also just before this.
>> Attached both master and rs log
>>
>> Thanks,
>> Murali Krishna
>>
>> ________________________________
>> From: Jean-Daniel Cryans <jdcryans@apache.org>
>> To: user@hbase.apache.org
>> Sent: Thu, 3 March, 2011 8:33:48 PM
>> Subject: Re: Log rolling failed with ioe
>>
>> Anything that comes before that? More exceptions? Can you see what's
>> at the beginning of them? I'm more interested in the symptoms rather
>> than the side effects.
>>
>> Thx,
>>
>> J-D
>>
>> On Thu, Mar 3, 2011 at 5:49 AM, Murali Krishna. P
>> <muralikpbhat@yahoo.com> wrote:
>>> Hi,
>>> I get following error frequently (twice in a day atleast) and the region
>>> server
>>> closes and exits after this. I am on 0.20.6 and we are stuck with this for
>>> some
>>> time. Will be upgrading to 0.90.0 later, but right now, would be great if
>>> we can
>>> have a workaround. Let me know if you need more info.
>>>
>>>
>>>  WARNING: DFSClient_1483413998 could not complete file
>>> /user/x/hbase/.logs/host.com,60020,1299145240135/hlog.dat.1299152151
>>> 119.  Retrying...
>>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.LogRoller run
>>> SEVERE: Log rolling failed with ioe:
>>> java.io.IOException: DFSClient_1483413998 could not complete file
>>> /user/x/hbase/.logs/host,60020,1299145240135/hlog.da
>>> t.1299152151119.  Giving up.
>>>        at
>>>
>>>org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3331)
>>>)
>>>
>>>        at
>>>
>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3240)
>>>        at
>>>
>>>org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>>>)
>>>
>>>        at
>>> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>>>        at
>>> org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:959)
>>>        at
>>>
>>> org.apache.hadoop.hbase.regionserver.HLog.cleanupCurrentWriter(HLog.java:491)
>>>        at
>>> org.apache.hadoop.hbase.regionserver.HLog.rollWriter(HLog.java:342)
>>>        at
>>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:80)
>>> Mar 3, 2011 3:53:25 AM org.apache.hadoop.hbase.regionserver.HLog
>>> optionalSync
>>>
>>> Thanks,
>>> Murali Krishna
>>>
>>
>

Mime
View raw message