Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates
 209.85.221.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=bTJNiLxyQMGjhFXR6OLnux+rIm5UGIfZVp3ePEXvhIVf1q6+N39MCD7pmMlSD0/zrj
         xR1RfH6y/o6+qWebk7gyR+0LvNs2nNuZZLv3/yXvc8vxeNPhUtfDNRmUaW/2f9O36vel
         mgcOsxcXO7164VAD2GYL1weMwzcUXFZ4cxUBM=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <7c962aed0912210835i7c7fe5c9n70daf8f67f9a5883@mail.gmail.com>
References: <7146859a0912172146r2c93a6f3kd1f585d3d4ec9e67@mail.gmail.com>
	 <7c962aed0912181149h19747990he7bba2ae9a69c55c@mail.gmail.com>
	 <7146859a0912201625r791abd38ubfcf02c4f5f522a4@mail.gmail.com>
	 <7146859a0912202326l716f640dp89058f7f6965a87c@mail.gmail.com>
	 <7c962aed0912210835i7c7fe5c9n70daf8f67f9a5883@mail.gmail.com>
Date: Mon, 21 Dec 2009 08:36:54 -0800
Message-ID: <7c962aed0912210836h508ce693i4363fee74ad7c39c@mail.gmail.com>
Subject: Re: Trailer 'header' is wrong; does the trailer size match content?
From: stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636426bef7b3fde047b3fb2c9

--001636426bef7b3fde047b3fb2c9
Content-Type: text/plain; charset=ISO-8859-1

I opened https://issues.apache.org/jira/browse/HBASE-2063 to cover this
issue.
Thanks,
St.Ack

On Mon, Dec 21, 2009 at 8:35 AM, stack <stack@duboce.net> wrote:

> I was going to write that perhaps you needed to
> turn mapred.reduce.tasks.speculative.execution off, but if enabling it and
> things work, that would seem to indicate that a our reducer first takes
> longer than the task timeout maximum and secondly, on failure, we should
> clean up the hfile.
>
> On the first issue, you are using KeyValueSortReducer?  Are your values
> large?  We set reducer status every 100 values.  Maybe this is not enough?
>  We should set status more frequently?  If you call context setstatus more
> frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile
> and then close it.  On kill, this code is not being called.   Let me see if
> can do something about that (e.g. register a shutdown hook to clean away
> incomplete files -- ).
>
> Thanks,
> St.Ack
>
>
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <chingshenchen@gmail.com>wrote:
>
>> I think I found a way.
>> I set the "mapred.reduce.tasks.speculative.execution" to true and output
>> hfiles again, then successfully load hfiles into hbase.
>> Is it best solution? or HFileOutputFormat bug?
>>
>> Shen
>>
>> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <chingshenchen@gmail.com>
>> wrote:
>>
>> > Thanks, stack.
>> >
>> > I checked this file that isn't empty. But I found that as long as the
>> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb
>> script
>> > to load hfiles then failed.
>> > How to avoid this problem?
>> >
>> > Thanks.
>> >
>> > Shen
>> >
>> >
>> > On Sat, Dec 19, 2009 at 3:49 AM, stack <stack@duboce.net> wrote:
>> >
>> >> Check the
>> >> file
>> >>
>> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
>> >>  Is it empty?  Was there an error during running of your MR job?
>>  Perhaps
>> >> a
>> >> task failed?
>> >>
>> >> St.Ack
>> >>
>> >>
>> >>
>> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <chingshenchen@gmail.com>
>> >> wrote:
>> >>
>> >> > Hi,
>> >> >  I use the script loadtable.rb to load my hfiles into hbase, but I
>> got
>> >> an
>> >> > exception as below.
>> >> >  Does anyone have any suggestions?
>> >> >
>> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
>> >> from
>> >> >
>> >> >
>> >>
>> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
>> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
>> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
>> >> match
>> >> > content? (NativeException)
>> >> >    from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
>> `readTrailer'
>> >> >    from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
>> >> `loadFileInfo'
>> >> >    from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
>> >> >    from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>> >> >    from java/lang/reflect/Method.java:597:in `invoke'
>> >> >    from org/jruby/javasupport/JavaMethod.java:298:in
>> >> > `invokeWithExceptionHandling'
>> >> >    from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>> >> >    from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in
>> `call'
>> >> >     ... 18 levels...
>> >> >    from org/jruby/Main.java:94:in `main'
>> >> >    from loadtable.rb:83:in `each'
>> >> >    from loadtable.rb:83
>> >> > Complete Java stackTrace
>> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
>> >> match
>> >> > content?
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
>> >> >    at
>> >> >
>> >>
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
>> >> >    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> >> >    at
>> >> >
>> >> >
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> >    at java.lang.reflect.Method.invoke(Method.java:597)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
>> >> >    at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
>> >> >    at
>> >> >
>> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
>> >> >    at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
>> >> >    at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
>> >> >    at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
>> >> Source)
>> >> >    at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
>> >> >    at org.jruby.runtime.Block.yield(Block.java:100)
>> >> >    at
>> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
>> >> >    at
>> >> >
>> >> >
>> >>
>> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
>> >> >    at loadtable.__file__(loadtable.rb:83)
>> >> >    at loadtable.load(loadtable.rb)
>> >> >    at org.jruby.Ruby.runScript(Ruby.java:577)
>> >> >    at org.jruby.Ruby.runNormally(Ruby.java:480)
>> >> >    at org.jruby.Ruby.runFromMain(Ruby.java:354)
>> >> >    at org.jruby.Main.run(Main.java:229)
>> >> >    at org.jruby.Main.run(Main.java:110)
>> >> >    at org.jruby.Main.main(Main.java:94)
>> >> >
>> >>
>> >
>> >
>> >
>> >
>>
>>
>> --
>> *****************************************************
>> Ching-Shen Chen
>> Advanced Technology Center,
>> Information & Communications Research Lab.
>> E-mail: chenchingshen@itri.org.tw
>> Tel:+886-3-5915542
>> *****************************************************
>>
>
>

--001636426bef7b3fde047b3fb2c9--