hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: hadoop 2.0 dependency upgraded to 2.0.3-alpha
Date Mon, 04 Mar 2013 23:56:47 GMT
Here is another blocker for running mapreduce on top hadoop 2.0.3-alpha:

https://issues.apache.org/jira/browse/YARN-449

On Tue, Feb 26, 2013 at 12:03 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> https://issues.apache.org/jira/browse/YARN-429 was logged for this issue.
>
> FYI
>
>
> On Fri, Feb 22, 2013 at 1:12 PM, Andrew Purtell <apurtell@apache.org>wrote:
>
>> I agree, thanks a lot for doing this.
>>
>>
>> On Fri, Feb 22, 2013 at 1:07 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> > bq. to nail down the actual issue will  help making our case.
>> >
>> > That is what I am trying to do. Will ping Arun at the meetup this
>> afternoon
>> > about when the next release is coming.
>> >
>> > Cheers
>> >
>> > On Fri, Feb 22, 2013 at 1:02 PM, Stack <stack@duboce.net> wrote:
>> >
>> > > Thanks for trying to fix it Ted.  I think we need to carry our
>> difficulty
>> > > back up to our parent project brethren (Sounds like you have done this
>> > > off-list; lets do it in jira/mail too).  Your digging in to nail down
>> the
>> > > actual issue will  help making our case.  Maybe we can get/help the
>> > > upstream crew make another release (I don't see a note on a new 2.0
>> > release
>> > > up on hadoop -- have I missed it)?
>> > >
>> > > Thanks,
>> > > St.Ack
>> > >
>> > >
>> > > On Fri, Feb 22, 2013 at 11:49 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> > >
>> > > > Trying out #2, I got:
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> testExcludeAllFromMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat)
>> > > >  Time elapsed: 35.746 sec  <<< FAILURE!
>> > > > java.lang.AssertionError
>> > > >         at org.junit.Assert.fail(Assert.java:86)
>> > > >         at org.junit.Assert.assertTrue(Assert.java:41)
>> > > >         at org.junit.Assert.assertTrue(Assert.java:52)
>> > > >         at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.runIncrementalPELoad(TestHFileOutputFormat.java:468)
>> > > >         at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction(TestHFileOutputFormat.java:713)
>> > > >
>> > > > Arun Murthy got to know this defect. Another 2.0 release is coming
>> soon
>> > > > (next week, hopefully).
>> > > >
>> > > > I plan to wait for the new release.
>> > > >
>> > > > Thanks
>> > > >
>> > > > On Fri, Feb 22, 2013 at 11:39 AM, Ted Yu <yuzhihong@gmail.com>
>> wrote:
>> > > >
>> > > > > The root cause was the removal of capacity-scheduler.xml<
>> > > >
>> > >
>> >
>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml
>> > > >
>> > > > from
>> > > > > 2.0.3-alpha maven artifacts.
>> > > > > This config file is needed to start CapacityScheduler which is
the
>> > > > default.
>> > > > >
>> > > > > After talking to Sid, I got two suggestions:
>> > > > > 1. embed minimal capacity-scheduler.xml<
>> > > >
>> > >
>> >
>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml
>> > > >
>> > > > in
>> > > > > our src/test/resources
>> > > > > 2. use FiFo scheduler instead which doesn't require presence
of
>> > config
>> > > > file
>> > > > >
>> > > > > I think #2, though temporary fix, is easy to change once a future
>> 2.0
>> > > > > release addresses this issue.
>> > > > >
>> > > > > Once consensus is reached, I will attach another patch to
>> HBASE-7904
>> > > > >
>> > > > > Cheers
>> > > > >
>> > > > >
>> > > > > On Fri, Feb 22, 2013 at 10:31 AM, Andrew Purtell <
>> > apurtell@apache.org
>> > > > >wrote:
>> > > > >
>> > > > >> For minicluster unit tests? In my opinion, no.
>> > > > >>
>> > > > >>
>> > > > >> On Fri, Feb 22, 2013 at 10:10 AM, Ted Yu <yuzhihong@gmail.com>
>> > wrote:
>> > > > >>
>> > > > >> > I can try to find the proper usage.
>> > > > >> >
>> > > > >> > The question is: do we have to maintain compatibility
with
>> > > > 2.0.2-alpha ?
>> > > > >> >
>> > > > >> > Thanks
>> > > > >> >
>> > > > >> > On Fri, Feb 22, 2013 at 10:07 AM, Andrew Purtell <
>> > > apurtell@apache.org
>> > > > >> > >wrote:
>> > > > >> >
>> > > > >> > > Sigh. Every release seems to break our uses of
MiniMRCluster
>> all
>> > > > over
>> > > > >> > > again.
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > On Fri, Feb 22, 2013 at 10:04 AM, Ted Yu <
>> yuzhihong@gmail.com>
>> > > > wrote:
>> > > > >> > >
>> > > > >> > > > I temporarily reverted this change due to:
>> > > > >> > > >
>> > > > >> > > > org.apache.hadoop.yarn.YarnException: Failed
to Start
>> > > > >> > > > org.apache.hadoop.mapred.MiniMRCluster
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >>
>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >>
>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >>
>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >>
>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:123)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1574)
>> > > > >> > > > ...
>> > > > >> > > > Caused by: org.apache.hadoop.yarn.YarnException:
>> > > > >> > > > java.lang.IllegalStateException: Queue configuration
>> missing
>> > > child
>> > > > >> > queue
>> > > > >> > > > names for root
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:181)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
>> > > > >> > > >         ... 38 more
>> > > > >> > > > Caused by: java.lang.IllegalStateException:
Queue
>> > configuration
>> > > > >> missing
>> > > > >> > > > child queue names for root
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226)
>> > > > >> > > >         at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:161)
>> > > > >> > > >
>> > > > >> > > > Cheers
>> > > > >> > > >
>> > > > >> > > > On Fri, Feb 22, 2013 at 9:55 AM, Andrew Purtell
<
>> > > > >> apurtell@apache.org>
>> > > > >> > > > wrote:
>> > > > >> > > >
>> > > > >> > > > > Have you tried running the mapreduce
unit tests with this
>> > > > version
>> > > > >> > Ted?
>> > > > >> > > > Just
>> > > > >> > > > > curious. They have given us grief in
the past.
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > > On Fri, Feb 22, 2013 at 9:35 AM, Ted
Yu <
>> > yuzhihong@gmail.com>
>> > > > >> wrote:
>> > > > >> > > > >
>> > > > >> > > > > > Hi,
>> > > > >> > > > > > I have integrated the following
JIRA to trunk:
>> > > > >> > > > > > HBASE-7904 Upgrade hadoop 2.0 dependency
to 2.0.3-alpha
>> > > > >> > > > > >
>> > > > >> > > > > > Earlier this week, Ramkrishna encountered
some problem
>> > where
>> > > > >> HBase
>> > > > >> > > was
>> > > > >> > > > > > built against 2.0.2-alpha but hadoop
cluster deployment
>> > was
>> > > > >> using
>> > > > >> > > > > > 2.0.3-alpha
>> > > > >> > > > > >
>> > > > >> > > > > > In case you use HBase against hadoop
2.0, please note
>> this
>> > > > >> change.
>> > > > >> > > > > >
>> > > > >> > > > > > Cheers
>> > > > >> > > > > >
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > > --
>> > > > >> > > > > Best regards,
>> > > > >> > > > >
>> > > > >> > > > >    - Andy
>> > > > >> > > > >
>> > > > >> > > > > Problems worthy of attack prove their
worth by hitting
>> > back. -
>> > > > >> Piet
>> > > > >> > > Hein
>> > > > >> > > > > (via Tom White)
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Best regards,
>> > > > >> > >
>> > > > >> > >    - Andy
>> > > > >> > >
>> > > > >> > > Problems worthy of attack prove their worth by
hitting back.
>> -
>> > > Piet
>> > > > >> Hein
>> > > > >> > > (via Tom White)
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Best regards,
>> > > > >>
>> > > > >>    - Andy
>> > > > >>
>> > > > >> Problems worthy of attack prove their worth by hitting back.
-
>> Piet
>> > > Hein
>> > > > >> (via Tom White)
>> > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message