hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars George (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1385) Revamp TableInputFormat, needs updating to match hadoop 0.20.x AND remove bit where we can make < maps than regions
Date Tue, 30 Jun 2009 10:39:47 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725571#action_12725571
] 

Lars George commented on HBASE-1385:
------------------------------------

Wow, that was nasty. The TestTableIndex failed with the error that no index directory was
found during the verify step. But that was collateral damage. The real error was buried in
the logs

{code}
009-06-30 12:06:43,935 WARN  [Thread-1100] mapred.LocalJobRunner$Job(255): job_local_0001
java.lang.ClassCastException: org.apache.hadoop.hbase.client.Result cannot be cast to org.apache.hadoop.hbase.mapreduce.LuceneDocumentWrapper
	at org.apache.hadoop.hbase.mapreduce.IndexRecordWriter.write(IndexRecordWriter.java:1)
	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
	at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:154)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:543)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:410)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:215)
2009-06-30 12:06:44,826 DEBUG [main-EventThread] client.HConnectionManager$TableServers(213):
Got ZooKeeper event, state: SyncConnected, type: None, path: null
{code}

Now upon checking I found this in IndexTableReducer

{code}
  public void reduce(ImmutableBytesWritable key, Iterator<Result> values, Context context)
  throws IOException, InterruptedException {
{code}

but when checking the stack trace above you note that Reducer.reduce() is called! That is
because while being abstract, the Reducer can run as-is and now implements a default identity
mapping. That means Result to Result. It is imperative nearly to add "@Override" annotation
because once I did I saw the error, the method should have been

{code}
  @Override
 public void reduce(ImmutableBytesWritable key, Iterable<Result> values, Context context)
  throws IOException, InterruptedException {
{code}

The only difference is the use of Iterable! With the old one it simply created a new reduce()
method that was never called but the inherited default reduce(). Once I changed it to the
proper signature it passed the test.

{code}
> ant -Dtestcase=mapreduce/Test* compile-test test
...
test:
    [junit] Running org.apache.hadoop.hbase.mapreduce.TestTableIndex
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 75.281 sec
    [junit] Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 70.172 sec
    [junit] Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 28.782 sec
{code}

All good now, patch below will have that included.

> Revamp TableInputFormat, needs updating to match hadoop 0.20.x AND remove bit where we
can make < maps than regions
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1385
>                 URL: https://issues.apache.org/jira/browse/HBASE-1385
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.21.0
>
>         Attachments: 1385-v5.patch, 1385-v6.patch, 1385-v7.patch, mr.patch
>
>
> Update TIF to match new MR.
> Remove the bit of logic where we will use number of configured maps as splits count rather
than regions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message