hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8326) mapreduce.TestTableInputFormatScan times out frequently
Date Sat, 13 Apr 2013 00:40:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630827#comment-13630827
] 

Nick Dimiduk commented on HBASE-8326:
-------------------------------------

Grepping the log files around after a test, it's easy to confirm that JarFinder-generated
jars are not reused between invocations.

{noformat}
$ grep -i 'for class .* using jar' hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan-output.txt
| cut -d ' ' -f 2,8-
17:16:16,381 org.apache.hadoop.hbase.HConstants, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-2782314318252813628.jar
17:16:17,344 org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-2092688855247426374.jar
17:16:17,692 org.apache.hadoop.hbase.client.Put, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-4612029855302264241.jar
17:16:17,714 org.apache.hadoop.hbase.CompatibilityFactory, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-1054386380400806295.jar
17:16:17,714 org.apache.zookeeper.ZooKeeper, using jar /Users/ndimiduk/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar
17:16:17,715 com.google.protobuf.Message, using jar /Users/ndimiduk/.m2/repository/com/google/protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar
17:16:17,715 com.google.common.collect.Lists, using jar /Users/ndimiduk/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar
17:16:17,716 org.cloudera.htrace.Trace, using jar /Users/ndimiduk/.m2/repository/org/cloudera/htrace/htrace/1.50/htrace-1.50.jar
17:16:17,792 org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-1759217497204054953.jar
17:16:17,859 org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-4547156579345911573.jar
17:16:19,680 org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar /Users/ndimiduk/repos/hbase/hbase-server/target/test-dir/hadoop-2278643074673130017.jar
17:16:19,681 org.apache.hadoop.io.LongWritable, using jar /Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
17:16:19,681 org.apache.hadoop.io.Text, using jar /Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
17:16:19,682 org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar /Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
17:16:19,682 org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /Users/ndimiduk/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar
{noformat}

However, notice the timestamps. This iteration generated 7 jars and located 8 others in less
than 4 seconds. I don't think this additional IO is sufficiently high to cause test timeouts.

bq. add additional logic to addDependencyJars such that it will not bundle a new jar if it
has already generated a jar containing the class 

I have two ideas for implementing this. The first is to keep a map of constructed jar content
to jar name. As JarFinder generates jars, it'll add a class listing of those jars to the map,
and future iterations can reference that map before performing jar build. The second is to
use a DynamicClassLoader, to which the generated jars are added. That loader can be inspected
using identical logic to the existing code which searches for jars in the default ClassLoader.

Both implementation add further mess to this code, and all for the sake of unit tests. I'm
not excited about either of them. Let me know if you have any other ideas.
                
> mapreduce.TestTableInputFormatScan times out frequently
> -------------------------------------------------------
>
>                 Key: HBASE-8326
>                 URL: https://issues.apache.org/jira/browse/HBASE-8326
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce, test
>    Affects Versions: 0.98.0, 0.94.7, 0.95.1
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>
> bq. It looks like since this change org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
has not passed a single time in the EC2 builds (http://54.241.6.143/job/HBase-0.94/). Change
was introduced in Build #72, since then this tests times out (or doesn't finish)
> via [~lhofhansl] in [HBASE-8140 comment|https://issues.apache.org/jira/browse/HBASE-8140?focusedCommentId=13629101&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13629101].

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message