hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7934) TableMapReduceUtil doesn't include all dependency jars in new modular build
Date Tue, 26 Feb 2013 19:22:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587420#comment-13587420
] 

Nick Dimiduk commented on HBASE-7934:
-------------------------------------

I agree that we should minimize the weight added to the user's jobs. On the other hand, if
someone wants to use some part of HBase API, I think they shouldn't have to resolve dependencies
manually the first time they run on a real cluster. Why force the user to pretend to be maven
when we can do it for them. Better still would be some kind of automatic dependency resolution,
but...

I think the logic of walking the dependency graph to decide precisely which jars are necessary
is too complex vs the pain suffered by a user for always including everything that might be
needed. As was pointed out, this operation isn't recursive, so it doesn't have the potential
of polluting the user's classpath with incompatible versions of 3rd party jars. Worse case
scenario, they get hbase-foo.jar when they didn't need it. So long as the list of dependencies
require by hbase doesn't stray far from those provided by the hadoop runtime, this should
be painless to the consumer.
                
> TableMapReduceUtil doesn't include all dependency jars in new modular build
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-7934
>                 URL: https://issues.apache.org/jira/browse/HBASE-7934
>             Project: HBase
>          Issue Type: Bug
>          Components: mapreduce
>    Affects Versions: 0.95.0, 0.96.0
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>         Attachments: 0001-HBASE-7934-TableMapReduceUtil-include-module-depende.patch,
0001-HBASE-7934-TableMapReduceUtil-include-module-depende.patch
>
>
> From post to hbase-dev:
> I'm seeing ClassNotFoundExceptions when running ImportTsv against trunk. Presumably the
old instructions for classpath assembly need an update. I guess I could create a hadoop jar
that ships the classpath entries in lib, but IMHO that's not a viable solution for users who
want to run one of the jobs we distribute. Am I missing something?
> For you reference:
> {code}
> $ HADOOP_CLASSPATH=$(./bin/hbase classpath) hadoop jar ./hbase-server/target/hbase-server-0.95-SNAPSHOT.jar
importtsv -Dimporttsv.columns=HBASE_ROW_KEY,d:c1,d:c2 -Dimporttsv.bulk.output=hdfs:///tmp/hfiles
simple hdfs:///tmp/simple.tsv
> ...
> Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Mutate$MutateType
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> 	at org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationSerializer.serialize(MutationSerialization.java:87)
> 	at org.apache.hadoop.hbase.mapreduce.MutationSerialization$MutationSerializer.serialize(MutationSerialization.java:70)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)
> 	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> 	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> 	at org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:151)
> 	at org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.map(TsvImporterMapper.java:37)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message