hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Himanshu Vashishtha (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-10264) [MapReduce]: CompactionTool in mapred mode is missing classes in its classpath
Date Thu, 02 Jan 2014 19:43:50 GMT

     [ https://issues.apache.org/jira/browse/HBASE-10264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Himanshu Vashishtha updated HBASE-10264:

    Attachment: HBase-10264.patch

Two issues in the mapred phase: 
a) It misses hbase specific jars, so TT doesn't get any hbase-xxx jars shipped to them.
b) The job also requires Counter.class from high_scalability-xxx library, as TT instantiates
HRegion object.

Here is a patch to do so. 

Testing: Ran the patched job on a yarn cluster.
[jenkins@tarball-target-2 hbase]$ bin/hbase org.apache.hadoop.hbase.regionserver.CompactionTool
-mapred -major  hdfs://`hostname`:8020/hbase/data/default/sampleTable_api-compat-8.ent.cloudera.com
2014-01-02 11:33:02,550 INFO  [main] mapreduce.Job: The url to track the job: http://tarball-target-2.ent.cloudera.com:8088/proxy/application_1388690541295_0011/
2014-01-02 11:33:02,551 INFO  [main] mapreduce.Job: Running job: job_1388690541295_0011
2014-01-02 11:33:14,018 INFO  [main] mapreduce.Job: Job job_1388690541295_0011 running in
uber mode : false
2014-01-02 11:33:14,020 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2014-01-02 11:33:23,151 INFO  [main] mapreduce.Job:  map 100% reduce 0%
2014-01-02 11:33:23,172 INFO  [main] mapreduce.Job: Job job_1388690541295_0011 completed successfully
2014-01-02 11:33:23,362 INFO  [main] mapreduce.Job: Counters: 27
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=109926
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=5749
		HDFS: Number of bytes written=968
		HDFS: Number of read operations=22
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=6
	Job Counters 
		Launched map tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=7181
		Total time spent by all reduces in occupied slots (ms)=0
	Map-Reduce Framework
		Map input records=1
		Map output records=0
		Input split bytes=154
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=77
		CPU time spent (ms)=1940
		Physical memory (bytes) snapshot=178450432
		Virtual memory (bytes) snapshot=883052544
		Total committed heap usage (bytes)=114360320
	File Input Format Counters 
		Bytes Read=143
	File Output Format Counters 
		Bytes Written=0


> [MapReduce]: CompactionTool in mapred mode is missing classes in its classpath
> ------------------------------------------------------------------------------
>                 Key: HBASE-10264
>                 URL: https://issues.apache.org/jira/browse/HBASE-10264
>             Project: HBase
>          Issue Type: Bug
>          Components: Compaction, mapreduce
>    Affects Versions: 0.98.0, 0.99.0
>            Reporter: Aleksandr Shulman
>            Assignee: Himanshu Vashishtha
>         Attachments: HBase-10264.patch
> Calling o.a.h.h.regionserver.CompactionTool fails due to classpath-related issues in
both MRv1 and MRv2.
> {code}hbase org.apache.hadoop.hbase.regionserver.CompactionTool -mapred -major hdfs://`hostname`:8020/hbase/data/default/orig_1388179858868{code}
> Results:
> {code}2013-12-27 13:31:49,478 INFO  [main] mapreduce.Job: Task Id : attempt_1388179525649_0011_m_000000_2,
Status : FAILED
> Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.TableInfoMissingException
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionWorker.compact(CompactionTool.java:115)
> 	at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionMapper.map(CompactionTool.java:231)
> 	at org.apache.hadoop.hbase.regionserver.CompactionTool$CompactionMapper.map(CompactionTool.java:207)
> 	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> 	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:165)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:160){code}

This message was sent by Atlassian JIRA

View raw message