Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: Levy Meny <Meny.Levy@comverse.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
CC: Yaniv Ofer <Ofer.Yaniv@comverse.com>
Date: Sun, 9 Jun 2013 15:35:50 +0300
Subject: Compression  class loading mismatch in 0.94.2 
Thread-Topic: Compression  class loading mismatch in 0.94.2 
Thread-Index: Ac5lDeAZEY7v/6YVQoqDdss1dUxfGg==
Message-ID: <85C7265F443C1249837736D58834E9E0CD4BD6F1CC@EXTLV.comverse.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_85C7265F443C1249837736D58834E9E0CD4BD6F1CCEXTLVcomverse_"
MIME-Version: 1.0

--_000_85C7265F443C1249837736D58834E9E0CD4BD6F1CCEXTLVcomverse_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,
Anyone knows why org.apache.hadoop.hbase.io.hfile.Compression has changed i=
n 0.94.2 to use the SystemClassLoader to load the snappy class, instead of =
the ContextClassLoader in previous versions (e.g. in 0.92.1)?

      private CompressionCodec buildCodec(Configuration conf) {
        try {
          Class<?> externalCodec =3D ClassLoader.getSystemClassLoader().loa=
dClass("org.apache.hadoop.io.compress.SnappyCodec");
          return (CompressionCodec) ReflectionUtils.newInstance(externalCod=
ec,conf);

(btw you will notice that ContextClassLoader is still used for loading e.g.=
 Lz4Codec)

The error I got:

2013-05-31 00:01:25,704 [ERROR] [BulkImportManager-2-thread-1] org.apache.h=
adoop.hbase.mapreduce.LoadIncrementalHFiles (LoadIncrementalHFiles.java:343=
) - Unexpected execution exception during splitting
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.l=
ang.ClassNotFoundException: org.apache.hadoop.io.compress.SnappyCodec
        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrS=
plitPhase(LoadIncrementalHFiles.java:333)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLo=
ad(LoadIncrementalHFiles.java:232)
...
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: or=
g.apache.hadoop.io.compress.SnappyCodec
        at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$4.buildCo=
dec(Compression.java:207)
        at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$4.getCode=
c(Compression.java:192)
        at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompre=
ssor(Compression.java:302)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileB=
lock.java:745)
        at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileW=
riterV2.java:134)
        at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWrite=
rV2.java:125)
        at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.c=
reateWriter(HFileWriterV2.java:105)
        at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFil=
e.java:394)
        at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.<init>(Sto=
reFile.java:1003)
        at org.apache.hadoop.hbase.regionserver.StoreFile$Writer.<init>(Sto=
reFile.java:948)
        at org.apache.hadoop.hbase.regionserver.StoreFile$WriterBuilder.bui=
ld(StoreFile.java:851)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.copyHFil=
eHalf(LoadIncrementalHFiles.java:541)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitSto=
reFile(LoadIncrementalHFiles.java:514)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitSto=
reFile(LoadIncrementalHFiles.java:375)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrS=
plit(LoadIncrementalHFiles.java:439)
       at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(Lo=
adIncrementalHFiles.java:323)
        at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(L=
oadIncrementalHFiles.java:321)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        ... 3 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.compress.=
SnappyCodec
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$4.buildCo=
dec(Compression.java:201)
        ... 21 more

This change make it impossible for me to bulk upload files into HBase from =
my app when my app runs as a Tomcat web application since Tomcat does not a=
llow setting the System class loader and all HBase and Hadoop jars are in m=
y WEB-INF/lib and are not available to the System class loader..

Thanks,

Meny Levy
R&D

T   +972-3-7663350
M +972-52-8543350
Meny.Levy@comverse.com <mailto:Meny.Levy@comverse.com>
www.comverse.com<http://www.comverse.com/>


________________________________
"This e-mail message may contain confidential, commercial or privileged inf=
ormation that constitutes proprietary information of Comverse Technology or=
 its subsidiaries. If you are not the intended recipient of this message, y=
ou are hereby notified that any review, use or distribution of this informa=
tion is absolutely prohibited and we request that you delete all copies and=
 contact us by e-mailing to: security@comverse.com. Thank You."

--_000_85C7265F443C1249837736D58834E9E0CD4BD6F1CCEXTLVcomverse_--