incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apurv Verma <dapu...@gmail.com>
Subject Re: java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream
Date Sun, 26 Feb 2012 12:46:02 GMT
@Thomas,

public class BSPMessageCompressorFactory<M extends Writable> {

  public static final String COMPRESSION_CODEC_CLASS =
"hama.messenger.compression.class";

  /**
   * Returns a compressor via reflection based on what was configured.
   *
   * @param conf
   * @return
   */
  @SuppressWarnings("unchecked")
  public BSPMessageCompressor<M> getCompressor(Configuration conf) {
    if (conf.get(COMPRESSION_CODEC_CLASS) != null) {
      try {
        return (BSPMessageCompressor<M>) ReflectionUtils.newInstance(conf
            .getClassByName(conf.get(COMPRESSION_CODEC_CLASS,
                SnappyCompressor.class.getCanonicalName())), conf);
      } catch (ClassNotFoundException e) {
        e.printStackTrace();
      }
    }
    return null;
  }
}



 ReflectionUtils.newInstance(conf
            .getClassByName(conf.get(COMPRESSION_CODEC_CLASS,
                SnappyCompressor.class.getCanonicalName())), conf);


Wouldn't Snappy be still the default compressor due to this line even if
one deletes "hama.messenger.compression.class" property in hama-default.xml.

--
thanks and regards,

Apurv Verma
B. Tech.(CSE)
IIT- Ropar






On Sun, Feb 26, 2012 at 5:59 PM, Thomas Jungblut <
thomas.jungblut@googlemail.com> wrote:

> You can delete the "hama.messenger.compression.class" property in
> hama-default.xml.
>
> 2012/2/26 Edward J. Yoon <edwardyoon@apache.org>
>
> > Is there currently no way to disable the message compression?
> >
> > On Fri, Feb 24, 2012 at 4:50 PM, Thomas Jungblut
> > <thomas.jungblut@googlemail.com> wrote:
> > > I have reworked the Avro testcase to test these issues.
> > > (org.apache.hama.bsp.message.TestAvroMessageManager).
> > >
> > > There I am sending 3 types of messages, booleans, doubles and ints.
> They
> > > are packed into a single bundle, compressed and transfered and back at
> > home
> > > they get deserialized.
> > > The boolean package seem to get to the desitnation correctly since B is
> > the
> > > first in the alphabet.
> > > However, it seems that Integers and Doubles aren't transferred
> correctly:
> > > EOF Exception while deserializing.
> > >
> > > Don't waste your time with it, I write him a mail later on.
> > > The JNI version works correctly and is just a bit slower on
> compression,
> > > whereas it is faster on decompression.
> > >
> > > No need to waste the time ;)
> > >
> > > 2012/2/24 Apurv Verma <dapurv5@gmail.com>
> > >
> > >> OK, I see it now. You mean to say the JNI-Snappy which is in
> org.xerial
> > >> package works correctly but not Dain's snappy which is in org.iq80 ,
> > which
> > >> is pure Java.
> > >>
> > >> I would try to check all of these after my exams, once again.(Day
> After
> > >> tomorrow) I remember that I used 0.3-Incubating version of Dain's
> snappy
> > >> and it seemed to work fine. I was able to run Pi Example with it.
> > >>
> > >> Now if we are to use the JNI version of snappy it might be slow so
> > instead
> > >> of using it as a default compressor, we would like to give option of
> > >> choosing between compression and no-compression?
> > >>
> > >>
> > >>
> > >> --
> > >> thanks and regards,
> > >>
> > >> Apurv Verma
> > >> B. Tech.(CSE)
> > >> IIT- Ropar
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Fri, Feb 24, 2012 at 4:36 AM, Apurv Verma <dapurv5@gmail.com>
> wrote:
> > >>
> > >> > What is buggy in Dain's snappy, can you let me know?
> > >> >
> > >> > --
> > >> > thanks and regards,
> > >> >
> > >> > Apurv Verma
> > >> > B. Tech.(CSE)
> > >> > IIT- Ropar
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Feb 23, 2012 at 5:06 PM, Thomas Jungblut <
> > >> > thomas.jungblut@googlemail.com> wrote:
> > >> >
> > >> >> It seems that Dain's snappy is buggy. I tried the Hadoop codec
as
> > well
> > >> as
> > >> >> directly using streams. However, I just changed it to the xerial
> > version
> > >> >> in
> > >> >> POM and it now works correctly.
> > >> >>
> > >> >> I will contact dain and try to sort out things.  Please update
the
> > trunk
> > >> >> and rebuild.
> > >> >> Sorry for the inconvenience.
> > >> >>
> > >> >> 2012/2/23 Thomas Jungblut <thomas.jungblut@googlemail.com>
> > >> >>
> > >> >> > Yep, maybe this is the issue. In the Maven POM is the dain
> > version, in
> > >> >> > impor statements the xerial version. Sorry for the
> inconsistency. I
> > >> fix
> > >> >> it.
> > >> >> >
> > >> >> >
> > >> >> > 2012/2/23 Thomas Jungblut <thomas.jungblut@googlemail.com>
> > >> >> >
> > >> >> >> Wait a minute, I guess this is the wrong library.
> > >> >> >>
> > >> >> >> My intention was to use the  https://github.com/dain/snappylib,
> > >> since
> > >> >> >> it is faster than the JNI port.
> > >> >> >> I investigate it after launch.
> > >> >> >>
> > >> >> >>
> > >> >> >> 2012/2/23 Edward J. Yoon <edwardyoon@apache.org>
> > >> >> >>
> > >> >> >>> Hmm, the snappy-0.2.jar is already in lib folder.
> > >> >> >>>
> > >> >> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
> > >> >> >>> ant-1.7.1.jar           commons-configuration-1.6.jar
> > >> >> >>> hadoop-test-1.0.0.jar         jetty-util-6.1.14.jar
> > >> >> >>> netty-3.2.6.Final.jar    zookeeper-3.3.3.jar
> > >> >> >>> ant-launcher-1.7.1.jar  commons-lang-2.6.jar
> > >> >> >>> jackson-core-asl-1.9.2.jar    jsp-2.1-6.1.14.jar
> > >> >> >>> servlet-api-6.0.32.jar
> > >> >> >>> avro-1.6.0.jar          commons-logging-1.1.1.jar
> > >> >> >>> jackson-mapper-asl-1.9.2.jar  jsp-api-2.1-6.1.14.jar
> > >> >> >>> slf4j-api-1.5.8.jar
> > >> >> >>> avro-ipc-1.6.0.jar      guava-10.0.1.jar
> > >> >> >>> jetty-6.1.14.jar              junit-4.8.1.jar
> > >> >> >>> slf4j-log4j12-1.5.8.jar
> > >> >> >>> commons-cli-1.2.jar     hadoop-core-1.0.0.jar
> > >> >> >>> jetty-annotations-6.1.14.jar  log4j-1.2.16.jar
> >  snappy-0.2.jar
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> On Thu, Feb 23, 2012 at 4:51 PM, Thomas Jungblut
> > >> >> >>> <thomas.jungblut@googlemail.com> wrote:
> > >> >> >>> > You have to update your jars, Snappy is a new
dependency for
> > >> >> >>> compression.
> > >> >> >>> >
> > >> >> >>> > 2012/2/23 Edward J. Yoon <edwardyoon@apache.org>
> > >> >> >>> >
> > >> >> >>> >> I received below error message with pi example.
> > >> >> >>> >>
> > >> >> >>> >> 12/02/23 15:26:16 DEBUG sync.ZooKeeperSyncClientImpl:
> > >> >> leaveBarrier()
> > >> >> >>> >> at superstep:-1 znode size: (0) znodes:[]
> > >> >> >>> >> 12/02/23 15:26:16 WARN bsp.GroomServer:
Error running child
> > >> >> >>> >> java.lang.NoClassDefFoundError:
> > >> org/xerial/snappy/SnappyInputStream
> > >> >> >>> >>        at java.lang.Class.forName0(Native
Method)
> > >> >> >>> >>        at java.lang.Class.forName(Class.java:247)
> > >> >> >>> >>        at
> > >> >> >>> >>
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819)
> > >> >> >>> >>        at
> > >> >> >>> >>
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.hama.bsp.message.compress.BSPMessageCompressorFactory.getCompressor(BSPMessageCompressorFactory.java:37)
> > >> >> >>> >>        at
> > >> >> >>> >>
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.hama.bsp.message.CompressableMessageManager.initCompression(CompressableMessageManager.java:36)
> > >> >> >>> >>        at
> > >> >> >>> >>
> > >> >> >>>
> > >> >>
> > >>
> >
> org.apache.hama.bsp.message.AvroMessageManagerImpl.init(AvroMessageManagerImpl.java:60)
> > >> >> >>> >>        at
> > >> >> org.apache.hama.bsp.BSPPeerImpl.<init>(BSPPeerImpl.java:145)
> > >> >> >>> >>        at
> > >> >> >>> >>
> > >> >> >>>
> > >> >>
> > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:969)
> > >> >> >>> >> Caused by: java.lang.ClassNotFoundException:
> > >> >> >>> >> org.xerial.snappy.SnappyInputStream
> > >> >> >>> >>        at
> > java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> > >> >> >>> >>        at java.security.AccessController.doPrivileged(Native
> > >> >> Method)
> > >> >> >>> >>        at
> > >> >> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> > >> >> >>> >>        at
> > java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > >> >> >>> >>        at
> > >> >> >>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> > >> >> >>> >>        at
> > java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> > >> >> >>> >>
> > >> >> >>> >>
> > >> >> >>> >> --
> > >> >> >>> >> Best Regards, Edward J. Yoon
> > >> >> >>> >> @eddieyoon
> > >> >> >>> >>
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > --
> > >> >> >>> > Thomas Jungblut
> > >> >> >>> > Berlin <thomas.jungblut@gmail.com>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> --
> > >> >> >>> Best Regards, Edward J. Yoon
> > >> >> >>> @eddieyoon
> > >> >> >>>
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> --
> > >> >> >> Thomas Jungblut
> > >> >> >> Berlin <thomas.jungblut@gmail.com>
> > >> >> >>
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > --
> > >> >> > Thomas Jungblut
> > >> >> > Berlin <thomas.jungblut@gmail.com>
> > >> >> >
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Thomas Jungblut
> > >> >> Berlin <thomas.jungblut@gmail.com>
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Thomas Jungblut
> > > Berlin <thomas.jungblut@gmail.com>
> >
> >
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
>
>
>
> --
> Thomas Jungblut
> Berlin <thomas.jungblut@gmail.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message