incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream
Date Sun, 26 Feb 2012 13:06:03 GMT
What is the best example to see the performance improvement by
compression? PageRank?

On Sun, Feb 26, 2012 at 9:49 PM, Thomas Jungblut
<thomas.jungblut@googlemail.com> wrote:
> @Apurv, no because of: if (conf.get(COMPRESSION_CODEC_CLASS) != null). If
> it is not mapped in the configuration, then the returned string is null.
>
> I think we should provide a setter for it so that user can specify
>> compression type and optimize their jobs.
>
>
> +1.
>
> 2012/2/26 Edward J. Yoon <edwardyoon@apache.org>
>
>> I think we should provide a setter for it so that user can specify
>> compression type and optimize their jobs.
>>
>> On Sun, Feb 26, 2012 at 9:29 PM, Thomas Jungblut
>> <thomas.jungblut@googlemail.com> wrote:
>> > You can delete the "hama.messenger.compression.class" property in
>> > hama-default.xml.
>> >
>> > 2012/2/26 Edward J. Yoon <edwardyoon@apache.org>
>> >
>> >> Is there currently no way to disable the message compression?
>> >>
>> >> On Fri, Feb 24, 2012 at 4:50 PM, Thomas Jungblut
>> >> <thomas.jungblut@googlemail.com> wrote:
>> >> > I have reworked the Avro testcase to test these issues.
>> >> > (org.apache.hama.bsp.message.TestAvroMessageManager).
>> >> >
>> >> > There I am sending 3 types of messages, booleans, doubles and ints.
>> They
>> >> > are packed into a single bundle, compressed and transfered and back
at
>> >> home
>> >> > they get deserialized.
>> >> > The boolean package seem to get to the desitnation correctly since
B
>> is
>> >> the
>> >> > first in the alphabet.
>> >> > However, it seems that Integers and Doubles aren't transferred
>> correctly:
>> >> > EOF Exception while deserializing.
>> >> >
>> >> > Don't waste your time with it, I write him a mail later on.
>> >> > The JNI version works correctly and is just a bit slower on
>> compression,
>> >> > whereas it is faster on decompression.
>> >> >
>> >> > No need to waste the time ;)
>> >> >
>> >> > 2012/2/24 Apurv Verma <dapurv5@gmail.com>
>> >> >
>> >> >> OK, I see it now. You mean to say the JNI-Snappy which is in
>> org.xerial
>> >> >> package works correctly but not Dain's snappy which is in org.iq80
,
>> >> which
>> >> >> is pure Java.
>> >> >>
>> >> >> I would try to check all of these after my exams, once again.(Day
>> After
>> >> >> tomorrow) I remember that I used 0.3-Incubating version of Dain's
>> snappy
>> >> >> and it seemed to work fine. I was able to run Pi Example with it.
>> >> >>
>> >> >> Now if we are to use the JNI version of snappy it might be slow
so
>> >> instead
>> >> >> of using it as a default compressor, we would like to give option
of
>> >> >> choosing between compression and no-compression?
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> thanks and regards,
>> >> >>
>> >> >> Apurv Verma
>> >> >> B. Tech.(CSE)
>> >> >> IIT- Ropar
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Fri, Feb 24, 2012 at 4:36 AM, Apurv Verma <dapurv5@gmail.com>
>> wrote:
>> >> >>
>> >> >> > What is buggy in Dain's snappy, can you let me know?
>> >> >> >
>> >> >> > --
>> >> >> > thanks and regards,
>> >> >> >
>> >> >> > Apurv Verma
>> >> >> > B. Tech.(CSE)
>> >> >> > IIT- Ropar
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Thu, Feb 23, 2012 at 5:06 PM, Thomas Jungblut <
>> >> >> > thomas.jungblut@googlemail.com> wrote:
>> >> >> >
>> >> >> >> It seems that Dain's snappy is buggy. I tried the Hadoop
codec as
>> >> well
>> >> >> as
>> >> >> >> directly using streams. However, I just changed it to
the xerial
>> >> version
>> >> >> >> in
>> >> >> >> POM and it now works correctly.
>> >> >> >>
>> >> >> >> I will contact dain and try to sort out things.  Please
update the
>> >> trunk
>> >> >> >> and rebuild.
>> >> >> >> Sorry for the inconvenience.
>> >> >> >>
>> >> >> >> 2012/2/23 Thomas Jungblut <thomas.jungblut@googlemail.com>
>> >> >> >>
>> >> >> >> > Yep, maybe this is the issue. In the Maven POM is
the dain
>> >> version, in
>> >> >> >> > impor statements the xerial version. Sorry for the
>> inconsistency. I
>> >> >> fix
>> >> >> >> it.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > 2012/2/23 Thomas Jungblut <thomas.jungblut@googlemail.com>
>> >> >> >> >
>> >> >> >> >> Wait a minute, I guess this is the wrong library.
>> >> >> >> >>
>> >> >> >> >> My intention was to use the  https://github.com/dain/snappylib,
>> >> >> since
>> >> >> >> >> it is faster than the JNI port.
>> >> >> >> >> I investigate it after launch.
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> 2012/2/23 Edward J. Yoon <edwardyoon@apache.org>
>> >> >> >> >>
>> >> >> >> >>> Hmm, the snappy-0.2.jar is already in lib
folder.
>> >> >> >> >>>
>> >> >> >> >>> edward@slave:~/workspace/hama-trunk$ ls lib/
>> >> >> >> >>> ant-1.7.1.jar           commons-configuration-1.6.jar
>> >> >> >> >>> hadoop-test-1.0.0.jar         jetty-util-6.1.14.jar
>> >> >> >> >>> netty-3.2.6.Final.jar    zookeeper-3.3.3.jar
>> >> >> >> >>> ant-launcher-1.7.1.jar  commons-lang-2.6.jar
>> >> >> >> >>> jackson-core-asl-1.9.2.jar    jsp-2.1-6.1.14.jar
>> >> >> >> >>> servlet-api-6.0.32.jar
>> >> >> >> >>> avro-1.6.0.jar          commons-logging-1.1.1.jar
>> >> >> >> >>> jackson-mapper-asl-1.9.2.jar  jsp-api-2.1-6.1.14.jar
>> >> >> >> >>> slf4j-api-1.5.8.jar
>> >> >> >> >>> avro-ipc-1.6.0.jar      guava-10.0.1.jar
>> >> >> >> >>> jetty-6.1.14.jar              junit-4.8.1.jar
>> >> >> >> >>> slf4j-log4j12-1.5.8.jar
>> >> >> >> >>> commons-cli-1.2.jar     hadoop-core-1.0.0.jar
>> >> >> >> >>> jetty-annotations-6.1.14.jar  log4j-1.2.16.jar
>> >>  snappy-0.2.jar
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> On Thu, Feb 23, 2012 at 4:51 PM, Thomas Jungblut
>> >> >> >> >>> <thomas.jungblut@googlemail.com> wrote:
>> >> >> >> >>> > You have to update your jars, Snappy
is a new dependency for
>> >> >> >> >>> compression.
>> >> >> >> >>> >
>> >> >> >> >>> > 2012/2/23 Edward J. Yoon <edwardyoon@apache.org>
>> >> >> >> >>> >
>> >> >> >> >>> >> I received below error message with
pi example.
>> >> >> >> >>> >>
>> >> >> >> >>> >> 12/02/23 15:26:16 DEBUG sync.ZooKeeperSyncClientImpl:
>> >> >> >> leaveBarrier()
>> >> >> >> >>> >> at superstep:-1 znode size: (0)
znodes:[]
>> >> >> >> >>> >> 12/02/23 15:26:16 WARN bsp.GroomServer:
Error running child
>> >> >> >> >>> >> java.lang.NoClassDefFoundError:
>> >> >> org/xerial/snappy/SnappyInputStream
>> >> >> >> >>> >>        at java.lang.Class.forName0(Native
Method)
>> >> >> >> >>> >>        at java.lang.Class.forName(Class.java:247)
>> >> >> >> >>> >>        at
>> >> >> >> >>> >>
>> >> >> >> >>>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:819)
>> >> >> >> >>> >>        at
>> >> >> >> >>> >>
>> >> >> >> >>>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hama.bsp.message.compress.BSPMessageCompressorFactory.getCompressor(BSPMessageCompressorFactory.java:37)
>> >> >> >> >>> >>        at
>> >> >> >> >>> >>
>> >> >> >> >>>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hama.bsp.message.CompressableMessageManager.initCompression(CompressableMessageManager.java:36)
>> >> >> >> >>> >>        at
>> >> >> >> >>> >>
>> >> >> >> >>>
>> >> >> >>
>> >> >>
>> >>
>> org.apache.hama.bsp.message.AvroMessageManagerImpl.init(AvroMessageManagerImpl.java:60)
>> >> >> >> >>> >>        at
>> >> >> >> org.apache.hama.bsp.BSPPeerImpl.<init>(BSPPeerImpl.java:145)
>> >> >> >> >>> >>        at
>> >> >> >> >>> >>
>> >> >> >> >>>
>> >> >> >>
>> >> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:969)
>> >> >> >> >>> >> Caused by: java.lang.ClassNotFoundException:
>> >> >> >> >>> >> org.xerial.snappy.SnappyInputStream
>> >> >> >> >>> >>        at
>> >> java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >> >> >> >>> >>        at
>> java.security.AccessController.doPrivileged(Native
>> >> >> >> Method)
>> >> >> >> >>> >>        at
>> >> >> >> java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >> >> >> >>> >>        at
>> >> java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>> >> >> >> >>> >>        at
>> >> >> >> >>> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >> >> >> >>> >>        at
>> >> java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> --
>> >> >> >> >>> >> Best Regards, Edward J. Yoon
>> >> >> >> >>> >> @eddieyoon
>> >> >> >> >>> >>
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > --
>> >> >> >> >>> > Thomas Jungblut
>> >> >> >> >>> > Berlin <thomas.jungblut@gmail.com>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> Best Regards, Edward J. Yoon
>> >> >> >> >>> @eddieyoon
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> Thomas Jungblut
>> >> >> >> >> Berlin <thomas.jungblut@gmail.com>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > Thomas Jungblut
>> >> >> >> > Berlin <thomas.jungblut@gmail.com>
>> >> >> >> >
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Thomas Jungblut
>> >> >> >> Berlin <thomas.jungblut@gmail.com>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Thomas Jungblut
>> >> > Berlin <thomas.jungblut@gmail.com>
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >>
>> >
>> >
>> >
>> > --
>> > Thomas Jungblut
>> > Berlin <thomas.jungblut@gmail.com>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>
>
> --
> Thomas Jungblut
> Berlin <thomas.jungblut@gmail.com>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message