hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: "Quickstart" on OmniOS/illumos not working, java eats CPU
Date Mon, 14 Nov 2016 18:08:12 GMT
Just a quick observation here: the number of max file descriptors
configured is way too low in my opinion. Also, it has been few years since
last time I played with OpenSolaris but I'm going to assume that Illumos
you could run truss and the prstat commands and trace the jvm to see whats
going on. My guess, is that the JVM is spinning way too fast due the
ZooKeeper server. One way to test this theory is to start each process on
separate JVM and see how things behave.

cheers,
esteban.



--
Cloudera, Inc.


On Mon, Nov 14, 2016 at 8:38 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Olaf:
> After switching away from OmniOS Java, did you still encounter similar
> problem ?
>
> If so, mind updating the thread with log / stack trace ?
>
> Thanks
>
> On Sun, Nov 13, 2016 at 8:40 AM, Olaf Marzocchi <lists@marzocchi.net>
> wrote:
>
> > I checked and they are located in the same folder.
> > I found that java and jstack come from two different packages (java from
> > java and jstack from jdk), but they are both preinstalled with OmniOS,
> > maintained by OmniTI and compiled at the same time as the operating
> system.
> > They should be 100% compatible.
> >
> > I will ask the OmniOS mailing list.
> >
> > Olaf
> >
> >
> > On 13/11/2016 17:28, Ted Yu wrote:
> >
> >> bq. Caused by: java.lang.UnsatisfiedLinkError: no saproc in
> >> java.library.path
> >>
> >> Was the jstack of version 1.7.0_101 ?
> >> I used google and found:
> >> http://umchee.blogspot.com/p/i-recently-needed-to-investigate.html
> >>
> >> but it was about Java 1.5
> >>
> >> You can use this command to find the commandline for hbase process
> >> (replace 16059
> >> with the new process Id):
> >> ps aux | grep 16059
> >>
> >> Then use the jstack from same path as java to capture stack trace:
> >>
> >> Thanks
> >>
> >> On Sun, Nov 13, 2016 at 8:16 AM, Olaf Marzocchi <lists@marzocchi.net>
> >> wrote:
> >>
> >> I'm not 100% sure about the stack trace (not a developer myself), but I
> >>> searched online and used jstack:
> >>> http://pastebin.com/AT6pScBv
> >>>
> >>> Additional info: after turning on DEBUG, the java process did not
> >>> immediately jump to 100% cpu as it used to do, it ramped up and it
> took 3
> >>> minutes to get there. It may however be a coincidence.
> >>>
> >>> Hope this can help pinpointing the issue.
> >>>
> >>> Olaf
> >>>
> >>>
> >>>
> >>>
> >>> On 13/11/2016 01:18, Ted Yu wrote:
> >>>
> >>> Can you turn on DEBUG logging ?
> >>>> In log4j.properties :
> >>>>
> >>>> log4j.logger.org.apache.hadoop.hbase=DEBUG
> >>>>
> >>>> Since this is reproducible, can you capture one or two stack traces
> >>>> before
> >>>> killing the server ?
> >>>>
> >>>> Thanks
> >>>>
> >>>> On Sat, Nov 12, 2016 at 3:09 PM, Olaf Marzocchi <lists@marzocchi.net>
> >>>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>>> I tested HBase with OpenTSDB some time ago on OS X locally and the
> >>>>> quickstart guide worked perfectly, merged with
> >>>>> http://opentsdb.net/setup-hbase.html where necessary.
> >>>>>
> >>>>> I tried yesterday to do the same on my OmniOS/illumos server and
I
> >>>>> encountered issues with HBase.
> >>>>>
> >>>>> First of all I set "JAVA_HOME=/usr/java" in "conf/hbase-env.sh".
> >>>>>
> >>>>> $ /usr/java/bin/java -version
> >>>>> openjdk version "1.7.0_101"
> >>>>> OpenJDK Runtime Environment (build 1.7.0_101-b00)
> >>>>> OpenJDK Server VM (build 24.101-b00, mixed mode)
> >>>>>
> >>>>> I edited also "hbase-site.xml":
> >>>>>
> >>>>> <configuration>
> >>>>>   <property>
> >>>>>     <name>hbase.rootdir</name>
> >>>>>     <value>file:///export/home/olaf/hbase</value>
> >>>>>   </property>
> >>>>>   <property>
> >>>>>     <name>hbase.zookeeper.property.dataDir</name>
> >>>>>     <value>/export/home/olaf/zookeeper</value>
> >>>>>   </property>
> >>>>> </configuration>
> >>>>>
> >>>>> I then launched "bin/start-hbase.sh" and it quits apparently
> normally.
> >>>>> A
> >>>>> directory "/export/home/olaf/zookeeper" appeared, but no
> >>>>> /export/home/olaf/hbase.
> >>>>>
> >>>>> $ ls -l /export/home/olaf/
> >>>>> drwxrwxr-x   3 olaf     olaf           3 Nov 12 01:16 zookeeper
> >>>>>
> >>>>> I also checked with prstat ("top") and I see that java is taking
a
> full
> >>>>> CPU. After some minutes it's still crunching something.
> >>>>> If I launch "./bin/hbase shell" it never gets to
> "hbase(main):001:0>".
> >>>>> At this point the only way is to kill -9 both java processes: the
> stop
> >>>>> script doesn't work, it outputs dots forever.
> >>>>>
> >>>>> I uploaded the two log files (.log and .out) to Pastebin:
> >>>>> http://pastebin.com/aFMXmWYU
> >>>>>
> >>>>> I'm not sure where to begin with the debugging, since I did no
> >>>>> customisation at all and I don't see any error logged.
> >>>>>
> >>>>> I would be grateful for some help.
> >>>>>
> >>>>> Thanks
> >>>>> Olaf
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message