hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Olaf Marzocchi <li...@marzocchi.net>
Subject Re: "Quickstart" on OmniOS/illumos not working, java eats CPU
Date Mon, 14 Nov 2016 22:18:36 GMT
Hi,
I will think about it, but both the OpenJDK1.7 by OmniTI and the Oracle 
JDK1.8 set the value to 256.

In case of problems I will check it again.

Olaf


On 14/11/2016 19:08, Esteban Gutierrez wrote:
> Just a quick observation here: the number of max file descriptors
> configured is way too low in my opinion. Also, it has been few years since
> last time I played with OpenSolaris but I'm going to assume that Illumos
> you could run truss and the prstat commands and trace the jvm to see whats
> going on. My guess, is that the JVM is spinning way too fast due the
> ZooKeeper server. One way to test this theory is to start each process on
> separate JVM and see how things behave.
>
> cheers,
> esteban.
>
>
>
> --
> Cloudera, Inc.
>
>
> On Mon, Nov 14, 2016 at 8:38 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Olaf:
>> After switching away from OmniOS Java, did you still encounter similar
>> problem ?
>>
>> If so, mind updating the thread with log / stack trace ?
>>
>> Thanks
>>
>> On Sun, Nov 13, 2016 at 8:40 AM, Olaf Marzocchi <lists@marzocchi.net>
>> wrote:
>>
>>> I checked and they are located in the same folder.
>>> I found that java and jstack come from two different packages (java from
>>> java and jstack from jdk), but they are both preinstalled with OmniOS,
>>> maintained by OmniTI and compiled at the same time as the operating
>> system.
>>> They should be 100% compatible.
>>>
>>> I will ask the OmniOS mailing list.
>>>
>>> Olaf
>>>
>>>
>>> On 13/11/2016 17:28, Ted Yu wrote:
>>>
>>>> bq. Caused by: java.lang.UnsatisfiedLinkError: no saproc in
>>>> java.library.path
>>>>
>>>> Was the jstack of version 1.7.0_101 ?
>>>> I used google and found:
>>>> http://umchee.blogspot.com/p/i-recently-needed-to-investigate.html
>>>>
>>>> but it was about Java 1.5
>>>>
>>>> You can use this command to find the commandline for hbase process
>>>> (replace 16059
>>>> with the new process Id):
>>>> ps aux | grep 16059
>>>>
>>>> Then use the jstack from same path as java to capture stack trace:
>>>>
>>>> Thanks
>>>>
>>>> On Sun, Nov 13, 2016 at 8:16 AM, Olaf Marzocchi <lists@marzocchi.net>
>>>> wrote:
>>>>
>>>> I'm not 100% sure about the stack trace (not a developer myself), but I
>>>>> searched online and used jstack:
>>>>> http://pastebin.com/AT6pScBv
>>>>>
>>>>> Additional info: after turning on DEBUG, the java process did not
>>>>> immediately jump to 100% cpu as it used to do, it ramped up and it
>> took 3
>>>>> minutes to get there. It may however be a coincidence.
>>>>>
>>>>> Hope this can help pinpointing the issue.
>>>>>
>>>>> Olaf
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 13/11/2016 01:18, Ted Yu wrote:
>>>>>
>>>>> Can you turn on DEBUG logging ?
>>>>>> In log4j.properties :
>>>>>>
>>>>>> log4j.logger.org.apache.hadoop.hbase=DEBUG
>>>>>>
>>>>>> Since this is reproducible, can you capture one or two stack traces
>>>>>> before
>>>>>> killing the server ?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sat, Nov 12, 2016 at 3:09 PM, Olaf Marzocchi <lists@marzocchi.net>
>>>>>> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>> I tested HBase with OpenTSDB some time ago on OS X locally and
the
>>>>>>> quickstart guide worked perfectly, merged with
>>>>>>> http://opentsdb.net/setup-hbase.html where necessary.
>>>>>>>
>>>>>>> I tried yesterday to do the same on my OmniOS/illumos server
and I
>>>>>>> encountered issues with HBase.
>>>>>>>
>>>>>>> First of all I set "JAVA_HOME=/usr/java" in "conf/hbase-env.sh".
>>>>>>>
>>>>>>> $ /usr/java/bin/java -version
>>>>>>> openjdk version "1.7.0_101"
>>>>>>> OpenJDK Runtime Environment (build 1.7.0_101-b00)
>>>>>>> OpenJDK Server VM (build 24.101-b00, mixed mode)
>>>>>>>
>>>>>>> I edited also "hbase-site.xml":
>>>>>>>
>>>>>>> <configuration>
>>>>>>>   <property>
>>>>>>>     <name>hbase.rootdir</name>
>>>>>>>     <value>file:///export/home/olaf/hbase</value>
>>>>>>>   </property>
>>>>>>>   <property>
>>>>>>>     <name>hbase.zookeeper.property.dataDir</name>
>>>>>>>     <value>/export/home/olaf/zookeeper</value>
>>>>>>>   </property>
>>>>>>> </configuration>
>>>>>>>
>>>>>>> I then launched "bin/start-hbase.sh" and it quits apparently
>> normally.
>>>>>>> A
>>>>>>> directory "/export/home/olaf/zookeeper" appeared, but no
>>>>>>> /export/home/olaf/hbase.
>>>>>>>
>>>>>>> $ ls -l /export/home/olaf/
>>>>>>> drwxrwxr-x   3 olaf     olaf           3 Nov 12 01:16 zookeeper
>>>>>>>
>>>>>>> I also checked with prstat ("top") and I see that java is taking
a
>> full
>>>>>>> CPU. After some minutes it's still crunching something.
>>>>>>> If I launch "./bin/hbase shell" it never gets to
>> "hbase(main):001:0>".
>>>>>>> At this point the only way is to kill -9 both java processes:
the
>> stop
>>>>>>> script doesn't work, it outputs dots forever.
>>>>>>>
>>>>>>> I uploaded the two log files (.log and .out) to Pastebin:
>>>>>>> http://pastebin.com/aFMXmWYU
>>>>>>>
>>>>>>> I'm not sure where to begin with the debugging, since I did no
>>>>>>> customisation at all and I don't see any error logged.
>>>>>>>
>>>>>>> I would be grateful for some help.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Olaf
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>
>

Mime
View raw message