cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paulo Motta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-9902) Windows dtest: invalid jolokia agent path
Date Thu, 30 Jul 2015 00:20:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646961#comment-14646961
] 

Paulo Motta edited comment on CASSANDRA-9902 at 7/30/15 12:19 AM:
------------------------------------------------------------------

The original error is caused by conflicting JDKs (namely 7 and 8) installed in the same system,
which are normally switched by changing the {{$JAVA_HOME}} and adding {{$JAVA_HOME\bin}} to
the {{$PATH}} environment variable. It may also necessary to update the registry when switching
JDKs by running the following command on PowerShell:

{noformat}
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment\'
CurrentVersion 1.7
{noformat}

When "java" without the parent directory is called via python's {{subprocess.check_output}}
on [cassandra-dtest/jmxutils.py|https://github.com/riptano/cassandra-dtest/blob/master/jmxutils.py#L80],
the {{$PATH}} env var is not considered due to this [bug|http://bugs.python.org/issue8557],
so instead, the java executable from {{c:\windows\system32\}} is picked, which can be pointing
to another version. The simple fix is to use the {{$JAVA_HOME}} variable, if present, when
invoking java, as done in this [v1|https://github.com/pauloricardomg/cassandra-dtest/tree/9902-v1]
patch.

However, this is not all the fun, after this fix, the following error appears:
{noformat}
Failed to start jolokia agent (command was: C:\Program Files\Java\jdk1.7.0_76\bin\java -jar
lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 1992 --debug --verbose): Command '('C:\\Program
Files\\Java\\jdk1.7.0_76\\bin\\java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host',
'127.0.0.1', 'start', '1992', '--debug', '--verbose')' returned non-zero exit status 1
Exit status was: 1
Output was: java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider:
Provider sun.tools.attach.WindowsAttachProvider could not be instantiated
InvocationTarget class com.sun.tools.attach.VirtualMachine (command: start) : null
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jolokia.jvmagent.client.util.VirtualMachineHandler.attachVirtualMachine(VirtualMachineHandler.java:66)
        at org.jolokia.jvmagent.client.AgentLauncher.main(AgentLauncher.java:57)
Caused by: com.sun.tools.attach.AttachNotSupportedException: no providers installed
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
        ... 6 more
{noformat}

According to this [blog post|http://wayne-adams.blogspot.com.br/2011/10/jdi-three-ways-to-attach-to-java.html],
this issue happens when the tools.jar is not loaded by the JRE. After inspecting the jolokia
[vm instantiation code|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java],
I found that the {{com.sun.tools.attach.VirtualMachine}} class is [dynamically loaded|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/ToolsClassFinder.java]
from the tools.jar, however when {{VirtualMachine.attach}} [is called|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java#L66],
the JVM is not able to load the {{sun.tools.attach.WindowsAttachProvider}} class because the
tools.jar is not on the classpath (so that's probably a bug with jolokia, or Java on Windows).

The simple fix is to explicitly add the tools.jar to the jolokia classpath, as suggested [here|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java#L206].
This was done in  [v2|https://github.com/pauloricardomg/cassandra-dtest/tree/9902-v2] patch.

Apparently this seems to make jolokia connect with the testing JVM, however, this makes the
Cassandra JVM crash without java crash logs or heap dumps. The only thing we have is a windows
crash log ([attached here|https://issues.apache.org/jira/secure/attachment/12747879/win-crashlogs.zip]),
which indicates the JVM crashes because of a [Data Execution Prevention|https://technet.microsoft.com/en-us/library/cc738483%28v=ws.10%29.aspx]
exception. There is a feature to [whitelist a program from DEP|http://windows.microsoft.com/en-us/windows7/Change-Data-Execution-Prevention-settings],
but apparently it's not possible to disable DEP for 64-bit programs o_O.

I don't know what might be causing this, if a JVM option that is set by Cassandra or an actual
Windows JVM bug. In order to simplify reproduction, I made a simple class to reproduce the
issue ([VMAttach.java|https://issues.apache.org/jira/secure/attachment/12747880/VMAttach.java]),
which receives the PID of a java process and tries to attach to it using sun attach tools
(the same used by jolokia). In order to compile and run:

{noformat}
javac -cp "$ENV:JAVA_HOME\lib\tools.jar;." VMAttach.java
java -cp "$ENV:JAVA_HOME\lib\tools.jar;." VMAttach.java <pid>
{noformat}

When this program is run against a cassandra process/pid, it makes the cassandra process crash
silently.

Could you please try reproducing this on your box [~JoshuaMcKenzie]? Do you have any idea
of what might be causing this?


was (Author: pauloricardomg):
The original error is caused by conflicting JDKs (namely 7 and 8) installed in the same system,
which are normally switched by changing the {{$JAVA_HOME}} and adding {{$JAVA_HOME\bin}} to
the {{$PATH}} environment variable. It may also necessary to update the registry when switching
JDKs by running the following command on PowerShell:

{noformat}
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\JavaSoft\Java Runtime Environment\'
CurrentVersion 1.7
{noformat}

When "java" without the parent directory is called via python's {{subprocess.check_output}}
on [cassandra-dtest/jmxutils.py|https://github.com/riptano/cassandra-dtest/blob/master/jmxutils.py#L80],
the {{$PATH}} env var is not considered due to this [bug|http://bugs.python.org/issue8557],
so instead, the java executable from {{c:\windows\system32\}} is picked, which can be pointing
to another version. The simple fix is to use the {{$JAVA_HOME}} variable, if present, as done
in this [v1|https://github.com/pauloricardomg/cassandra-dtest/tree/9902-v1] patch.

However, this is not all the fun, after this fix, the following error appears:
{noformat}
Failed to start jolokia agent (command was: C:\Program Files\Java\jdk1.7.0_76\bin\java -jar
lib\jolokia-jvm-1.2.3-agent.jar --host 127.0.0.1 start 1992 --debug --verbose): Command '('C:\\Program
Files\\Java\\jdk1.7.0_76\\bin\\java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar', '--host',
'127.0.0.1', 'start', '1992', '--debug', '--verbose')' returned non-zero exit status 1
Exit status was: 1
Output was: java.util.ServiceConfigurationError: com.sun.tools.attach.spi.AttachProvider:
Provider sun.tools.attach.WindowsAttachProvider could not be instantiated
InvocationTarget class com.sun.tools.attach.VirtualMachine (command: start) : null
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jolokia.jvmagent.client.util.VirtualMachineHandler.attachVirtualMachine(VirtualMachineHandler.java:66)
        at org.jolokia.jvmagent.client.AgentLauncher.main(AgentLauncher.java:57)
Caused by: com.sun.tools.attach.AttachNotSupportedException: no providers installed
        at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
        ... 6 more
{noformat}

According to this [blog post|http://wayne-adams.blogspot.com.br/2011/10/jdi-three-ways-to-attach-to-java.html],
this issue happens when the tools.jar is not loaded by the JRE. After inspecting the jolokia
[vm instatiation code|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java],
I found that the {{com.sun.tools.attach.VirtualMachine}} class is [dynamically loaded|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/ToolsClassFinder.java]
from the tools.jar, however when {{VirtualMachine.attach}} is called, the JVM is not able
to load the {{sun.tools.attach.WindowsAttachProvider}} class because the tools.jar is not
on the classpath (so that's probably a bug with jolokia, or Java on Windows).

The simple fix is to explicitly add the tools.jar to the jolokia classpath, as suggested [here|https://github.com/rhuss/jolokia/blob/aadf5c19c2052bdfa2bad470c55f4c15c1005d26/agent/jvm/src/main/java/org/jolokia/jvmagent/client/util/VirtualMachineHandler.java#L206].
This was done in  [v2|https://github.com/pauloricardomg/cassandra-dtest/tree/9902-v2] patch.

Apparently this seems to make jolokia connect with the testing JVM, however, this makes the
Cassandra JVM crash without java crash logs or heap dumps. The only thing we have is a windows
crash log ([attached here|https://issues.apache.org/jira/secure/attachment/12747879/win-crashlogs.zip]),
which indicates the JVM crashes because of a [Data Execution Prevention|https://technet.microsoft.com/en-us/library/cc738483%28v=ws.10%29.aspx]
exception. There is a feature to [whitelist a program from DEP|http://windows.microsoft.com/en-us/windows7/Change-Data-Execution-Prevention-settings],
but apparently it's not possible to disable DEP for 64-bit programs o_O.

I don't know what might be causing this, if a JVM option that is set by Cassandra or an actual
Windows JVM bug. In order to simplify reproduction, I made a simple class to reproduce the
issue ([VMAttach.java|https://issues.apache.org/jira/secure/attachment/12747880/VMAttach.java]),
which receives the PID of a java process and tries to attach to it using sun attach tools
(the same used by jolokia). In order to compile and run:

{noformat}
javac -cp "$ENV:JAVA_HOME\lib\tools.jar;." VMAttach.java
java -cp "$ENV:JAVA_HOME\lib\tools.jar;." VMAttach.java <pid>
{noformat}

When this program is run against a cassandra process/pid, it makes the cassandra process crash
silently.

Could you please try reproducing this on your box [~JoshuaMcKenzie]? Do you have any idea
of what might be causing this?

> Windows dtest: invalid jolokia agent path
> -----------------------------------------
>
>                 Key: CASSANDRA-9902
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9902
>             Project: Cassandra
>          Issue Type: Sub-task
>            Reporter: Joshua McKenzie
>            Assignee: Paulo Motta
>            Priority: Minor
>              Labels: Windows
>             Fix For: 2.2.x
>
>         Attachments: VMAttach.java, win-crashlogs.zip
>
>
> A variety of tests are failing with similar:
> {noformat}
> 05:39:07 ======================================================================
> 05:39:07 ERROR: change_durable_writes_test (configuration_test.TestConfiguration)
> 05:39:07 ----------------------------------------------------------------------
> 05:39:07 Traceback (most recent call last):
> 05:39:07   File "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py",
line 63, in change_durable_writes_test
> 05:39:07     durable_init_size = commitlog_size(durable_node)
> 05:39:07   File "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\configuration_test.py",
line 127, in commitlog_size
> 05:39:07     with JolokiaAgent(node) as jmx:
> 05:39:07   File "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py",
line 188, in __enter__
> 05:39:07     self.start()
> 05:39:07   File "D:\jenkins\workspace\cassandra-2.2_dtest_win32\cassandra-dtest\jmxutils.py",
line 80, in start
> 05:39:07     subprocess.check_output(args, stderr=subprocess.STDOUT)
> 05:39:07   File "D:\Python27\lib\subprocess.py", line 573, in check_output
> 05:39:07     raise CalledProcessError(retcode, cmd, output=output)
> 05:39:07 CalledProcessError: Command '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar',
'--host', '127.0.0.1', 'start', '3436')' returned non-zero exit status 2
> 05:39:07 -------------------- >> begin captured logging << --------------------
> 05:39:07 dtest: DEBUG: cluster ccm directory: d:\temp\dtest-kjkypw
> 05:39:07 --------------------- >> end captured logging << ---------------------
> {noformat}
> Likely related:
> {noformat}
> 20:54:46 Failed to start jolokia agent (command was: java -jar lib\jolokia-jvm-1.2.3-agent.jar
--host 127.0.0.1 start 3436): Command '('java', '-jar', 'lib\\jolokia-jvm-1.2.3-agent.jar',
'--host', '127.0.0.1', 'start', '3436')' returned non-zero exit status 2
> 20:54:46 Exit status was: 2
> 20:54:46 Output was: Error: Registry key 'Software\JavaSoft\Java Runtime Environment'\CurrentVersion'
> 20:54:46 has value '1.8', but '1.7' is required.
> 20:54:46 Error: could not find java.dll
> 20:54:46 Error: Could not find Java SE Runtime Environment.
> {noformat}
> And at the top of jmxutils.py we see:
> {code}
> JOLOKIA_JAR = os.path.join('lib', 'jolokia-jvm-1.2.3-agent.jar')
> {code}
> So that's not going to work out too well on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message