db-derby-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Db-derby Wiki] Update of "DebuggingDerbyJITIssues" by MyrnavanLunteren
Date Thu, 17 Oct 2013 22:18:00 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The "DebuggingDerbyJITIssues" page has been changed by MyrnavanLunteren:

New page:
= DebuggingJITIssues =
JIT problems manifest themselves in many ways. Often the problem is an absurd incorrect return
value or expression evaluation, which can lead to a wide variety of symptoms.  Intermittent
issues are good JIT suspects. A JIT issue is always a JVM problem.

However,  intermittent issues can be caused by other factors, such as race conditions or timing
issues. The first step for analyzing intermittent issues is to run the program multiple times,
to get a feeling for the frequency. Then the first step for debugging an issue where JIT involvement
is suspected is to create a (small) test case, and run it with JIT forced on and with off.
If JIT is implicated, it will fail with JIT on, and pass with JIT off.

Waiting for a JVM's JIT fix can be impractical, and switching JIT off completely may cause
a severe reduction in performance. But it may be possible to analyze the problem and work
around it, either in the source code, or by using tools of the JVM.

For instance, the IBM JVM provides tools for diagnosing JIT issues, described in the section
"Diagnosing a JIT or AOT problem" here: http://pic.dhe.ibm.com/infocenter/java7sdk/v7r0/topic/com.ibm.java.zos.70.doc/diag/tools/jitpd_diagnose.html

== Example ==
Below is an example of analysis as it might be applied to a problem in Derby. The details
(file numbers, failing methods) have been invented.

0 ) Confirm it's JIT

 . Confirm that the JIT is implicated by running the program showing intermittent wrong behavior
with JIT off and ensure it passes. This can be done by running with -Xint or -Xnojit. You
may need to run multiple times to be sure, depending on how frequent the failure is. If the
suspected problem is in a 'old harness' test, you can pass the properties on with -Djvmparams,
but it may be easier to set the properties using the JVM's environment variable IBM_JAVA_OPTIONS.

1) Get a reliable small test case.

 . This is especially important with JIT issues, because  the process of fully analyzing a
JIT issue requires many iterations. Ensure that the test case does indeed reproduce the problem.
It can be frustrating to get a small test case for a JIT issue because seemingly minor code
changes (such as System.out.println lines) may make the problem disappear.

2) Iterate to get the problem to occur consistently.

 . -Xjit:count=0 will force JIT compilation from the first iteration. This will often make
an intermittent JIT issue fail reliably.  If this does not work try the various optimization
  . -Xjit:count=0,optLevel=scorching
  . -Xjit:count=0,optLevel=veryHot
  . -Xjit:count=0,optLevel=hot
  . -Xjit:count=0,optLevel=warm
  . -Xjit:count=0,optLevel=cold
  . -Xjit:count=0,optLevel=noOpt
  . java -Xjit:count=0,optLevel=scorching org.apache.derby.tools.ij repro.sql
 . or
  . java -Xjit:count=0 org.apache.derby.tools.ij repro.sql

3) Locate the failing method.

 . Once you have a reliably failing test case you can locate the failing method with a binary
search through the JIT's log file.

 . a) Obtain the method compilation levels.
  . Run the small test case by adding  the 'verbose' and 'vlog' options ('-Xjit:count=0.verbose.vlog=<filename>')
to the command line. This will list the compiled methods in a file named  <filename>.<date>.<time>.<pid>
   . java -Xjit:count=0,verbose,vlog=vlog.out org.apache.derby.tools.ij repro.sql
  . will create a file called vlog.out.20131015.104401.6840.

 . b) Iterate through the vlog file.
  . First identify how many lines (methods) there are in the file resulting from the run with
'verbose.vlog' (For instance, open it in an editor that has a line count, or use command line
options like shell's 'wc' command). Then run the program again with the JIT parameter 'limitFile=(<filename>,<m>,<n>)',
where <filename> is the path to the vlog result file, and <m> and <n> are
line numbers indicating the first and the last methods in the 'limit' file that should be
compiled. Using this approach, you can perform a binary search to find the line identifying
the failing method. For example, for a 1000 line file you might run with
   . java -Xjit:count=0,limitfile=\(vlog.out,1,500\) org.apache.derby.tools.ij repro.sql
  Repeat, depending on whether this reproduces the problem, switch to the other half, or narrow
down further. The iterations might look something like this:
   . java -Xjit:count=0,limitfile=\(vlog.out,1,500\) org.apache.derby.tools.ij repro.sql 
-> reproduces
   . java -Xjit:count=0,limitfile=\(vlog.out,1,250\) org.apache.derby.tools.ij repro.sql 
-> not
   . java -Xjit:count=0,limitfile=\(vlog.out,250,500\) org.apache.derby.tools.ij repro.sql
 -> reproduces
   . ..
   java -Xjit:count=0,limitfile=\(vlog.out,250,258\) org.apache.derby.tools.ij repro.sql 
-> reproduces
   . java -Xjit:count=0,limitfile=\(vlog.out,250,257\) org.apache.derby.tools.ij repro.sql
 -> not
   . java -Xjit:count=0,limitfile=\(vlog.out,258,258\) org.apache.derby.tools.ij repro.sql
 -> reproduces

  . Open the file to find the problematic method, for instance:
   . :258: (warm) java/lang/Math.max(II)I @ 067FE280-067FE2A5 JavaOrdinaryMethod Q_SZ=0 Q_SZI=0
QW=6 bcsz=11 sync compThread=0 OSR=0 CpuLoad=0%(0%avg) JvmCpu=0%

4) report the issue.

 . You should report the problem to the JVM vendor using your support channels. Assuming it
is a problem that shows up using Derby, you can additionally log an issue in JIRA, identifying
the full details of the JVM and Derby version. This will help other users running into the
same issue. However, the Derby community will not report a JVM issue for you, and the issue
will be closed as 'Invalid' in JIRA.

5) Work around the issue.

 . Once you have located the failing method, you can work around the issue. One way is to
use the JVM's tooling by either telling the JIT to compile the method at a lower optimization
  . java -Xjit:{java/lang/Math.max(II)I}(optLevel=cold,count=0) org.apache.derby.tools.ij
 or excluding just this method from JIT compilation, e.g.
  . java -Xjit:exclude={java/lang/Math.max(II)I} org.apache.derby.tools.ij repro.sql

View raw message