Mailing-List: contact dev-help@jmeter.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@jmeter.apache.org
Received-SPF: pass (nike.apache.org: domain of philippe.mouawad@gmail.com
 designates 209.85.223.180 as permitted sender)
MIME-Version: 1.0
Date: Sun, 16 Nov 2014 14:32:55 +0100
Message-ID: 
 <CAH9fUpYV1SJBvWZkKhnoHUX_n7EZBaSAczzDQK3yX8FQDm-SHg@mail.gmail.com>
Subject: Graphs and reporting in Apache JMeter : Today and Future
From: Philippe Mouawad <philippe.mouawad@gmail.com>
To: "dev@jmeter.apache.org" <dev@jmeter.apache.org>
Content-Type: multipart/alternative; boundary=089e0149c0284f7cce0507f9e697

--089e0149c0284f7cce0507f9e697
Content-Type: text/plain; charset=ISO-8859-1

Hello,
I would like to start a discussion about the future of graphs and reporting
in Apache JMeter.

Todays state:

   - My feeling (I don't want to hurt anybody who worked on those listeners
   which are interesting and I am aware that what can be great at some time
   can become less few years after) is that today:
      -  we have the following graphs
      - Aggregate Graph : currently affected by bug on
         https://issues.apache.org/bugzilla/show_bug.cgi?id=57221 in Java 8
         - Response Time Graph : interesting but slow to generate and I get
         OOM when I try to reload an existing CSV in some configuration, I will
         report a bug
         - Distribution Graph (alpha) : Alpha is not encouraging for users,
         is Alpha really needed or is it stable
         - Graph Results : Does not look very nice compared with what you
         can get today
         -  we have the following reports:
         - Aggregate Report : Interesting
         - Summary Report: Differs from  Aggregate Report by having
         additional Avg Bytes and Std Dev but not Median nor 90%Line

My opinion and analysis on the current state is the following:

   - We lack of clean reporting that would be generated automatically at
   end of load test, all the previous graphs need to be generated from GUI and
   makes you lose some time. Also as we discourage Load Testing from GUI we
   should propose a way to generate reporting from NON GUI Mode
   - Some graphs (Response Time Graph, Aggregate Graph) rely on JCharts
   which seems to be abandoned, last release is from 2004 and current bug
   57221 may be related to a bug in the library ,by the way this is an
   additional argument to get rid of obsolete libraries in JMeter:
      - Avalon (Logging + DataSource)
      - Excalibur (logging + DataSource)
      - jCharts
      - Reports are not sufficient and partly redundant , for example it
   seems that in terms of information Aggregate report and Summary Report
   should be merged. I looked at their code they are really very similar , I
   think one should be removed and the  infos in both merged


I must reckon that today my colleagues and I for example , we all use
JMeter-Plugins graphs combined with GraphGeneratorListener, and we are very
happy with what we get:

   -
   http://www.ubik-ingenierie.com/blog/automatically-generating-nice-graphs-at-end-of-your-load-test-with-apache-jmeter-and-jmeter-plugins/

I would be interested to know how users generate their reports and graphs
currently.


But some may object that JMeter-Plugins is not JMeter and conclude that
JMeter lacks of an essential feature.

So I have 2 propositions to improve JMeter in this field:

   - 1/ Create a Brand new Report Generator Listener that would at end of
   load test generate a report in HTML or PDF containing:
      - Merge of informations contained in current reports
      - Generate graphs based on JS Libraries, I am impressed by the number
      of libraries that generate nice and sexy graphs:
         - http://www.chartjs.org/
         - https://github.com/keen/dashboards/tree/gh-pages/examples
      - Having them in HTML would allow some dynamice behaviour but the
      ideal result would be to be able to export graphs also as images
   - 2/ I don't know if it is possible but what about merging
   JMeter-Plugins graphing suite + GraphGeneratorListener in JMeter core if no
   library is in conflict in terms of licensing, of course it would required
   their agreement :-)


I think today any Load Testing tool comes with  reporting feature wether
Commercial and Open Source and JMeter as such should improve in this field
to stay one of the leaders in Load Testing field.

Thoughts ?

-- 
Regards
Philippe M.
@philmdot

--089e0149c0284f7cce0507f9e697--