creadur-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sebb (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (RAT-138) RAT runs very slowly on some input
Date Fri, 17 May 2013 23:07:15 GMT

    [ https://issues.apache.org/jira/browse/RAT-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13661134#comment-13661134
] 

Sebb commented on RAT-138:
--------------------------

Actually, the issue seems to be multi-line matching.

The OASIS license matcher looks for the initial copyright line, and only then tries using
the full text matcher.

If the copyright line is not present in a file, it's very quick to scan.
However, if the OASIS copyright line is present, but the rest of the license is not, then
the matching is very slow, even in 0.8.

It helps to use String.contains rather than regexes, but that does not entirely solve the
slowness.
The technique used in FullTextMatchingLicense is clearly not very efficient.

I think the problem is that the text buffer keeps getting bigger even if there is no chance
of a match for the first part of the license, so matching gets more and more expensive. The
OASIS code avoids this for valid licences, but allows the initial Copyright to be far away
from the rest of the license (which is why it is slow if that is missing)

If the initial line is not immediately followed by the rest of the license, then it should
be possible to restart looking for the first line. However the entire buffer cannot be discarded
without first checking whether the first line has been seen again already. This is unlikely,
but possible.
                
> RAT runs very slowly on some input
> ----------------------------------
>
>                 Key: RAT-138
>                 URL: https://issues.apache.org/jira/browse/RAT-138
>             Project: Apache Rat
>          Issue Type: Bug
>          Components: engine
>    Affects Versions: 0.9
>            Reporter: Sebb
>              Labels: perfomance, regression
>
> Commons IO discovered that mvn site was spending a lot of time running RAT 0.9. Reverting
to 0.8 fixes the problem.
> Turns out that certain files seem to cause RAT to chew CPU in the Pattern.matcher; here
is a sample stack trace from a thread dump:
> "main" prio=6 tid=0x003c8c00 nid=0xa4c runnable [0x00a5e000]
>    java.lang.Thread.State: RUNNABLE
>         at java.util.regex.Pattern$Curly.match0(Pattern.java:4166)
>         at java.util.regex.Pattern$Curly.match(Pattern.java:4132)
>         at java.util.regex.Matcher.match(Matcher.java:1221)
>         at java.util.regex.Matcher.matches(Matcher.java:559)
>         at org.apache.rat.analysis.license.FullTextMatchingLicense.match(FullTextMatchingLicense.java:79)
>         at org.apache.rat.analysis.util.HeaderMatcherMultiplexer.match(HeaderMatcherMultiplexer.java:42)
>         at org.apache.rat.analysis.HeaderCheckWorker.readLine(HeaderCheckWorker.java:113)
>         at org.apache.rat.analysis.HeaderCheckWorker.read(HeaderCheckWorker.java:84)
>         at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:43)
>         at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60)
>         at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:37)
>         at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42)
>         at org.apache.rat.mp.FilesReportable.run(FilesReportable.java:68)
>         at org.apache.rat.Report.report(Report.java:393)
>         at org.apache.rat.Report.report(Report.java:373)
>         at org.apache.rat.mp.AbstractRatMojo.createReport(AbstractRatMojo.java:462)
>         at org.apache.rat.mp.RatReportMojo.createReport(RatReportMojo.java:148)
>         at org.apache.rat.mp.RatReportMojo.generate(RatReportMojo.java:310)
>         at org.apache.rat.mp.RatReportMojo.execute(RatReportMojo.java:210)
> I assume there must be a problem with one of the REs which is triggering lots of backtracking
when applied to files under site-content/, which is a working copy of:
> https://svn.apache.org/repos/infra/websites/production/commons/content/proper/commons-io
> Last Changed Rev: 861378
> This directory should not have been included in the RAT scan, but the files don't cause
problems for RAT 0.8.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message