Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Thu, 9 Jun 2016 16:41:21 +0000 (UTC)
From: "Adrien Grand (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.12962536.1461663216000.41378.1465490481067@Atlassian.JIRA>
In-Reply-To: <JIRA.12962536.1461663216000@Atlassian.JIRA>
References: <JIRA.12962536.1461663216000@Atlassian.JIRA> <JIRA.12962536.1461663216141@arcas>
Subject: [jira] [Closed] (LUCENE-7256) PatternReplaceCharFilter can make
 Lucene hang
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 09 Jun 2016 16:41:23 -0000


     [ https://issues.apache.org/jira/browse/LUCENE-7256?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Adrien Grand closed LUCENE-7256.
--------------------------------
    Resolution: Won't Fix

Closing then.

> PatternReplaceCharFilter can make Lucene hang
> ---------------------------------------------
>
>                 Key: LUCENE-7256
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7256
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: modules/analysis
>    Affects Versions: 5.4.1
>         Environment: alpine linux v3.3
>            Reporter: Tom Fotherby
>            Priority: Minor
>
> I'm using ElasticSearch (v2.2.0 , Lucene v5.4.1) and it's [Pattern Replac=
e Char Filter|https://www.elastic.co/guide/en/elasticsearch/reference/curre=
nt/analysis-pattern-replace-charfilter.html] (Lucenes PatternReplaceCharFil=
ter) . I need to filter out urls from my query text before it is tokenised.=
 But I found that some input strings cause ElasticSearch to "hang" (slowly =
eating more CPU and memory) until the system crashes.
> ----
> *Example*
> {code}
> // Character filters are used to "tidy up" a string *before* it is tokeni=
zed.
> 'char_filter' =3D> [
>     'url_removal_pattern' =3D> [
>         'type'        =3D> 'pattern_replace',
>         'pattern'     =3D> '(?mi)\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|=
www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^=
\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".=
,<>?=C2=AB=C2=BB""'']))',
>         'replacement' =3D> '',
>     ],
> {code}
> This filter was working fine for some weeks until suddenly ElasticSearch =
started crashing. We found someone was trying to do a javascript injection =
attack in our search box.
> I pasted the regex and the attack string into https://regex101.com=20
> * Regexp:=20
>  * {code}(?mi)\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0=
-9.\-]+[.][a-z]{2,4}\/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?=
:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s!()\[\]{};:\'".,<>?=C2=AB=C2=BB""''=
])){code}
> * Test string:=20
>  * {code}twitter.com/widgets.js\";fjs.parentNode.insertBefore(js,fjs);}}(=
document,\"script\",\"twitter-wjs\"{code}
> https://regex101.com shows the problem to be "Catastrophic backtracking"
> bq. Catastrophic backtracking has been detected and the execution of your=
 expression has been halted. To find out more what this is, please read the=
 following article: [Runaway Regular Expressions|http://www.regular-express=
ions.info/catastrophic.html].
> It would be great if Lucene could detect "Catastrophic backtracking" and =
throw a error or return null.
> ----
> As an aside, I created a unit test for our PHP application that uses the =
same regexp and test string. (PHP can understand the same regexp, even thou=
gh it's obviously for Java in the ElasticSearch case) . Interestingly in ph=
p, the regex results in `null` which is the documented response of [preg_re=
place|http://php.net/manual/en/function.preg-replace.php] when a error occu=
rs. If PHP can return a error rather than crashing - surely Lucene / Java c=
an too :trollface: ?
> {code}
> namespace app\tests\unit;
> use \yii\codeception\TestCase;
> class TagsControllerTest extends TestCase
> {
>     public function testRegexForURLDetection()
>     {
>         $regex =3D '(?mi)\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,=
3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+=
\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:\'".,<>?=C2=
=AB=C2=BB""'']))';
>         // Test the Catastrophic backtracking problem
>         $testString =3D "twitter.com/widgets.js\";fjs.parentNode.insertBe=
fore(js,fjs);}}(document,\"script\",\"twitter-wjs\"";
>         // This shows the regex is not working for our test string - it g=
ives null but should give 'hello '
>         $this->assertEquals(null, preg_replace("/$regex/", '', "hello $te=
stString"));
>     }
> }
> {code}
> ----
> (I originally [opened a ticket|https://github.com/elastic/elasticsearch/i=
ssues/17934] to the ElasticSearch project but got told opening it here woul=
d be more appropriate - sorry if I'm wrong)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org