manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brad Dennis <>
Subject RE: Webconnector: Comparison operator '<' in the body of a script tag
Date Wed, 24 Jun 2015 14:49:10 GMT

Thank you for investigating the issue.  My concern is that I expect it's fairly common to
use '<' in embedded, uncommented, Javascript and this bug excludes any content that appears
after one and before a second end script tag from being crawled with ManifoldCF.  Unfortunately,
I don't have any suggestions other than using a stack to push open tags onto and pop off when
an end tag is seen.  I believe that would satisfy your example, but who knows what other problems
a stack brings.

Do you have any suggestions for work arounds I could implement locally?


-----Original Message-----
From: Karl Wright [] 
Sent: Wednesday, June 24, 2015 9:33 AM
To: dev
Subject: Re: Webconnector: Comparison operator '<' in the body of a script tag


The issue is complex because according to spec the code is doing the right thing.  Typically,
<script> blocks look something like this:

<script ...>



The reason for the comment area is because without it, tags within the script block are supposed
to be recognized as such, even if they are ignored.  Within comments, this does not happen,
of course, which is why comments are used.

I don't believe it is a real standard, but some browsers try to interpret script blocks differently
even when no comment is given.  We can try to emulate that behavior but it is likely that
our emulation will not work for all web pages, since it's not a standard.  Exploring how this
works on various browsers would be the first step.  Specifically, if you do something like

<script ...>

foo = "<script></script>";
bar = "hello";


... what happens?  Does the script end at the first </script>, or the second?  And,
in what browsers?

Until we get more clarity it's going to be hard to do a feature that actually helps rather
than hurts...


On Wed, Jun 24, 2015 at 10:05 AM, Karl Wright <> wrote:

> Hi Brad,
> I've created a ticket: CONNECTORS-1215.  Looking into this now.
> Karl
> On Wed, Jun 24, 2015 at 9:45 AM, Brad Dennis 
> <
> > wrote:
>> Hi,
>> There appears to be a bug in the TagParseState when the comparison 
>> operator '<'  is encountered in the body of  a script tag.  It 
>> appears to get flagged as an open tag and then the next '</' closes 
>> it.  In my case, the next '</' is the script tag.  The 
>> ScriptParseState chomps everything until it encounters a second </script> tag.
>> A live link that demonstrates this bug is here:
>> -days-page-1-pagesize-20
>> The '<' near line 2826 in the script body that begins near   line 2759
>> begins a new tag 'arraykeywords.length' which gets closed by the '</' 
>> in the closing script tag.  The ScriptParseState chomps all the html 
>> until it sees the end script tag near line 3385.
>> At the moment, I'm not sure of a solution other than pushing the 
>> script tag handling up to the TagParseState and treating it like CDATA is.
>> Thanks,
>> Brad Dennis
View raw message