nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Fellows (JIRA)" <j...@apache.org>
Subject [jira] Commented: (NUTCH-18) Windows servers include illegal characters in URLs
Date Wed, 26 Apr 2006 21:22:24 GMT
    [ http://issues.apache.org/jira/browse/NUTCH-18?page=comments#action_12376554 ] 

Chris Fellows commented on NUTCH-18:
------------------------------------

Was looking into nutch-18 bug that revolves around
illegal, non-ascii, characters in a url. An example of
a high bit character is 'ç' that maps to a high bit
set.

Before applying any fix, did a brief test with 0.8
trunk. After fetching and indexing on
http://en.wiktionary.org/wiki/ç, I was able to search
on ç and got the following result off the browser
results:

ç - Wiktionary
... Letter [ edit ] Translingual [ edit ] Letter Ç , ç
C with a cedilla ... visit IRC or Wiktionary:AOL . ç
...
http://en.wiktionary.org/wiki/%C3%A7 (cached)
(explain) (anchors) (more from en.wiktionary.org) 

So it looks like it will fetch and parse off of urls
with high bit set characters. Additionally, the
display url has the ç encoded correctly as %C3%A7. 

Is this really a bug?

Doing a similar test off of Google on keywords: "ç"
wiktionary. yields:

ç - Wiktionary
AOL users can access Wiktionary through this link
after accepting the CACERT certificate. ... Ç, ç.
"tʃə", the fourth letter of the Albanian
alphabet. ...
en.wiktionary.org/wiki/ç - 15k - Cached - Similar
pages
[ More results from en.wiktionary.org ]

Nearly identical, but the see that the ç is in its
decoded form, not %C3%A7.

I'd say if anything, the bug is that the display urls are in encoded form and not human readable.

> Windows servers include illegal characters in URLs
> --------------------------------------------------
>
>          Key: NUTCH-18
>          URL: http://issues.apache.org/jira/browse/NUTCH-18
>      Project: Nutch
>         Type: Bug

>   Components: fetcher
>     Reporter: Stefan Groschupf
>     Priority: Minor

>
> Transfered from:
> http://sourceforge.net/tracker/index.php?func=detail&aid=1110243&group_id=59548&atid=491356
> submitted by:
> Ken Meltsner
> While spidering our intranet, I found that IIS may include 
> illegal characters in URLs -- specifically, characters with 
> the high bit set to produce non-English letters. In 
> addition, both Firefox and IE will accept URLs with high-
> bit characters, but Java won't.
> While this may not be Nutch's (or Java's) fault, it would 
> help if high-bit characters (and other illegal characters) 
> in URLs could be escaped (using percent-hex notation) 
> as part of the URL fix-up process, probably right after 
> the hostname lower-case conversion.
> Example document name in Portuguese(with high-bit 
> characters) taken from a longer URL:
> Nota%20tecnica%20-%20Alteração%20de%
> 20escopo.doc
> and with percent-escaped characters:
> Nota%20tecnica%20-%20Altera%e7%e3o%20de%
> 20escopo.doc

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message