tika-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Allison (Jira)" <j...@apache.org>
Subject [jira] [Commented] (TIKA-3048) Tika unable to parse html files with non UTF-8 charset
Date Wed, 26 Feb 2020 13:35:00 GMT

    [ https://issues.apache.org/jira/browse/TIKA-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045541#comment-17045541
] 

Tim Allison commented on TIKA-3048:
-----------------------------------

[~akki1607], my proposal isn't really a fix.  It happens to work on the documents you have
but it isn't a fix by any stretch of the imagination.

To my mind, the only way to handle conflicting information or the case of a wrong http-meta
header is to run content extraction with both/all three charsets and then see which extracted
text is more language-y, e.g. with tika-eval's out-of-vocabulary statistic.  If you take a
look at the report I referenced above, that's where I'd like to head eventually on Tika, but
I have no funding to work on that at this point.

> Tika unable to parse html files with non UTF-8 charset
> ------------------------------------------------------
>
>                 Key: TIKA-3048
>                 URL: https://issues.apache.org/jira/browse/TIKA-3048
>             Project: Tika
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.9
>            Reporter: Akash
>            Priority: Major
>             Fix For: 1.9
>
>         Attachments: ChineseFile.html
>
>
> Tika is returning junk characters when parsing chinese characters present inside html
file. Html file have charset mentioned as GB2312 explicitly.
> <head><meta http-equiv=Content-Type content="text/html; charset=gb2312"><meta
name=Generator content="Microsoft Word 15 (filtered medium)">
>  
> If we remove this charset from the html meta tag, then parsing works fine.
>  
> Similar issue is observed for Arabic, Russain, Korean, Japanese, Hungarian and Spanish
languages.
> Charset mentioned for each languages - 
> Hungarian - iso-8859-1
> Chinese - gb2312
> Spanish - iso-8859-1
> Russian - koi8-r
> Korean - ks_c_5601-1987
> Japanese - iso-2022-jp
> Arabic - windows-1256



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message