Return-Path: 
 <user-return-27891-apmail-commons-user-archive=commons.apache.org@commons.apache.org>
X-Original-To: apmail-commons-user-archive@www.apache.org
Delivered-To: apmail-commons-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BDA2DBAA
	for <apmail-commons-user-archive@www.apache.org>;
 Fri, 26 Oct 2012 23:26:34 +0000 (UTC)
Received: (qmail 18076 invoked by uid 500); 26 Oct 2012 23:26:32 -0000
Delivered-To: apmail-commons-user-archive@commons.apache.org
Received: (qmail 17677 invoked by uid 500); 26 Oct 2012 23:26:32 -0000
Mailing-List: contact user-help@commons.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@commons.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@commons.apache.org>
List-Post: <mailto:user@commons.apache.org>
List-Id: <user.commons.apache.org>
Reply-To: "Commons Users List" <user@commons.apache.org>
Delivered-To: mailing list user@commons.apache.org
Received: (qmail 17560 invoked by uid 99); 26 Oct 2012 23:26:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Oct 2012 23:26:32 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sebbaz@gmail.com designates
 209.85.212.43 as permitted sender)
Received: from [209.85.212.43] (HELO mail-vb0-f43.google.com) (209.85.212.43)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Oct 2012 23:26:25 +0000
Received: by mail-vb0-f43.google.com with SMTP id fq11so3327195vbb.30
        for <multiple recipients>; Fri, 26 Oct 2012 16:26:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        bh=3Gip5tAbCq6gaTS9WuN3yeP3BBshkFJMTU1lpCBDNLE=;
        b=vRu0H1gZczhBm+bBasInGQxyArbcUfQtjyehnxt8rjCREntFc7aP85yNpuRwFMyfAe
         1dpMFIwtdVyZB40YkZ9YVza9bOfOVGftsoTjb1vzliNG7kX9+NcYPCZjs7ZRmYCURJAX
         45C/+klAwqZj6CpkFkhGoSNST9VQyL/dd7WA8DQlAyRGKpnOTkTlLFlMMBlypBPxXMU0
         55Kx+DkxyjquFcknqJ6vOtARQwAuGlWkKIlxE/gKhG8Q1VAO4sIVsWjWb2UT+ZzrXFzi
         SKTMgWOEEiXkaSako0s5RUmmaqh//foIxzMf75audsQ2jDyTWW75R3xJoC7J2/429cM5
         BSoQ==
MIME-Version: 1.0
Received: by 10.220.150.134 with SMTP id y6mr21099503vcv.20.1351293964178;
 Fri, 26 Oct 2012 16:26:04 -0700 (PDT)
Received: by 10.58.172.71 with HTTP; Fri, 26 Oct 2012 16:26:04 -0700 (PDT)
In-Reply-To: 
 <CAN7udUKOVpXBksyqx6EM9yuLy0NKZAs45tjZCxOYAuoWba=eRg@mail.gmail.com>
References: <CCB055E1.14E99%liyu_yi@symantec.com>
	<CAN7udUKOVpXBksyqx6EM9yuLy0NKZAs45tjZCxOYAuoWba=eRg@mail.gmail.com>
Date: Sat, 27 Oct 2012 00:26:04 +0100
Message-ID: 
 <CAOGo0VbN4D1Opo2hDuZJ=kqHs9DwHmnXDJ0PgUme27vVkgDz6Q@mail.gmail.com>
Subject: Re: Commons IO Tailer does not respect UTF-8 Charset
From: sebb <sebbaz@gmail.com>
To: Commons Developers List <dev@commons.apache.org>
Cc: user@commons.apache.org, Liyu Yi <liyuyi@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On 26 October 2012 23:03, Liyu Yi <liyuyi@gmail.com> wrote:
> I just realized there is a defect in the source code of
> "org.apache.commons.io.input.Tailer.java".

Thanks for the feedback.

Bug fixes are better provided as JIRA bugs.

It's hard to keep track of bug reports and patches when they are mixed
in with all the other mailing list traffic.

Could you create a JIRA issue for this problem please?

Also, in future, please do not cross-post to both the developer and user lists.
The developers follow the user list.

> Basically, the current
> implementation does not work for multi-byte encoded files. See the
> following snippet,
>
> 448 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#448>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>     private long
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>readLines(RandomAccessFile
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/RandomAccessFile.java#RandomAccessFile>
> reader) throws IOException
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/IOException.java#IOException>
> {
>
> 449 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#449>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         StringBuilder
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/StringBuilder.java#StringBuilder>
> sb = new StringBuilder
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/StringBuilder.java#StringBuilder>();
>
> 450 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#450>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
> 451 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#451>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         long pos = reader.getFilePointer
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/RandomAccessFile.java#RandomAccessFile.getFilePointer%28%29>();
>
> 452 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#452>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         long rePos = pos; // position to re-read
>
> 453 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#453>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
> 454 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#454>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         int num;
>
> 455 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#455>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         boolean seenCR = false;
>
> 456 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#456>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         while (run
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#Tailer.0run>
> && ((num = reader.read
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/RandomAccessFile.java#RandomAccessFile.read%28byte%5B%5D%29>(inbuf
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#Tailer.0inbuf>))
> != -1)) {
>
> 457 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#457>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>             for (int i = 0; i < num; i++) {
>
> 458 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#458>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                 byte ch = inbuf
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#Tailer.0inbuf>[i];
>
> 459 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#459>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                 switch (ch) {
>
> 460 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#460>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                 case '\n':
>
> 461 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#461>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     seenCR = false; // swallow CR before LF
>
> 462 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#462>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     listener
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#Tailer.0listener>.handle
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/TailerListener.java#TailerListener.handle%28java.lang.String%29>(sb.toString
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/StringBuilder.java#StringBuilder.toString%28%29>());
>
> 463 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#463>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     sb.setLength
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/AbstractStringBuilder.java#AbstractStringBuilder.setLength%28int%29>(0);
>
> 464 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#464>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     rePos = pos + i + 1;
>
> 465 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#465>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     break;
>
> 466 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#466>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                 case '\r':
>
> 467 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#467>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     if (seenCR) {
>
> 468 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#468>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                         sb.append
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/StringBuilder.java#StringBuilder.append%28char%29>('\r');
>
> 469 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#469>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     }
>
> 470 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#470>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     seenCR = true;
>
> 471 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#471>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     break;
>
> 472 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#472>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                 default:
>
> 473 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#473>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     if (seenCR) {
>
> 474 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#474>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                         seenCR = false; // swallow final CR
>
> 475 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#475>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                         listener
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#Tailer.0listener>.handle
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/TailerListener.java#TailerListener.handle%28java.lang.String%29>(sb.toString
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/StringBuilder.java#StringBuilder.toString%28%29>());
>
> 476 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#476>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                         sb.setLength
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/AbstractStringBuilder.java#AbstractStringBuilder.setLength%28int%29>(0);
>
> 477 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#477>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                         rePos = pos + i + 1;
>
> 478 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#478>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     }
>
> 479 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#479>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                     sb.append
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/lang/StringBuilder.java#StringBuilder.append%28char%29>((char)
> ch); // add character, not its ascii value
>
> 480 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#480>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>                 }
>
> 481 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#481>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>             }
>
> 482 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#482>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
> 483 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#483>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>             pos = reader.getFilePointer
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/RandomAccessFile.java#RandomAccessFile.getFilePointer%28%29>();
>
> 484 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#484>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         }
>
> 485 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#485>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
> 486 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#486>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         reader.seek
> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/io/RandomAccessFile.java#RandomAccessFile.seek%28long%29>(rePos);
> // Ensure we can re-read if necessary
>
> 487 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#487>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>         return rePos;
>
> 488 <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#488>
>
> <http://grepcode.com/file/repo1.maven.org/maven2/commons-io/commons-io/2.4/org/apache/commons/io/input/Tailer.java#>
>
>     }
>
>
> At line 479, the conversion of byte to char types breaks the encoding.
>
> I used a "hacky" fix to reconstruct the String with right encoding in the
> handler class.
>
> private String rebuildUTF8String(String line) {
>
> int len = line.length();
>
> byte[] bytes = new byte[len];
>
> for (int i=0; i<len; i++) {
>
> bytes[i] = (byte)line.charAt(i);
>
> }
>
> return new String(bytes, UTF8);
>
> }
>
> However, the right approach is to pass in the encoding to the "create"
> method and handle it in the Tailer.
>
> Regards,
>
> Liyu Yi

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org