Mailing-List: contact dev-help@mahout.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mahout.apache.org
Message-ID: <14915710.229041296088305089.JavaMail.jira@thor>
Date: Wed, 26 Jan 2011 19:31:45 -0500 (EST)
From: "Shige Takeda (JIRA)" <jira@apache.org>
To: dev@mahout.apache.org
Subject: [jira] Commented: (MAHOUT-594) FileWriter may garble non-ASCII
 output if the environment variable LANG/LC_ALL is not appropriate.
In-Reply-To: <4498189.182051295943103531.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/MAHOUT-594?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1298=
7312#action_12987312 ]=20

Shige Takeda commented on MAHOUT-594:
-------------------------------------

no problem as long as this issue is addressed. thanks!

> FileWriter may garble non-ASCII output if the environment variable LANG/L=
C_ALL is not appropriate.
> -------------------------------------------------------------------------=
-------------------------
>
>                 Key: MAHOUT-594
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-594
>             Project: Mahout
>          Issue Type: Bug
>          Components: Utils
>    Affects Versions: 0.4
>         Environment: RHL Linux 2.6.18
>            Reporter: Shige Takeda
>            Assignee: Sean Owen
>            Priority: Minor
>             Fix For: 0.5
>
>         Attachments: 0001-set-file-reader-and-writer-character-encoding-t=
o-utf.patch
>
>
> For non-ASCII output data, java.io.FileWriter should be replaced with jav=
a.io.OutputStreamWriter in UTF-8.
> For example, if you dump centroids of clusters using ClusterDumper, you m=
ay get the following output:
> {noformat}
> ...
> C-0{n=3D2 c=3D[brown:2.099, c?t:1.957, dogs:1.916, fox:0.652, jumped:2.09=
9, l?zy:1.884, over:2.099, quick:2.099, red:1.916, ?:0.871, ?:0.871, ?:0.87=
1, ?:0.871] r=3D[c?t:0.652, fox:0.652, l?zy:1.131, ?:0.871, ?:0.871, ?:0.87=
1, ?:0.871]}
>     Top Terms:
>         quick                                   =3D>  2.0986123085021973
>         over                                    =3D>  2.0986123085021973
>         jumped                                  =3D>  2.0986123085021973
>         brown                                   =3D>  2.0986123085021973
>         c?t                                     =3D>   1.957078456878662
>         red                                     =3D>  1.9162907600402832
>         dogs                                    =3D>  1.9162907600402832
>         l?zy                                    =3D>  1.8843144178390503
>         ?                                       =3D>  0.8706584572792053
>         ?                                       =3D>  0.8706584572792053
>     Weight:  Point:
>     1.0: P(0) =3D [brown:2.099, dogs:1.916, fox:2.609, jumped:2.099, over=
:2.099, quick:2.099, red:1.916, ?:2.322, ?:2.322, ?:2.322, ?:2.322]
>     1.0: P(1) =3D [brown:2.099, dogs:1.916, fox:2.609, jumped:2.099, over=
:2.099, quick:2.099, red:1.916, ?:2.322, ?:2.322, ?:2.322, ?:2.322]
>     1.0: P(2) =3D [brown:2.099, c?t:2.609, dogs:1.916, jumped:2.099, over=
:2.099, quick:2.099, red:1.916, ?:2.322, ?:2.322, ?:2.322, ?:2.322]
> ...
> {noformat}
> where "?" characters were garbled by FileWriter. NOTE: this test case is =
a tweaked version of TestClusterDumper. E.g., lazy =3D> l=C3=A4zy
> The cause of this is the line in ClusterDumper.java:
> {code}
> Writer writer =3D this.outputFile =3D=3D null ? new OutputStreamWriter(Sy=
stem.out) : new FileWriter(this.outputFile);
> {code}
> This can be around by setting the environment variables LC_ALL/LANG to en=
_US.UTF-8, but many environments have LC_ALL/LANG=3DC by default, and in so=
me cases, you even may not have choices but C for various reasons.
> To address this issue, I would like to propose to hard code the output en=
coding to UTF-8 as follows:
> {code}
> Writer writer =3D this.outputFile =3D=3D null ? new OutputStreamWriter(Sy=
stem.out) : new OutputStreamWriter(new FileInputStream(this.outputFile), UT=
F8);
> {code}
> This way, the output file encoding will not be affected by environments.
> And if this proposal is agreed, a similar fix should be applied to the fo=
llowing files:
> - ./core/src/main/java/org/apache/mahout/classifier/sgd/ModelSerializer.j=
ava
> - ./core/src/test/java/org/apache/mahout/fpm/pfpgrowth/PFPGrowthTest.java
> - ./examples/src/main/java/org/apache/mahout/classifier/sgd/TrainLogistic=
.java
> - ./examples/src/main/java/org/apache/mahout/clustering/display/DisplaySp=
ectralKMeans.java
> - ./utils/src/main/java/org/apache/mahout/clustering/lda/LDAPrintTopics.j=
ava
> - ./utils/src/main/java/org/apache/mahout/utils/SequenceFileDumper.java
> - ./utils/src/main/java/org/apache/mahout/utils/clustering/ClusterDumper.=
java
> - ./utils/src/main/java/org/apache/mahout/utils/vectors/VectorDumper.java
> - ./utils/src/main/java/org/apache/mahout/utils/vectors/arff/Driver.java
> - ./utils/src/main/java/org/apache/mahout/utils/vectors/lucene/ClusterLab=
els.java
> - ./utils/src/main/java/org/apache/mahout/utils/vectors/lucene/Driver.jav=
a
> Hope not many folks prefer ISO-8859-1 or other 'legacy' character sets.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.