harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Beyer <nbe...@gmail.com>
Subject Re: Shall we change our file.encoding
Date Fri, 17 Jul 2009 03:17:05 GMT
On Thu, Jul 16, 2009 at 9:30 PM, Charles Lee<littlee1032@gmail.com> wrote:
> Thanks Nathan!
>
> I will try this :-)

Where do we define the user's locale and system locale? It seems like
all of this should be located there and associated with that process.

>
> On Fri, Jul 17, 2009 at 10:05 AM, Nathan Beyer <ndbeyer@apache.org> wrote:
>
>> On Thu, Jul 16, 2009 at 8:50 PM, Nathan Beyer<ndbeyer@apache.org> wrote:
>> > On Thu, Jul 16, 2009 at 8:35 PM, Nathan Beyer<ndbeyer@apache.org> wrote:
>> >> On Thu, Jul 16, 2009 at 8:26 PM, Nathan Beyer<ndbeyer@apache.org>
>> wrote:
>> >>> On Thu, Jul 16, 2009 at 8:18 PM, Charles Lee<littlee1032@gmail.com>
>> wrote:
>> >>>> Hi Nathan,
>> >>>>
>> >>>> What I got is 936, the code page identifier. Is there a api for
us to
>> map
>> >>>> 936 to the gb2312?
>> >>>
>> >>> Oh, the 'identifier' bit was missing - yeah, we'll need to translate
>> >>> that into a name of some sort. I'll poke around a bit and see what I
>> >>> can find.
>> >>
>> >> We'll probably just have to put in a mapping ourselves based on the
>> >> documentation. We'd call GetACP [1] and map that to a known alias in
>> >> java.nio.charset that matches the definitions[2] of the identifiers.
>> >>
>> >> [1] http://msdn.microsoft.com/en-us/library/dd318070%28VS.85%29.aspx
>> >> [2] http://msdn.microsoft.com/en-us/library/dd317756%28VS.85%29.aspx
>> >
>> > This may be better - APR has a function for getting the OS default
>> > encoding. This would work across all platforms that APR supports and I
>> > believe we already use APR.
>> >
>> >
>> http://apr.apache.org/docs/apr/1.3/group__apr__portabile.html#g6e21845a4a5f3b7dd107b2beea50c91e
>>
>> However, the Windows version of this is simply - return
>> apr_psprintf(pool, "CP%u", (unsigned) GetACP());. Which is essentially
>> "CP" + codePageId.
>>
>> And the Unix version of this method doesn't look very good for our
>> purposes.
>> >
>> > -Nathan
>> >>
>> >>>
>> >>>> If we put 936 in the file.encoding, can we successfully get the
>> encoder and
>> >>>> decoder by charset?
>> >>>>
>> >>>> On Fri, Jul 17, 2009 at 9:05 AM, Nathan Beyer <ndbeyer@apache.org>
>> wrote:
>> >>>>
>> >>>>> On Thu, Jul 16, 2009 at 1:28 AM, Charles Lee<littlee1032@gmail.com>
>> wrote:
>> >>>>> > Hi guys,
>> >>>>> >
>> >>>>> > I have add the locale function in the drlvm, the patch
is attached.
>> >>>>> Please
>> >>>>> > try this new patch on the linux.
>> >>>>> >
>> >>>>> > The patch should work on the linux but fail on the windows.
Because
>> >>>>> windows
>> >>>>> > returns code page not charset from the setlocale.
>> >>>>>
>> >>>>> Code page and character set are the same thing. We shouldn't
need to
>> >>>>> convert it as the Charset APIs will have to support the values
>> anyway.
>> >>>>>
>> >>>>> What's the value you're getting? If it's 'Cp1252', then we're
good,
>> as
>> >>>>> that's just an alias for 'Windows-1252' (or vice-versa).
>> >>>>>
>> >>>>> -Nathan
>> >>>>>
>> >>>>>
>> >>>>> > I hv tried long time to
>> >>>>> > get the charset name from the codepage, for example:
>> >>>>> > CPINFOEX cpInfoEx;
>> >>>>> > BOOL iReturn = GetCPInfoEx(CP_ACP,0, &cPInfoEx);
>> >>>>> > if (iReturn > 0) {
>> >>>>> >     printf("FULL NAME %s\n", cPinfoEx,CodePageName);
>> >>>>> > }
>> >>>>> > But I only get the full name without any format.
>> >>>>> >
>> >>>>> > There is code page identifiers map in the msdn, detail
here. I may
>> hard
>> >>>>> code
>> >>>>> > this map in the file. But the note on the msdn says:
>> >>>>> >      "ANSI code pages can be different on different
computers, or
>> can be
>> >>>>> > changed for a single computer, leading to data corruption.
For the
>> most
>> >>>>> > consistent results, applications should use Unicode, such
as UTF-8
>> or
>> >>>>> > UTF-16, instead of a specific code page."
>> >>>>> > I am afraid hard-code will fail on some machines. (By the
way, this
>> seems
>> >>>>> > the UTF-8 is suggested to be the default again :-)
>> >>>>> >
>> >>>>> > There is also a class Encoding in the VC++, detail here.
But we can
>> not
>> >>>>> use
>> >>>>> > it here.
>> >>>>> >
>> >>>>> > So anyone knows some thing about locale on the windows?
>> >>>>> > Again, shall use UTF-8 as our default?
>> >>>>> >
>> >>>>> > On Wed, Jul 15, 2009 at 2:12 PM, Charles Lee <
>> littlee1032@gmail.com>
>> >>>>> wrote:
>> >>>>> >>
>> >>>>> >> That seems we should add it in the drlvm.
>> >>>>> >>
>> >>>>> >> On Wed, Jul 15, 2009 at 1:58 PM, Regis <xu.regis@gmail.com>
>> wrote:
>> >>>>> >>>
>> >>>>> >>> Nathan Beyer wrote:
>> >>>>> >>>>
>> >>>>> >>>> Is the IBM VME dealing with this correctly?
Do we just need to
>> fix
>> >>>>> >>>> DRLVM?
>> >>>>> >>>
>> >>>>> >>> Yes, I only tested on Linux, IBM VME set the property
correctly.
>> >>>>> >>>
>> >>>>> >>>>
>> >>>>> >>>> On Wed, Jul 15, 2009 at 12:25 AM, Regis<xu.regis@gmail.com>
>> wrote:
>> >>>>> >>>>>
>> >>>>> >>>>> Kevin Zhou wrote:
>> >>>>> >>>>>>
>> >>>>> >>>>>> Yea, from luniglob.c, CL attempts to
read the "file.encoding"
>> >>>>> property
>> >>>>> >>>>>> adown
>> >>>>> >>>>>> VM but fails to get the correct encoding.
>> >>>>> >>>>>>
>> >>>>> >>>>>> Regis, do you know any other specific
ways that CL can gain
>> the
>> >>>>> right
>> >>>>> >>>>>> property?
>> >>>>> >>>>>
>> >>>>> >>>>> We can get from OS directly. Maybe just
read env variables on
>> Linux?
>> >>>>> >>>>>
>> >>>>> >>>>>> Wed, Jul 15, 2009 at 9:59 AM, Regis
<xu.regis@gmail.com>
>> wrote:
>> >>>>> >>>>>>
>> >>>>> >>>>>>> Charles Lee wrote:
>> >>>>> >>>>>>>
>> >>>>> >>>>>>>> Hi Nanthan,
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>> If the file encoding derive
from the OS, it should be the
>> some
>> >>>>> bugs
>> >>>>> >>>>>>>> in
>> >>>>> >>>>>>>> it
>> >>>>> >>>>>>>> because on my LINUX machine
the locale is en_US.UTF-8. Our
>> default
>> >>>>> >>>>>>>> codec
>> >>>>> >>>>>>>> is
>> >>>>> >>>>>>>> still ISO8859-1. Do you know
where can we found such codes?
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>> Classlib expected vm do this and
set the property, but it
>> didn't,
>> >>>>> so
>> >>>>> >>>>>>> we
>> >>>>> >>>>>>> have to do this by ourselves.
>> >>>>> >>>>>>>
>> >>>>> >>>>>>>
>> >>>>> >>>>>>>
>> >>>>> >>>>>>>> On Tue, Jul 14, 2009 at 10:17
PM, Nathan Beyer <
>> nbeyer@gmail.com>
>> >>>>> >>>>>>>> wrote:
>> >>>>> >>>>>>>>
>> >>>>> >>>>>>>>  Are we talking about windows
or linux?the default file
>> encoding
>> >>>>> >>>>>>>> should
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> derive from the OS. I believe
that's defined by the specs.
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> Sent from my iPhone
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>> On Jul 14, 2009, at 5:51
AM, Charles Lee <
>> littlee1032@gmail.com>
>> >>>>> >>>>>>>>> wrote:
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>>  On Tue, Jul 14, 2009
at 6:12 PM, Jimmy,Jing Lv
>> >>>>> >>>>>>>>> <firepure@gmail.com>
>> >>>>> >>>>>>>>>
>> >>>>> >>>>>>>>>> wrote:
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>>  Hi,
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>>>  Charles, I believe
UTF-8 is the default encoding for RI,
>> and
>> >>>>> it
>> >>>>> >>>>>>>>>>> sounds
>> >>>>> >>>>>>>>>>> reasonable.
>> >>>>> >>>>>>>>>>>  BTW, it may encounter
some compatibility problem, maybe
>> we
>> >>>>> need
>> >>>>> >>>>>>>>>>> to
>> >>>>> >>>>>>>>>>> run
>> >>>>> >>>>>>>>>>> more tests to verify?
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>> 2009/7/14 Charles
Lee <littlee1032@gmail.com>
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>  Hi guys:
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> I am doing
some test cases on the ant junit test case
>> and
>> >>>>> >>>>>>>>>>>> meeting
>> >>>>> >>>>>>>>>>>> some
>> >>>>> >>>>>>>>>>>> encoding problems.
I find they are maybe caused by the
>> >>>>> different
>> >>>>> >>>>>>>>>>>> default
>> >>>>> >>>>>>>>>>>> encoding from
RI and harmony. My local is en_US.UTF-8,
>> RI
>> >>>>> >>>>>>>>>>>> default is
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>  UTF-8
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>  but harmony is
8859-1. And then I have encountered
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> HARMONY-3736<
>> >>>>> https://issues.apache.org/jira/browse/HARMONY-3736>,
>> >>>>> >>>>>>>>>>>> and the two
diffs attached on that issue. It seems we
>> always
>> >>>>> get
>> >>>>> >>>>>>>>>>>> 8859-1.
>> >>>>> >>>>>>>>>>>> Because: (correct
me if wrong :-)
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> 1. we remove
the set code in the vm. we will always get
>> null
>> >>>>> if
>> >>>>> >>>>>>>>>>>> we
>> >>>>> >>>>>>>>>>>> call
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>  vm
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>  method
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> 2. we set the
file.encode in the libglob.c, if we got
>> null
>> >>>>> from
>> >>>>> >>>>>>>>>>>> vm,
>> >>>>> >>>>>>>>>>>> we
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>  set
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>  Sorry, it should
be luniglob.c
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>  8859-1.
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> 3. we can not
set file.encode on the run time.
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> ant use UTF-8
to encode filename which contains the
>> non-ascii
>> >>>>> >>>>>>>>>>>> character.
>> >>>>> >>>>>>>>>>>> So why we use
iso8859-1 as our unchangeable default?
>> >>>>> >>>>>>>>>>>> From the wiki
http://en.wikipedia.org/wiki/ISO8859-1,
>> it says
>> >>>>> >>>>>>>>>>>> "In
>> >>>>> >>>>>>>>>>>> computing
>> >>>>> >>>>>>>>>>>> applications,
encodings that provide full UCS support
>> (such as
>> >>>>> >>>>>>>>>>>> UTF-8<http://en.wikipedia.org/wiki/UTF-8>and
>> >>>>> >>>>>>>>>>>> UTF-16 <http://en.wikipedia.org/wiki/UTF-16>)
are
>> finding
>> >>>>> >>>>>>>>>>>> increasing
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>  favor
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>  over encodings
based on ISO 8859-1." Should we simply
>> change
>> >>>>> >>>>>>>>>>> iso8859-1
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> to
>> >>>>> >>>>>>>>>>>> utf-8?
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>> --
>> >>>>> >>>>>>>>>>>> Yours sincerely,
>> >>>>> >>>>>>>>>>>> Charles Lee
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>>>
>> >>>>> >>>>>>>>>>> --
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>> Best Regards!
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>> Jimmy, Jing Lv
>> >>>>> >>>>>>>>>>> China Software
Development Lab, IBM
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>>>
>> >>>>> >>>>>>>>>> --
>> >>>>> >>>>>>>>>> Yours sincerely,
>> >>>>> >>>>>>>>>> Charles Lee
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>>>>>
>> >>>>> >>>>>>> --
>> >>>>> >>>>>>> Best Regards,
>> >>>>> >>>>>>> Regis.
>> >>>>> >>>>>>>
>> >>>>> >>>>>
>> >>>>> >>>>> --
>> >>>>> >>>>> Best Regards,
>> >>>>> >>>>> Regis.
>> >>>>> >>>>>
>> >>>>> >>>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> --
>> >>>>> >>> Best Regards,
>> >>>>> >>> Regis.
>> >>>>> >>
>> >>>>> >>
>> >>>>> >>
>> >>>>> >> --
>> >>>>> >> Yours sincerely,
>> >>>>> >> Charles Lee
>> >>>>> >>
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > --
>> >>>>> > Yours sincerely,
>> >>>>> > Charles Lee
>> >>>>> >
>> >>>>> >
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Yours sincerely,
>> >>>> Charles Lee
>> >>>>
>> >>>
>> >>
>> >
>>
>
>
>
> --
> Yours sincerely,
> Charles Lee
>

Mime
View raw message