harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Lee <littlee1...@gmail.com>
Subject Re: Shall we change our file.encoding
Date Fri, 17 Jul 2009 02:30:54 GMT
Thanks Nathan!

I will try this :-)

On Fri, Jul 17, 2009 at 10:05 AM, Nathan Beyer <ndbeyer@apache.org> wrote:

> On Thu, Jul 16, 2009 at 8:50 PM, Nathan Beyer<ndbeyer@apache.org> wrote:
> > On Thu, Jul 16, 2009 at 8:35 PM, Nathan Beyer<ndbeyer@apache.org> wrote:
> >> On Thu, Jul 16, 2009 at 8:26 PM, Nathan Beyer<ndbeyer@apache.org>
> wrote:
> >>> On Thu, Jul 16, 2009 at 8:18 PM, Charles Lee<littlee1032@gmail.com>
> wrote:
> >>>> Hi Nathan,
> >>>>
> >>>> What I got is 936, the code page identifier. Is there a api for us to
> map
> >>>> 936 to the gb2312?
> >>>
> >>> Oh, the 'identifier' bit was missing - yeah, we'll need to translate
> >>> that into a name of some sort. I'll poke around a bit and see what I
> >>> can find.
> >>
> >> We'll probably just have to put in a mapping ourselves based on the
> >> documentation. We'd call GetACP [1] and map that to a known alias in
> >> java.nio.charset that matches the definitions[2] of the identifiers.
> >>
> >> [1] http://msdn.microsoft.com/en-us/library/dd318070%28VS.85%29.aspx
> >> [2] http://msdn.microsoft.com/en-us/library/dd317756%28VS.85%29.aspx
> >
> > This may be better - APR has a function for getting the OS default
> > encoding. This would work across all platforms that APR supports and I
> > believe we already use APR.
> >
> >
> http://apr.apache.org/docs/apr/1.3/group__apr__portabile.html#g6e21845a4a5f3b7dd107b2beea50c91e
>
> However, the Windows version of this is simply - return
> apr_psprintf(pool, "CP%u", (unsigned) GetACP());. Which is essentially
> "CP" + codePageId.
>
> And the Unix version of this method doesn't look very good for our
> purposes.
> >
> > -Nathan
> >>
> >>>
> >>>> If we put 936 in the file.encoding, can we successfully get the
> encoder and
> >>>> decoder by charset?
> >>>>
> >>>> On Fri, Jul 17, 2009 at 9:05 AM, Nathan Beyer <ndbeyer@apache.org>
> wrote:
> >>>>
> >>>>> On Thu, Jul 16, 2009 at 1:28 AM, Charles Lee<littlee1032@gmail.com>
> wrote:
> >>>>> > Hi guys,
> >>>>> >
> >>>>> > I have add the locale function in the drlvm, the patch is attached.
> >>>>> Please
> >>>>> > try this new patch on the linux.
> >>>>> >
> >>>>> > The patch should work on the linux but fail on the windows.
Because
> >>>>> windows
> >>>>> > returns code page not charset from the setlocale.
> >>>>>
> >>>>> Code page and character set are the same thing. We shouldn't need
to
> >>>>> convert it as the Charset APIs will have to support the values
> anyway.
> >>>>>
> >>>>> What's the value you're getting? If it's 'Cp1252', then we're good,
> as
> >>>>> that's just an alias for 'Windows-1252' (or vice-versa).
> >>>>>
> >>>>> -Nathan
> >>>>>
> >>>>>
> >>>>> > I hv tried long time to
> >>>>> > get the charset name from the codepage, for example:
> >>>>> > CPINFOEX cpInfoEx;
> >>>>> > BOOL iReturn = GetCPInfoEx(CP_ACP,0, &cPInfoEx);
> >>>>> > if (iReturn > 0) {
> >>>>> >     printf("FULL NAME %s\n", cPinfoEx,CodePageName);
> >>>>> > }
> >>>>> > But I only get the full name without any format.
> >>>>> >
> >>>>> > There is code page identifiers map in the msdn, detail here.
I may
> hard
> >>>>> code
> >>>>> > this map in the file. But the note on the msdn says:
> >>>>> >      "ANSI code pages can be different on different computers,
or
> can be
> >>>>> > changed for a single computer, leading to data corruption.
For the
> most
> >>>>> > consistent results, applications should use Unicode, such as
UTF-8
> or
> >>>>> > UTF-16, instead of a specific code page."
> >>>>> > I am afraid hard-code will fail on some machines. (By the way,
this
> seems
> >>>>> > the UTF-8 is suggested to be the default again :-)
> >>>>> >
> >>>>> > There is also a class Encoding in the VC++, detail here. But
we can
> not
> >>>>> use
> >>>>> > it here.
> >>>>> >
> >>>>> > So anyone knows some thing about locale on the windows?
> >>>>> > Again, shall use UTF-8 as our default?
> >>>>> >
> >>>>> > On Wed, Jul 15, 2009 at 2:12 PM, Charles Lee <
> littlee1032@gmail.com>
> >>>>> wrote:
> >>>>> >>
> >>>>> >> That seems we should add it in the drlvm.
> >>>>> >>
> >>>>> >> On Wed, Jul 15, 2009 at 1:58 PM, Regis <xu.regis@gmail.com>
> wrote:
> >>>>> >>>
> >>>>> >>> Nathan Beyer wrote:
> >>>>> >>>>
> >>>>> >>>> Is the IBM VME dealing with this correctly? Do
we just need to
> fix
> >>>>> >>>> DRLVM?
> >>>>> >>>
> >>>>> >>> Yes, I only tested on Linux, IBM VME set the property
correctly.
> >>>>> >>>
> >>>>> >>>>
> >>>>> >>>> On Wed, Jul 15, 2009 at 12:25 AM, Regis<xu.regis@gmail.com>
> wrote:
> >>>>> >>>>>
> >>>>> >>>>> Kevin Zhou wrote:
> >>>>> >>>>>>
> >>>>> >>>>>> Yea, from luniglob.c, CL attempts to read
the "file.encoding"
> >>>>> property
> >>>>> >>>>>> adown
> >>>>> >>>>>> VM but fails to get the correct encoding.
> >>>>> >>>>>>
> >>>>> >>>>>> Regis, do you know any other specific ways
that CL can gain
> the
> >>>>> right
> >>>>> >>>>>> property?
> >>>>> >>>>>
> >>>>> >>>>> We can get from OS directly. Maybe just read
env variables on
> Linux?
> >>>>> >>>>>
> >>>>> >>>>>> Wed, Jul 15, 2009 at 9:59 AM, Regis <xu.regis@gmail.com>
> wrote:
> >>>>> >>>>>>
> >>>>> >>>>>>> Charles Lee wrote:
> >>>>> >>>>>>>
> >>>>> >>>>>>>> Hi Nanthan,
> >>>>> >>>>>>>>
> >>>>> >>>>>>>> If the file encoding derive from
the OS, it should be the
> some
> >>>>> bugs
> >>>>> >>>>>>>> in
> >>>>> >>>>>>>> it
> >>>>> >>>>>>>> because on my LINUX machine the
locale is en_US.UTF-8. Our
> default
> >>>>> >>>>>>>> codec
> >>>>> >>>>>>>> is
> >>>>> >>>>>>>> still ISO8859-1. Do you know where
can we found such codes?
> >>>>> >>>>>>>>
> >>>>> >>>>>>> Classlib expected vm do this and set
the property, but it
> didn't,
> >>>>> so
> >>>>> >>>>>>> we
> >>>>> >>>>>>> have to do this by ourselves.
> >>>>> >>>>>>>
> >>>>> >>>>>>>
> >>>>> >>>>>>>
> >>>>> >>>>>>>> On Tue, Jul 14, 2009 at 10:17 PM,
Nathan Beyer <
> nbeyer@gmail.com>
> >>>>> >>>>>>>> wrote:
> >>>>> >>>>>>>>
> >>>>> >>>>>>>>  Are we talking about windows or
linux?the default file
> encoding
> >>>>> >>>>>>>> should
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> derive from the OS. I believe
that's defined by the specs.
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> Sent from my iPhone
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>> On Jul 14, 2009, at 5:51 AM,
Charles Lee <
> littlee1032@gmail.com>
> >>>>> >>>>>>>>> wrote:
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>>  On Tue, Jul 14, 2009 at 6:12
PM, Jimmy,Jing Lv
> >>>>> >>>>>>>>> <firepure@gmail.com>
> >>>>> >>>>>>>>>
> >>>>> >>>>>>>>>> wrote:
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>>  Hi,
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>>>  Charles, I believe
UTF-8 is the default encoding for RI,
> and
> >>>>> it
> >>>>> >>>>>>>>>>> sounds
> >>>>> >>>>>>>>>>> reasonable.
> >>>>> >>>>>>>>>>>  BTW, it may encounter
some compatibility problem, maybe
> we
> >>>>> need
> >>>>> >>>>>>>>>>> to
> >>>>> >>>>>>>>>>> run
> >>>>> >>>>>>>>>>> more tests to verify?
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>> 2009/7/14 Charles Lee
<littlee1032@gmail.com>
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>  Hi guys:
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>> I am doing some
test cases on the ant junit test case
> and
> >>>>> >>>>>>>>>>>> meeting
> >>>>> >>>>>>>>>>>> some
> >>>>> >>>>>>>>>>>> encoding problems.
I find they are maybe caused by the
> >>>>> different
> >>>>> >>>>>>>>>>>> default
> >>>>> >>>>>>>>>>>> encoding from RI
and harmony. My local is en_US.UTF-8,
> RI
> >>>>> >>>>>>>>>>>> default is
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>  UTF-8
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>  but harmony is 8859-1.
And then I have encountered
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> HARMONY-3736<
> >>>>> https://issues.apache.org/jira/browse/HARMONY-3736>,
> >>>>> >>>>>>>>>>>> and the two diffs
attached on that issue. It seems we
> always
> >>>>> get
> >>>>> >>>>>>>>>>>> 8859-1.
> >>>>> >>>>>>>>>>>> Because: (correct
me if wrong :-)
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> 1. we remove the
set code in the vm. we will always get
> null
> >>>>> if
> >>>>> >>>>>>>>>>>> we
> >>>>> >>>>>>>>>>>> call
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>  vm
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>  method
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> 2. we set the file.encode
in the libglob.c, if we got
> null
> >>>>> from
> >>>>> >>>>>>>>>>>> vm,
> >>>>> >>>>>>>>>>>> we
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>  set
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>  Sorry, it should be
luniglob.c
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>  8859-1.
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> 3. we can not set
file.encode on the run time.
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> ant use UTF-8 to
encode filename which contains the
> non-ascii
> >>>>> >>>>>>>>>>>> character.
> >>>>> >>>>>>>>>>>> So why we use iso8859-1
as our unchangeable default?
> >>>>> >>>>>>>>>>>> From the wiki http://en.wikipedia.org/wiki/ISO8859-1,
> it says
> >>>>> >>>>>>>>>>>> "In
> >>>>> >>>>>>>>>>>> computing
> >>>>> >>>>>>>>>>>> applications, encodings
that provide full UCS support
> (such as
> >>>>> >>>>>>>>>>>> UTF-8<http://en.wikipedia.org/wiki/UTF-8>and
> >>>>> >>>>>>>>>>>> UTF-16 <http://en.wikipedia.org/wiki/UTF-16>)
are
> finding
> >>>>> >>>>>>>>>>>> increasing
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>  favor
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>  over encodings based
on ISO 8859-1." Should we simply
> change
> >>>>> >>>>>>>>>>> iso8859-1
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> to
> >>>>> >>>>>>>>>>>> utf-8?
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>> --
> >>>>> >>>>>>>>>>>> Yours sincerely,
> >>>>> >>>>>>>>>>>> Charles Lee
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>>>
> >>>>> >>>>>>>>>>> --
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>> Best Regards!
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>> Jimmy, Jing Lv
> >>>>> >>>>>>>>>>> China Software Development
Lab, IBM
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>>>
> >>>>> >>>>>>>>>> --
> >>>>> >>>>>>>>>> Yours sincerely,
> >>>>> >>>>>>>>>> Charles Lee
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>>>>>
> >>>>> >>>>>>> --
> >>>>> >>>>>>> Best Regards,
> >>>>> >>>>>>> Regis.
> >>>>> >>>>>>>
> >>>>> >>>>>
> >>>>> >>>>> --
> >>>>> >>>>> Best Regards,
> >>>>> >>>>> Regis.
> >>>>> >>>>>
> >>>>> >>>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> --
> >>>>> >>> Best Regards,
> >>>>> >>> Regis.
> >>>>> >>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> Yours sincerely,
> >>>>> >> Charles Lee
> >>>>> >>
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>> > --
> >>>>> > Yours sincerely,
> >>>>> > Charles Lee
> >>>>> >
> >>>>> >
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Yours sincerely,
> >>>> Charles Lee
> >>>>
> >>>
> >>
> >
>



-- 
Yours sincerely,
Charles Lee

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message