Mailing-List: contact dev-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@qpid.apache.org
Received-SPF: pass (athena.apache.org: domain of
 prvs=1945baea6b=rhs@alum.mit.edu designates 18.7.68.15 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANVjZ_TiVJpfpY57NXjzK+3YCRV0K6TcFxCvq+O+DcFqDD4H=w@mail.gmail.com>
References: <20130821124410.256210@gmx.com>
	<CANVjZ_TiVJpfpY57NXjzK+3YCRV0K6TcFxCvq+O+DcFqDD4H=w@mail.gmail.com>
Date: Wed, 21 Aug 2013 11:49:45 -0400
Message-ID: 
 <CACL1BnBrNANyEvs+z8ggfHz+9TakGwz9RrnJQdOU7kTqisDcbQ@mail.gmail.com>
Subject: Re: UTF8 / binary strings in dynamic languages
From: Rafael Schloming <rhs@alum.mit.edu>
To: dev@qpid.apache.org
Cc: "users@qpid.apache.org" <users@qpid.apache.org>
Content-Type: multipart/alternative; boundary=047d7b2e4d705f753304e4771f16

--047d7b2e4d705f753304e4771f16
Content-Type: text/plain; charset=ISO-8859-1

I think for python at least if we were to treat ambiguous string values as
text rather than data, we would be at odds with the python community's 2->3
migration plan. The following thread has a useful discussion of this that
is worth a careful read:

http://stackoverflow.com/questions/1736228/python-data-vs-text/1736279#1736279

--Rafael


On Wed, Aug 21, 2013 at 11:31 AM, Justin Ross <justin.ross@gmail.com> wrote:

> Jimmy, thanks for getting this started.  I'd love your feedback to
> help sort this out.
>
> I think these are the cases:
>
> 1. If the language string is unambiguously textual, send it as amqp str16
> 2. If the language string is unambiguously arbitrary bytes, send it as
> amqp vbin
>
> These are easy.  We can tell the user's intention, and we can do the
> right thing.
>
> 3. If the language string is an overloaded text/bytes type, as is
> regrettably quite common, what do we do then?
>
> The current answer to this question is "send it as vbin".  That's very
> safe, insofar as it won't throw any sort of encoding exception.  It
> does not, however, always honor what I think is the user's more
> typical intention: produce an ascii string at the other end.
>
> So for 3, I'd like to consider the possibility of, by default, sending
> ambiguous language strings as ascii rendered to amqp str16.  This
> requires an encoding step that may produce errors.  And maybe that's
> just too obnoxious!  That's what I'd like to know.
>
> In summary, if we have a way to determine what the user wanted (text
> or bytes), we should try to carry that through on the wire.  At the
> following URL I've tried to map out what type information we can get
> for each language.  Please update it as you please.
>
>
> https://cwiki.apache.org/confluence/display/qpid/Language+support+for+unambiguous+text+string+and+byte+array+types
>
> On Wed, Aug 21, 2013 at 8:44 AM, Jimmy Jones <jimmyjones2@gmx.co.uk>
> wrote:
> >> > AFAIK in perl, if you include unicode characters in a string it'll
> >> > set the utf8 flag. If you don't include any unicode characters (eg. 7
> >> > bit ascii, or raw bytes) the flag won't be set. So given a perl
> >> > scalar that doesn't contain any utf8 characters, you don't know if
> >> > its a textual string (str16) or a binary string (vbin). There is a
> >> > is_utf8_string function, but that'll only tell you if the string
> >> > would be valid utf8, but it could be a binary string that happens to
> >> > be valid utf8, so that's not really safe.
> >>
> >> You can explicitly mark it as utf8 using utf8::upgrade() though, right?
> >> Certainly I tried that in a simple test and the property in question was
> >> then sent as str16.
> >
> > Yes, if I as a user had a string that was textual, I could call
> utf8::upgrade() to ensure it got sent as str16. I guess this is similar in
> concept to calling setEncoding in C++, although maybe less natural in a
> dynamically typed language.
>
> It would be more reasonable to treat perl scalars as textual for our
> API if perl offered a good way to explicitly handle byte arrays.  My
> (certainly insufficient) web browsing suggested that wasn't really
> available, or not in a form recommended for use.  Any candidates for a
> serviceable explicitly-arbitrary-bytes-and-not-text-at-all "type" in
> perl?
>
> Thanks!
> Justin
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>

--047d7b2e4d705f753304e4771f16--