esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Jewett <>
Subject Metadata handling (was "Release planning")
Date Sat, 17 Jul 2010 18:08:44 GMT
Oh .... I see how it works. Thanks for the example. I was never able
to understand this before. This functionality is not what we want, but
just for completeness, here is a summary of how passing in metadata
through the API currently works (basic XML example):

Send in: <outer><meta><metameta>Hello</metameta></meta><onlymeta>Meta</onlymeta></outer>

Get out: HelloMeta

Note: this is the text contents of the XML tags concatenated together
in "order", whatever "order" means for XML content.

Here's a JSON example:

Send in: <anytag>"meta":[{"place":{"place_type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Down"}}]</anytag>

Get out: &quot;meta&quot;:[{&quot;place&quot;:{&quot;place_type&quot;:&quot;city&quot;,&quot;region&quot;:&quot;CA+&quot;}},{&quot;song&quot;:{&quot;artist&quot;:&quot;Prince&quot;,&quot;songtitle&quot;:&quot;Never+Let+Me+Down&quot;}}]

Note: This is, again, the text contents of the XML tags. The output is
HTML entity encoded, as it was before, but there weren't any
characters in the previous example that would have been encoded.

Second note: The input *must* be wrapped in valid XML tags or it is ignored.

I believe that this is completely wrong. The API should simply take
the text assigned to the metadata parameter, store it as part of the
message, and return exactly the same string in the message response.
It should not care what is in the text and it should not modify it in
any way.

I think that making this work is going to take some revamping of the
internal handling of metadata, moving any kind of parsing and/or HTML
entity encoding to the UI code and have the internal Message model
simply store the metadata without thinking about it so hard. This
could clearly result in breakage, so I'm not sure how best to proceed.

We could just put code into the API to make this behave how we want by
URLencoding it wrapping it in a special XML tag before storing it as
part of the message, then un-entity encode it and then un-URLencode it
before sending it back to the client, but that is going to result in a
horrible mess stored in the actual message. I think it will be better
to change the way that the actual Message class handles metadata.

Does anyone know what functionality of the web UI currently uses
metadata, so that we can test for breakage if we change how this

Imtiaz, if you want to work on making this behave in a sane way, that
would be fantastic. It's a big job, but we've got to start somewhere.
I think ESME-242 is a good umbrella issue for this and we can modify
the issue to better reflect the problem.


On Sat, Jul 17, 2010 at 6:57 PM, Imtiaz Ahmed H E <> wrote:
> For example, for the response of the following (GET) maybe we need to url
> decode the response before we (the api) respond with it ? (Reference:  see
> the reponse to the POST of a message in my previous mail, forwarded below).
> curl -b headers
> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/user/messages
> Of course this may be a typical naive one from me...! But let me know. Dick
> had said maybe I look at 242 before Vassil made his point & this discussion
> took place...
> Imtiaz
> ----- Original Message ----- From: "Imtiaz Ahmed H E" <>
> To: <>; "Ethan Jewett" <>
> Sent: Saturday, July 17, 2010 9:35 PM
> Subject: Re: Release planning
>> you are...
>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>> $ curl --dump-header headers -d "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL"
>> http:/
>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session
>> <?xml version="1.0" encoding="UTF-8"?>
>> <api><session><user><id>3</id><nickname>imtiaz2</nickname><image>None</image><wh
>> ole_name>I A 2 H E</whole_name></user></session></api>
>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>> $ cat headers
>> HTTP/1.1 200 OK
>> Server: Apache-Coyote/1.1
>> Set-Cookie: JSESSIONID=28DBA5607A4D94C9E6E2BD1E61492EAD;
>> Path=/esme-server-apach
>> e-esme-1.0-RC1-incubating
>> Expires: Sat, 17 Jul 2010 15:58:22 UTC
>> Date: Sat, 17 Jul 2010 15:58:22 GMT
>> Pragma: no-cache
>> Cache-Control: no-cache; private; no-store
>> X-Lift-Version: 2.0-SNAPSHOT
>> Content-Type: text/xml;charset=utf-8
>> Content-Length: 178
>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>> $ curl -b headers -d
>> 'message=test72&metadata=<metadata>"meta":[{"place":{"plac
>> e_type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+L
>> et+Me+Down"}}]</metadata>'
>> http://localhost:8080/esme-server-apache-esme-1.0-RC
>> 1-incubating/api2/user/messages
>> <?xml version="1.0" encoding="UTF-8"?>
>> <api><message>
>>  <id>10</id>
>>  <date>Sat, 17 Jul 2010 15:58:44 UTC</date>
>>  <source>api2</source>
>>  <body>test72</body>
>> <metadata>&quot;meta&quot;:[{&quot;place&quot;:{&quot;place_type&quot;:&quot;c
>> ity&quot;,&quot;region&quot;:&quot;CA
>> &quot;}},{&quot;song&quot;:{&quot;artist&q
>> uot;:&quot;Prince&quot;,&quot;songtitle&quot;:&quot;Never Let
>> Down&quot;}}]</
>> metadata>
>>  <author><nickname>imtiaz2</nickname><id>3</id></author>
>>  <tags></tags><replyto></replyto><conversation></conversation>
>> </message></api>
>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>> $
>> Imtiaz
>> ----- Original Message ----- From: "Ethan Jewett" <>
>> To: <>; <>
>> Sent: Saturday, July 17, 2010 8:30 PM
>> Subject: Re: Release planning
>> Re ESME-242 - I think you (and Vassil) are right, but I haven't been
>> able to independently confirm and I haven't seen a working API call
>> attached to the Jira issue, so I'm not willing to close it yet. If we
>> had working metadata examples using properly escaped XML and JSON, I
>> would be very happy :-)
>> Ethan
>> On Sat, Jul 17, 2010 at 4:32 PM,  <> wrote:
>>> About ESME-242,
>>> Per Vassil Dichev's mail to this group that http POST text used in the
>>> shell curl command must not be quoted and must be url encoded or must be
>>> quoted using single quotes, this Jira issue should be resolved to 'not a
>>> bug', if I recall correctly, that is!
>>> Imtiaz
>>> Sent from BlackBerryŽ on Airtel
>>> -----Original Message-----
>>> From: Ethan Jewett <>
>>> Date: Sat, 17 Jul 2010 16:11:44
>>> To: <>
>>> Reply-To:
>>> Subject: Re: Release planning
>>> Thoughts on items remaining for this release:
>>> ESME-225 - "need unit test for GET api2/users/actions don't retrieve
>>> disabled actions" - I don't think this is required for release and
>>> should be moved to the backlog.
>>> ESME-242 - "The use of metadata in the api2 currently doesn't work" -
>>> I think this is important but also not required for release and should
>>> be moved to the backlog.
>>> I agree that ESME-205 and ESME-212 are major. ESME-212 at least is a
>>> blocker.
>>> Regarding changes to be made to API2: I've created JIRA issues for all
>>> the todos at the bottom of the file (except a couple that were already
>>> done). It was lazy of me to put a todo-list in the file - I should
>>> really avoid that and force myself to have everything in Jira. I will
>>> be checking in a version of the file without the todo-list shortly.
>>> Cheers,
>>> Ethan
>>> On Fri, Jul 16, 2010 at 12:40 PM, Richard Hirsch <>
>>> wrote:
>>>> I've been working on the JIRA items and we have 10 items left for this
>>>> release.
>>>> The following items are linked to IE 7:
>>>> * ESME-207
>>>> * ESME-239
>>>> * ESME-208
>>>> Some of these items are CSS-related (for example, 207) but the other
>>>> ones are javascript / jquery related.
>>>> I think the two main issues are:
>>>> * ESME-205 Search is broken (This only happens when you use a JDBC
>>>> based compass indexes - first noticed on Stax but it also is present
>>>> locally)
>>>> * ESME-212 Messages from pools aren't hidden
>>>> I'm going to be gone the next four weeks on vacation, so either
>>>> someone else can coordinate the release or we wait until I return to
>>>> continue with our release preparations.
>>>> If anyone wants to work on other items until the release, here are
>>>> other items that I consider important:
>>>> * ESME-108 - View my pools" functionality
>>>> * ESME-170 Pubsubhubbub support for Atom & RSS actions#
>>>> * ESME-214 Add container-based authentication
>>>> * ESME-213 Twitter API is out-of-date and API calls don't always
>>>> return the correct info
>>>> -- other api2 related changes - see the ToDos at the bottom of the
>>>> API2.scala page .
>>>> D.

View raw message