esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ethan Jewett <esjew...@gmail.com>
Subject Re: Metadata handling (was "Release planning")
Date Fri, 23 Jul 2010 13:53:58 GMT
Done. The commit email should show up in a few minutes.

@Imtiaz - I also committed (separately) changes to the two failing
unit tests for the metadata issue on the metadata branch. You can run
the unit tests by doing "mvn test" or "mvn clean test", the same way
you would do "mvn jetty:run".

The two remaining problems as far as I can tell:

1. Response is HTML entity-encoded. We need to decode it before we
send it back to the client, or make sure it is never encoded in the
first place.

2. There is an extra <metadata></metadata> tag around the metadata
(which is then entity-encoded). You can see this in your test results.

I'm not sure if these problems are internal to the Message class or in
the API code, but if you want to work on fixing them, please feel
free.

Ethan

On Fri, Jul 23, 2010 at 11:20 AM, Anne Kathrine Petter√łe
<yojibee@gmail.com> wrote:
> Thank you for you work on ESME-242 Imtiaz!
> Ethan, will you commit the changes to metadata branc so that we can test it?
>
> /Anne
>
>
> On 23. juli 2010, at 11.03, Imtiaz Ahmed H E wrote:
>
>> Maybe I'm just cc'ing myself too !!!
>> Sorry!
>>
>> ----- Original Message ----- From: "Imtiaz Ahmed H E" <in.imtiaz@gmail.com>
>> To: <esme-dev@incubator.apache.org>
>> Sent: Friday, July 23, 2010 2:30 PM
>> Subject: Re: Metadata handling (was "Release planning")
>>
>>
>>> By the way, someone please tell me why I have started receiving my own posts
to the lists also since the last couple or so of posts. It's pretty irritating :)
>>>
>>> Imtiaz
>>>
>>> ----- Original Message ----- From: "Imtiaz Ahmed H E" <in.imtiaz@gmail.com>
>>> To: <esme-dev@incubator.apache.org>
>>> Sent: Friday, July 23, 2010 2:24 PM
>>> Subject: Re: Metadata handling (was "Release planning")
>>>
>>>
>>>> I have attached a fix (Esme_branches_metadata_Jira242Fix.diff) to Jira ticket
242, with the following change
>>>>
>>>> The line,
>>>>
>>>> lazy val metadata: String = originalXml \ "metadata" text
>>>>
>>>> commented and replaced by the line
>>>>
>>>> lazy val metadata: String = (originalXml \ "metadata").toString
>>>>
>>>> See my earlier mail to esme-dev for results from the fix.
>>>>
>>>> The fix is only for the metadata branch in the SVN repo directory 'branches'.
>>>>
>>>> This is pending further evaluation by esme-dev and improvements etc...
>>>>
>>>> Imtiaz
>>>>
>>>> ----- Original Message ----- From: "Ethan Jewett" <esjewett@gmail.com>
>>>> To: <esme-dev@incubator.apache.org>
>>>> Sent: Wednesday, July 21, 2010 7:51 PM
>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>
>>>>
>>>>> Sounds good to me, at least until we can figure out what the next
>>>>> step/problem is on the metadata front. Just ping the list when you've
>>>>> uploaded the patch.
>>>>>
>>>>> You don't happen to have any JDBC/JNDI and/or Lucene experience do
>>>>> you? I'm totally stumped on the issue with search when we have it run
>>>>> using the database instead of the filesystem (ESME-205).
>>>>>
>>>>> Ethan
>>>>>
>>>>> On Wed, Jul 21, 2010 at 4:06 PM, Imtiaz Ahmed H E <in.imtiaz@gmail.com>
wrote:
>>>>>> Oops! I think I wrote that in a hurry, not even thinking!!
>>>>>>
>>>>>> Actually, I think I should just submit a patch to the metadata branch
with
>>>>>> my current change and move on to other Jira tickets while we wait
for
>>>>>> consensus/finalization.
>>>>>>
>>>>>> Imtiaz
>>>>>>
>>>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E" <in.imtiaz@gmail.com>
>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>> Sent: Wednesday, July 21, 2010 7:30 PM
>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>>
>>>>>>
>>>>>>> How about it if I just store the metadata as a literal string
and return
>>>>>>> it as such for now and submit a patch to the metadata branch
with that and
>>>>>>> move on to another Jira ticket ?
>>>>>>>
>>>>>>> Then, when we all agree on something final for the output format
and
>>>>>>> representation of message metadata we can look at further work
on ESME-242.
>>>>>>>
>>>>>>> Imtiaz
>>>>>>>
>>>>>>> ----- Original Message ----- From: "Ethan Jewett" <esjewett@gmail.com>
>>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>>> Sent: Wednesday, July 21, 2010 7:01 PM
>>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>>>
>>>>>>>
>>>>>>> Imtiaz,
>>>>>>>
>>>>>>> Excellent! That's good progress.
>>>>>>>
>>>>>>> Any idea what the API response is still getting HTML entity-encoded?
>>>>>>> That's what all of those &lt; and &gt; things are. Does
the API do
>>>>>>> this for all of its responses or is it just the metadata ones?
We
>>>>>>> probably don't need to be entity-encoding since we're returning
XML,
>>>>>>> not HTML.
>>>>>>>
>>>>>>> Also, I just created a branch called "metadata". Can you work
against
>>>>>>> this branch and feel free to submit patches against that branch
to the
>>>>>>> Jira ticket. Vassil, Anne, or I can then apply the patches directly
to
>>>>>>> the branch without the level of review that would be needed for
a
>>>>>>> patch against trunk. This should help allow us to try out your
changes
>>>>>>> and come to consensus and understanding more quickly.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Ethan
>>>>>>>
>>>>>>> On Wed, Jul 21, 2010 at 7:20 AM, Imtiaz Ahmed H E <in.imtiaz@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> By the way, all I did was,
>>>>>>>>
>>>>>>>> in Message.scala,
>>>>>>>> replace,
>>>>>>>>
>>>>>>>> lazy val metadata: String = originalXml \ "metadata" text
>>>>>>>>
>>>>>>>> by...
>>>>>>>>
>>>>>>>> lazy val metadata: String = (originalXml \ "metadata").toString
>>>>>>>>
>>>>>>>> to get the results you see in my previous mail...
>>>>>>>>
>>>>>>>> Imtiaz
>>>>>>>>
>>>>>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E"
>>>>>>>> <in.imtiaz@gmail.com>
>>>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>>>> Sent: Wednesday, July 21, 2010 10:34 AM
>>>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> In the absence of decisive consensus yet, I made a small
change in
>>>>>>>>> Message.scala as a first step in fixing ESME-242.
>>>>>>>>> Here's curl output for the Jira ticket examples given
there by
>>>>>>>>> Ethan...There is maybe obvious unacceptable stuff here,
but let me
>>>>>>>>> know...
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $ curl --dump-header headers -d "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL"
>>>>>>>>> http:/
>>>>>>>>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session
>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <api><session><user><id>3</id><nickname>imtiaz2</nickname><image>None</image><wh
>>>>>>>>> ole_name>I A 2 H E</whole_name></user></session></api>
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $ cat headers
>>>>>>>>> HTTP/1.1 200 OK
>>>>>>>>> Server: Apache-Coyote/1.1
>>>>>>>>> Set-Cookie: JSESSIONID=C7B688BAF8E99B2A638EF432885E310E;
>>>>>>>>> Path=/esme-server-apach
>>>>>>>>> e-esme-1.0-RC1-incubating
>>>>>>>>> Expires: Wed, 21 Jul 2010 04:56:51 UTC
>>>>>>>>> Date: Wed, 21 Jul 2010 04:56:51 GMT
>>>>>>>>> Pragma: no-cache
>>>>>>>>> Cache-Control: no-cache; private; no-store
>>>>>>>>> X-Lift-Version: 2.0-SNAPSHOT
>>>>>>>>> Content-Type: text/xml;charset=utf-8
>>>>>>>>> Content-Length: 178
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $ curl -b headers -d
>>>>>>>>> 'message=test200&metadata=<outer><meta><metameta>Hello</me
>>>>>>>>> tameta></meta><onlymeta>Meta</onlymeta></outer>'
>>>>>>>>> http://localhost:8080/esme-ser
>>>>>>>>> ver-apache-esme-1.0-RC1-incubating/api2/user/messages
>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>> <api><message>
>>>>>>>>> <id>28</id>
>>>>>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date>
>>>>>>>>> <source>api2</source>
>>>>>>>>> <body>test200</body>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <metadata>&lt;metadata&gt;&lt;outer&gt;&lt;meta&gt;&lt;metameta&gt;Hello&lt;/m
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> etameta&gt;&lt;/meta&gt;&lt;onlymeta&gt;Meta&lt;/onlymeta&gt;&lt;/outer&gt;&lt;/
>>>>>>>>> metadata&gt;</metadata>
>>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>>> </message></api>
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $ curl -b headers -d
>>>>>>>>> 'message=test201&metadata=<anytag>"meta":[{"place":{"place
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Le
>>>>>>>>> t+Me+Down"}}]</anytag>'
>>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-i
>>>>>>>>> ncubating/api2/user/messages
>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>> <api><message>
>>>>>>>>> <id>29</id>
>>>>>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date>
>>>>>>>>> <source>api2</source>
>>>>>>>>> <body>test201</body>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <metadata>&lt;metadata&gt;&lt;anytag&gt;&amp;quot;meta&amp;quot;:[{&amp;quot;p
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> lace&amp;quot;:{&amp;quot;place_type&amp;quot;:&amp;quot;city&amp;quot;,&amp;quo
>>>>>>>>> t;region&amp;quot;:&amp;quot;CA
>>>>>>>>> &amp;quot;}},{&amp;quot;song&amp;quot;:{&amp;quo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> t;artist&amp;quot;:&amp;quot;Prince&amp;quot;,&amp;quot;songtitle&amp;quot;:&amp
>>>>>>>>> ;quot;Never Let Me
>>>>>>>>> Down&amp;quot;}}]&lt;/anytag&gt;&lt;/metadata&gt;</metadata>
>>>>>>>>>
>>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>>> </message></api>
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $ curl -b headers -d
>>>>>>>>> 'message=test202&metadata="meta":[{"place":{"place_type":"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Dow
>>>>>>>>> n"}}]'
>>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/us
>>>>>>>>> er/messages
>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>> <api><message>
>>>>>>>>> <id>30</id>
>>>>>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date>
>>>>>>>>> <source>api2</source>
>>>>>>>>> <body>test202</body>
>>>>>>>>> <metadata></metadata>
>>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>>> </message></api>
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $ curl -b headers
>>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubat
>>>>>>>>> ing/api2/user/messages
>>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>>> <api><messages><message>
>>>>>>>>> <id>30</id>
>>>>>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date>
>>>>>>>>> <source>api2</source>
>>>>>>>>> <body>test202</body>
>>>>>>>>> <metadata></metadata>
>>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>>> </message><message>
>>>>>>>>> <id>29</id>
>>>>>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date>
>>>>>>>>> <source>api2</source>
>>>>>>>>> <body>test201</body>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <metadata>&lt;metadata&gt;&lt;anytag&gt;&amp;quot;meta&amp;quot;:[{&amp;quot;p
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> lace&amp;quot;:{&amp;quot;place_type&amp;quot;:&amp;quot;city&amp;quot;,&amp;quo
>>>>>>>>> t;region&amp;quot;:&amp;quot;CA
>>>>>>>>> &amp;quot;}},{&amp;quot;song&amp;quot;:{&amp;quo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> t;artist&amp;quot;:&amp;quot;Prince&amp;quot;,&amp;quot;songtitle&amp;quot;:&amp
>>>>>>>>> ;quot;Never Let Me
>>>>>>>>> Down&amp;quot;}}]&lt;/anytag&gt;&lt;/metadata&gt;</metadata>
>>>>>>>>>
>>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>>> </message><message>
>>>>>>>>> <id>28</id>
>>>>>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date>
>>>>>>>>> <source>api2</source>
>>>>>>>>> <body>test200</body>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <metadata>&lt;metadata&gt;&lt;outer&gt;&lt;meta&gt;&lt;metameta&gt;Hello&lt;/m
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> etameta&gt;&lt;/meta&gt;&lt;onlymeta&gt;Meta&lt;/onlymeta&gt;&lt;/outer&gt;&lt;/
>>>>>>>>> metadata&gt;</metadata>
>>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>>> </message></messages></api>
>>>>>>>>>
>>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>>> $
>>>>>>>>>
>>>>>>>>> ----- Original Message ----- From: "Vassil Dichev" <vdichev@apache.org>
>>>>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>>>>> Sent: Tuesday, July 20, 2010 11:27 AM
>>>>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> I'd prefer option 1 (separate attribute from
text). Within this
>>>>>>>>>>> separate attribute there is the question of how
data is
>>>>>>>>>>> stored/represented. I'm ok with either raw string
or a tuple-based
>>>>>>>>>>> structure like Twitter's. I kind of like the
tuple (key-value)
>>>>>>>>>>> approach.
>>>>>>>>>>
>>>>>>>>>> I'm not too interested in how the data is stored,
because it's fairly
>>>>>>>>>> trivial to implement either way. It's currently not
yet clear to me
>>>>>>>>>> what the requirements for the output format are.
>>>>>>>>>>
>>>>>>>>>>> What I'm insisting on and what I was saying we
got wrong is that what
>>>>>>>>>>> goes in needs to be the same as what comes out.
If it's tuple-based
>>>>>>>>>>> and I send in a tuple, then I should get that
tuple (key and value)
>>>>>>>>>>> back out when I request the metadata for a message.
Right now I think
>>>>>>>>>>> we only get a concatenated list of values from
the metadata and
>>>>>>>>>>> metaData methods and we're bound to an XML format.
>>>>>>>>>>
>>>>>>>>>> I don't get it. A tuple is an abstraction which might
be expressed in
>>>>>>>>>> a specific format. So what goes in is not what comes
out depending on
>>>>>>>>>> the format. Let me quote the specific example Twitter
provides.
>>>>>>>>>>
>>>>>>>>>> This comes in:
>>>>>>>>>>
>>>>>>>>>> "annotations":
>>>>>>>>>> [{"type":{"another_attribute":"value", "attribute":"value"}}]
>>>>>>>>>>
>>>>>>>>>> This comes out:
>>>>>>>>>>
>>>>>>>>>> <annotations type="array">
>>>>>>>>>> <annotation>
>>>>>>>>>> <type>foo</type>
>>>>>>>>>> <attributes>
>>>>>>>>>> <attribute>
>>>>>>>>>> <name>bar</name>
>>>>>>>>>> <value>baz</value>
>>>>>>>>>> </attribute>
>>>>>>>>>> </attributes>
>>>>>>>>>> </annotation>
>>>>>>>>>> </annotations>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> As far as requiring a particular format, I think
the internal format
>>>>>>>>>>> should be either a raw string or a immutable
hashmap with raw strings
>>>>>>>>>>> as keys and values. We can handle converting
this to XML or JSON in
>>>>>>>>>>> the API or view code.
>>>>>>>>>>
>>>>>>>>>> Again, it's not so interesting what's internally
there, let's just
>>>>>>>>>> treat it as a black box. What I want to know is,
do we want to have
>>>>>>>>>> for instance XML in a JSON reply returned:
>>>>>>>>>>
>>>>>>>>>> "annotations":
>>>>>>>>>> [{"type":{"<attributes> <attribute> <name>bar</name>
>>>>>>>>>> <value>baz</value> </attribute>
</attributes>"}}]
>>>>>>>>>>
>>>>>>>>>> or, inversely, do we want JSON inside an XML reply?
Something like:
>>>>>>>>>>
>>>>>>>>>> <annotations type="array">
>>>>>>>>>> <annotation>
>>>>>>>>>> <type>foo</type>
>>>>>>>>>> <attributes>
>>>>>>>>>> {"another_attribute":"value", "attribute":"value"}
>>>>>>>>>> </attributes>
>>>>>>>>>> </annotation>
>>>>>>>>>> </annotations>
>>>>>>>>>>
>>>>>>>>>> because the latter will need to be escaped.
>>>>>>>>>>
>>>>>>>>>> Sorry for being too dense, but in any case, we either
have to escape
>>>>>>>>>> the metadata or we have to transform the structure
to XML/JSON when we
>>>>>>>>>> return it back to the user. None of these is "what
goes in needs to be
>>>>>>>>>> the same as what comes out"
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>
>

Mime
View raw message