esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anne Kathrine Petter√łe <yoji...@gmail.com>
Subject Re: Metadata handling (was "Release planning")
Date Fri, 23 Jul 2010 09:22:58 GMT
Oh and sorry for being a bit out of touch again, getting into things at Adobe was more time
consuming than I had expected. Should be back on-list over the weekend and next week.


On 23. juli 2010, at 11.03, Imtiaz Ahmed H E wrote:

> Maybe I'm just cc'ing myself too !!!
> Sorry!
> 
> ----- Original Message ----- From: "Imtiaz Ahmed H E" <in.imtiaz@gmail.com>
> To: <esme-dev@incubator.apache.org>
> Sent: Friday, July 23, 2010 2:30 PM
> Subject: Re: Metadata handling (was "Release planning")
> 
> 
>> By the way, someone please tell me why I have started receiving my own posts to the
lists also since the last couple or so of posts. It's pretty irritating :)
>> 
>> Imtiaz
>> 
>> ----- Original Message ----- From: "Imtiaz Ahmed H E" <in.imtiaz@gmail.com>
>> To: <esme-dev@incubator.apache.org>
>> Sent: Friday, July 23, 2010 2:24 PM
>> Subject: Re: Metadata handling (was "Release planning")
>> 
>> 
>>> I have attached a fix (Esme_branches_metadata_Jira242Fix.diff) to Jira ticket
242, with the following change
>>> 
>>> The line,
>>> 
>>> lazy val metadata: String = originalXml \ "metadata" text
>>> 
>>> commented and replaced by the line
>>> 
>>> lazy val metadata: String = (originalXml \ "metadata").toString
>>> 
>>> See my earlier mail to esme-dev for results from the fix.
>>> 
>>> The fix is only for the metadata branch in the SVN repo directory 'branches'.
>>> 
>>> This is pending further evaluation by esme-dev and improvements etc...
>>> 
>>> Imtiaz
>>> 
>>> ----- Original Message ----- From: "Ethan Jewett" <esjewett@gmail.com>
>>> To: <esme-dev@incubator.apache.org>
>>> Sent: Wednesday, July 21, 2010 7:51 PM
>>> Subject: Re: Metadata handling (was "Release planning")
>>> 
>>> 
>>>> Sounds good to me, at least until we can figure out what the next
>>>> step/problem is on the metadata front. Just ping the list when you've
>>>> uploaded the patch.
>>>> 
>>>> You don't happen to have any JDBC/JNDI and/or Lucene experience do
>>>> you? I'm totally stumped on the issue with search when we have it run
>>>> using the database instead of the filesystem (ESME-205).
>>>> 
>>>> Ethan
>>>> 
>>>> On Wed, Jul 21, 2010 at 4:06 PM, Imtiaz Ahmed H E <in.imtiaz@gmail.com>
wrote:
>>>>> Oops! I think I wrote that in a hurry, not even thinking!!
>>>>> 
>>>>> Actually, I think I should just submit a patch to the metadata branch
with
>>>>> my current change and move on to other Jira tickets while we wait for
>>>>> consensus/finalization.
>>>>> 
>>>>> Imtiaz
>>>>> 
>>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E" <in.imtiaz@gmail.com>
>>>>> To: <esme-dev@incubator.apache.org>
>>>>> Sent: Wednesday, July 21, 2010 7:30 PM
>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>> 
>>>>> 
>>>>>> How about it if I just store the metadata as a literal string and
return
>>>>>> it as such for now and submit a patch to the metadata branch with
that and
>>>>>> move on to another Jira ticket ?
>>>>>> 
>>>>>> Then, when we all agree on something final for the output format
and
>>>>>> representation of message metadata we can look at further work on
ESME-242.
>>>>>> 
>>>>>> Imtiaz
>>>>>> 
>>>>>> ----- Original Message ----- From: "Ethan Jewett" <esjewett@gmail.com>
>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>> Sent: Wednesday, July 21, 2010 7:01 PM
>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>> 
>>>>>> 
>>>>>> Imtiaz,
>>>>>> 
>>>>>> Excellent! That's good progress.
>>>>>> 
>>>>>> Any idea what the API response is still getting HTML entity-encoded?
>>>>>> That's what all of those &lt; and &gt; things are. Does the
API do
>>>>>> this for all of its responses or is it just the metadata ones? We
>>>>>> probably don't need to be entity-encoding since we're returning XML,
>>>>>> not HTML.
>>>>>> 
>>>>>> Also, I just created a branch called "metadata". Can you work against
>>>>>> this branch and feel free to submit patches against that branch to
the
>>>>>> Jira ticket. Vassil, Anne, or I can then apply the patches directly
to
>>>>>> the branch without the level of review that would be needed for a
>>>>>> patch against trunk. This should help allow us to try out your changes
>>>>>> and come to consensus and understanding more quickly.
>>>>>> 
>>>>>> Thanks!
>>>>>> Ethan
>>>>>> 
>>>>>> On Wed, Jul 21, 2010 at 7:20 AM, Imtiaz Ahmed H E <in.imtiaz@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> By the way, all I did was,
>>>>>>> 
>>>>>>> in Message.scala,
>>>>>>> replace,
>>>>>>> 
>>>>>>> lazy val metadata: String = originalXml \ "metadata" text
>>>>>>> 
>>>>>>> by...
>>>>>>> 
>>>>>>> lazy val metadata: String = (originalXml \ "metadata").toString
>>>>>>> 
>>>>>>> to get the results you see in my previous mail...
>>>>>>> 
>>>>>>> Imtiaz
>>>>>>> 
>>>>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E"
>>>>>>> <in.imtiaz@gmail.com>
>>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>>> Sent: Wednesday, July 21, 2010 10:34 AM
>>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>>> 
>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> In the absence of decisive consensus yet, I made a small
change in
>>>>>>>> Message.scala as a first step in fixing ESME-242.
>>>>>>>> Here's curl output for the Jira ticket examples given there
by
>>>>>>>> Ethan...There is maybe obvious unacceptable stuff here, but
let me
>>>>>>>> know...
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $ curl --dump-header headers -d "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL"
>>>>>>>> http:/
>>>>>>>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> <api><session><user><id>3</id><nickname>imtiaz2</nickname><image>None</image><wh
>>>>>>>> ole_name>I A 2 H E</whole_name></user></session></api>
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $ cat headers
>>>>>>>> HTTP/1.1 200 OK
>>>>>>>> Server: Apache-Coyote/1.1
>>>>>>>> Set-Cookie: JSESSIONID=C7B688BAF8E99B2A638EF432885E310E;
>>>>>>>> Path=/esme-server-apach
>>>>>>>> e-esme-1.0-RC1-incubating
>>>>>>>> Expires: Wed, 21 Jul 2010 04:56:51 UTC
>>>>>>>> Date: Wed, 21 Jul 2010 04:56:51 GMT
>>>>>>>> Pragma: no-cache
>>>>>>>> Cache-Control: no-cache; private; no-store
>>>>>>>> X-Lift-Version: 2.0-SNAPSHOT
>>>>>>>> Content-Type: text/xml;charset=utf-8
>>>>>>>> Content-Length: 178
>>>>>>>> 
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $ curl -b headers -d
>>>>>>>> 'message=test200&metadata=<outer><meta><metameta>Hello</me
>>>>>>>> tameta></meta><onlymeta>Meta</onlymeta></outer>'
>>>>>>>> http://localhost:8080/esme-ser
>>>>>>>> ver-apache-esme-1.0-RC1-incubating/api2/user/messages
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> <api><message>
>>>>>>>> <id>28</id>
>>>>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date>
>>>>>>>> <source>api2</source>
>>>>>>>> <body>test200</body>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> <metadata>&lt;metadata&gt;&lt;outer&gt;&lt;meta&gt;&lt;metameta&gt;Hello&lt;/m
>>>>>>>> 
>>>>>>>> 
>>>>>>>> etameta&gt;&lt;/meta&gt;&lt;onlymeta&gt;Meta&lt;/onlymeta&gt;&lt;/outer&gt;&lt;/
>>>>>>>> metadata&gt;</metadata>
>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>> </message></api>
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $ curl -b headers -d
>>>>>>>> 'message=test201&metadata=<anytag>"meta":[{"place":{"place
>>>>>>>> 
>>>>>>>> 
>>>>>>>> _type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Le
>>>>>>>> t+Me+Down"}}]</anytag>'
>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-i
>>>>>>>> ncubating/api2/user/messages
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> <api><message>
>>>>>>>> <id>29</id>
>>>>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date>
>>>>>>>> <source>api2</source>
>>>>>>>> <body>test201</body>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> <metadata>&lt;metadata&gt;&lt;anytag&gt;&amp;quot;meta&amp;quot;:[{&amp;quot;p
>>>>>>>> 
>>>>>>>> 
>>>>>>>> lace&amp;quot;:{&amp;quot;place_type&amp;quot;:&amp;quot;city&amp;quot;,&amp;quo
>>>>>>>> t;region&amp;quot;:&amp;quot;CA
>>>>>>>> &amp;quot;}},{&amp;quot;song&amp;quot;:{&amp;quo
>>>>>>>> 
>>>>>>>> 
>>>>>>>> t;artist&amp;quot;:&amp;quot;Prince&amp;quot;,&amp;quot;songtitle&amp;quot;:&amp
>>>>>>>> ;quot;Never Let Me
>>>>>>>> Down&amp;quot;}}]&lt;/anytag&gt;&lt;/metadata&gt;</metadata>
>>>>>>>> 
>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>> </message></api>
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $ curl -b headers -d
>>>>>>>> 'message=test202&metadata="meta":[{"place":{"place_type":"
>>>>>>>> 
>>>>>>>> 
>>>>>>>> city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Dow
>>>>>>>> n"}}]'
>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/us
>>>>>>>> er/messages
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> <api><message>
>>>>>>>> <id>30</id>
>>>>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date>
>>>>>>>> <source>api2</source>
>>>>>>>> <body>test202</body>
>>>>>>>> <metadata></metadata>
>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>> </message></api>
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $ curl -b headers
>>>>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubat
>>>>>>>> ing/api2/user/messages
>>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>>> <api><messages><message>
>>>>>>>> <id>30</id>
>>>>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date>
>>>>>>>> <source>api2</source>
>>>>>>>> <body>test202</body>
>>>>>>>> <metadata></metadata>
>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>> </message><message>
>>>>>>>> <id>29</id>
>>>>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date>
>>>>>>>> <source>api2</source>
>>>>>>>> <body>test201</body>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> <metadata>&lt;metadata&gt;&lt;anytag&gt;&amp;quot;meta&amp;quot;:[{&amp;quot;p
>>>>>>>> 
>>>>>>>> 
>>>>>>>> lace&amp;quot;:{&amp;quot;place_type&amp;quot;:&amp;quot;city&amp;quot;,&amp;quo
>>>>>>>> t;region&amp;quot;:&amp;quot;CA
>>>>>>>> &amp;quot;}},{&amp;quot;song&amp;quot;:{&amp;quo
>>>>>>>> 
>>>>>>>> 
>>>>>>>> t;artist&amp;quot;:&amp;quot;Prince&amp;quot;,&amp;quot;songtitle&amp;quot;:&amp
>>>>>>>> ;quot;Never Let Me
>>>>>>>> Down&amp;quot;}}]&lt;/anytag&gt;&lt;/metadata&gt;</metadata>
>>>>>>>> 
>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>> </message><message>
>>>>>>>> <id>28</id>
>>>>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date>
>>>>>>>> <source>api2</source>
>>>>>>>> <body>test200</body>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> <metadata>&lt;metadata&gt;&lt;outer&gt;&lt;meta&gt;&lt;metameta&gt;Hello&lt;/m
>>>>>>>> 
>>>>>>>> 
>>>>>>>> etameta&gt;&lt;/meta&gt;&lt;onlymeta&gt;Meta&lt;/onlymeta&gt;&lt;/outer&gt;&lt;/
>>>>>>>> metadata&gt;</metadata>
>>>>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>>>>> </message></messages></api>
>>>>>>>> 
>>>>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>>>>> $
>>>>>>>> 
>>>>>>>> ----- Original Message ----- From: "Vassil Dichev" <vdichev@apache.org>
>>>>>>>> To: <esme-dev@incubator.apache.org>
>>>>>>>> Sent: Tuesday, July 20, 2010 11:27 AM
>>>>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> I'd prefer option 1 (separate attribute from text).
Within this
>>>>>>>>>> separate attribute there is the question of how data
is
>>>>>>>>>> stored/represented. I'm ok with either raw string
or a tuple-based
>>>>>>>>>> structure like Twitter's. I kind of like the tuple
(key-value)
>>>>>>>>>> approach.
>>>>>>>>> 
>>>>>>>>> I'm not too interested in how the data is stored, because
it's fairly
>>>>>>>>> trivial to implement either way. It's currently not yet
clear to me
>>>>>>>>> what the requirements for the output format are.
>>>>>>>>> 
>>>>>>>>>> What I'm insisting on and what I was saying we got
wrong is that what
>>>>>>>>>> goes in needs to be the same as what comes out. If
it's tuple-based
>>>>>>>>>> and I send in a tuple, then I should get that tuple
(key and value)
>>>>>>>>>> back out when I request the metadata for a message.
Right now I think
>>>>>>>>>> we only get a concatenated list of values from the
metadata and
>>>>>>>>>> metaData methods and we're bound to an XML format.
>>>>>>>>> 
>>>>>>>>> I don't get it. A tuple is an abstraction which might
be expressed in
>>>>>>>>> a specific format. So what goes in is not what comes
out depending on
>>>>>>>>> the format. Let me quote the specific example Twitter
provides.
>>>>>>>>> 
>>>>>>>>> This comes in:
>>>>>>>>> 
>>>>>>>>> "annotations":
>>>>>>>>> [{"type":{"another_attribute":"value", "attribute":"value"}}]
>>>>>>>>> 
>>>>>>>>> This comes out:
>>>>>>>>> 
>>>>>>>>> <annotations type="array">
>>>>>>>>> <annotation>
>>>>>>>>> <type>foo</type>
>>>>>>>>> <attributes>
>>>>>>>>> <attribute>
>>>>>>>>> <name>bar</name>
>>>>>>>>> <value>baz</value>
>>>>>>>>> </attribute>
>>>>>>>>> </attributes>
>>>>>>>>> </annotation>
>>>>>>>>> </annotations>
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> As far as requiring a particular format, I think
the internal format
>>>>>>>>>> should be either a raw string or a immutable hashmap
with raw strings
>>>>>>>>>> as keys and values. We can handle converting this
to XML or JSON in
>>>>>>>>>> the API or view code.
>>>>>>>>> 
>>>>>>>>> Again, it's not so interesting what's internally there,
let's just
>>>>>>>>> treat it as a black box. What I want to know is, do we
want to have
>>>>>>>>> for instance XML in a JSON reply returned:
>>>>>>>>> 
>>>>>>>>> "annotations":
>>>>>>>>> [{"type":{"<attributes> <attribute> <name>bar</name>
>>>>>>>>> <value>baz</value> </attribute> </attributes>"}}]
>>>>>>>>> 
>>>>>>>>> or, inversely, do we want JSON inside an XML reply? Something
like:
>>>>>>>>> 
>>>>>>>>> <annotations type="array">
>>>>>>>>> <annotation>
>>>>>>>>> <type>foo</type>
>>>>>>>>> <attributes>
>>>>>>>>> {"another_attribute":"value", "attribute":"value"}
>>>>>>>>> </attributes>
>>>>>>>>> </annotation>
>>>>>>>>> </annotations>
>>>>>>>>> 
>>>>>>>>> because the latter will need to be escaped.
>>>>>>>>> 
>>>>>>>>> Sorry for being too dense, but in any case, we either
have to escape
>>>>>>>>> the metadata or we have to transform the structure to
XML/JSON when we
>>>>>>>>> return it back to the user. None of these is "what goes
in needs to be
>>>>>>>>> the same as what comes out"
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
> 


Mime
View raw message