incubator-esme-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imtiaz Ahmed H E" <in.imt...@gmail.com>
Subject Re: Metadata handling (was "Release planning")
Date Wed, 21 Jul 2010 14:48:33 GMT
<<<<<<<<<<<<
> Sounds good to me, at least until we can figure out what the next
> step/problem is on the metadata front. Just ping the list when you've
> uploaded the patch.
>
>>>>>>>>>>>>

Okay.

<<<<<<<<<<<<
> You don't happen to have any JDBC/JNDI and/or Lucene experience do
> you? I'm totally stumped on the issue with search when we have it run
> using the database instead of the filesystem (ESME-205).
>>>>>>>>>>>>

No, no experience on those. Except have used JDBC for JavaDB access, in my 
last job and less than that of it with Oracle 8 years back.

I just tried out a search which had a result of one match, on my system, and 
it worked fine with no exceptions.

If I could reproduce the bug, I could comment on whether I should be 
assigned the Jira ticket...saw the exception trace in the ticket, 205, and 
the exception seems like a Scala coding problem...

Imtiaz

----- Original Message ----- 
From: "Ethan Jewett" <esjewett@gmail.com>
To: <esme-dev@incubator.apache.org>
Sent: Wednesday, July 21, 2010 7:51 PM
Subject: Re: Metadata handling (was "Release planning")


> Sounds good to me, at least until we can figure out what the next
> step/problem is on the metadata front. Just ping the list when you've
> uploaded the patch.
>
> You don't happen to have any JDBC/JNDI and/or Lucene experience do
> you? I'm totally stumped on the issue with search when we have it run
> using the database instead of the filesystem (ESME-205).
>
> Ethan
>
> On Wed, Jul 21, 2010 at 4:06 PM, Imtiaz Ahmed H E <in.imtiaz@gmail.com> 
> wrote:
>> Oops! I think I wrote that in a hurry, not even thinking!!
>>
>> Actually, I think I should just submit a patch to the metadata branch 
>> with
>> my current change and move on to other Jira tickets while we wait for
>> consensus/finalization.
>>
>> Imtiaz
>>
>> ----- Original Message ----- From: "Imtiaz Ahmed H E" 
>> <in.imtiaz@gmail.com>
>> To: <esme-dev@incubator.apache.org>
>> Sent: Wednesday, July 21, 2010 7:30 PM
>> Subject: Re: Metadata handling (was "Release planning")
>>
>>
>>> How about it if I just store the metadata as a literal string and return
>>> it as such for now and submit a patch to the metadata branch with that 
>>> and
>>> move on to another Jira ticket ?
>>>
>>> Then, when we all agree on something final for the output format and
>>> representation of message metadata we can look at further work on 
>>> ESME-242.
>>>
>>> Imtiaz
>>>
>>> ----- Original Message ----- From: "Ethan Jewett" <esjewett@gmail.com>
>>> To: <esme-dev@incubator.apache.org>
>>> Sent: Wednesday, July 21, 2010 7:01 PM
>>> Subject: Re: Metadata handling (was "Release planning")
>>>
>>>
>>> Imtiaz,
>>>
>>> Excellent! That's good progress.
>>>
>>> Any idea what the API response is still getting HTML entity-encoded?
>>> That's what all of those &lt; and &gt; things are. Does the API do
>>> this for all of its responses or is it just the metadata ones? We
>>> probably don't need to be entity-encoding since we're returning XML,
>>> not HTML.
>>>
>>> Also, I just created a branch called "metadata". Can you work against
>>> this branch and feel free to submit patches against that branch to the
>>> Jira ticket. Vassil, Anne, or I can then apply the patches directly to
>>> the branch without the level of review that would be needed for a
>>> patch against trunk. This should help allow us to try out your changes
>>> and come to consensus and understanding more quickly.
>>>
>>> Thanks!
>>> Ethan
>>>
>>> On Wed, Jul 21, 2010 at 7:20 AM, Imtiaz Ahmed H E <in.imtiaz@gmail.com>
>>> wrote:
>>>>
>>>> By the way, all I did was,
>>>>
>>>> in Message.scala,
>>>> replace,
>>>>
>>>> lazy val metadata: String = originalXml \ "metadata" text
>>>>
>>>> by...
>>>>
>>>> lazy val metadata: String = (originalXml \ "metadata").toString
>>>>
>>>> to get the results you see in my previous mail...
>>>>
>>>> Imtiaz
>>>>
>>>> ----- Original Message ----- From: "Imtiaz Ahmed H E"
>>>> <in.imtiaz@gmail.com>
>>>> To: <esme-dev@incubator.apache.org>
>>>> Sent: Wednesday, July 21, 2010 10:34 AM
>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> In the absence of decisive consensus yet, I made a small change in
>>>>> Message.scala as a first step in fixing ESME-242.
>>>>> Here's curl output for the Jira ticket examples given there by
>>>>> Ethan...There is maybe obvious unacceptable stuff here, but let me
>>>>> know...
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $ curl --dump-header headers -d 
>>>>> "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL"
>>>>> http:/
>>>>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>
>>>>>
>>>>> <api><session><user><id>3</id><nickname>imtiaz2</nickname><image>None</image><wh
>>>>> ole_name>I A 2 H E</whole_name></user></session></api>
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $ cat headers
>>>>> HTTP/1.1 200 OK
>>>>> Server: Apache-Coyote/1.1
>>>>> Set-Cookie: JSESSIONID=C7B688BAF8E99B2A638EF432885E310E;
>>>>> Path=/esme-server-apach
>>>>> e-esme-1.0-RC1-incubating
>>>>> Expires: Wed, 21 Jul 2010 04:56:51 UTC
>>>>> Date: Wed, 21 Jul 2010 04:56:51 GMT
>>>>> Pragma: no-cache
>>>>> Cache-Control: no-cache; private; no-store
>>>>> X-Lift-Version: 2.0-SNAPSHOT
>>>>> Content-Type: text/xml;charset=utf-8
>>>>> Content-Length: 178
>>>>>
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $ curl -b headers -d
>>>>> 'message=test200&metadata=<outer><meta><metameta>Hello</me
>>>>> tameta></meta><onlymeta>Meta</onlymeta></outer>'
>>>>> http://localhost:8080/esme-ser
>>>>> ver-apache-esme-1.0-RC1-incubating/api2/user/messages
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <api><message>
>>>>> <id>28</id>
>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date>
>>>>> <source>api2</source>
>>>>> <body>test200</body>
>>>>>
>>>>>
>>>>>
>>>>> <metadata>&lt;metadata&gt;&lt;outer&gt;&lt;meta&gt;&lt;metameta&gt;Hello&lt;/m
>>>>>
>>>>>
>>>>> etameta&gt;&lt;/meta&gt;&lt;onlymeta&gt;Meta&lt;/onlymeta&gt;&lt;/outer&gt;&lt;/
>>>>> metadata&gt;</metadata>
>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>> </message></api>
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $ curl -b headers -d
>>>>> 'message=test201&metadata=<anytag>"meta":[{"place":{"place
>>>>>
>>>>>
>>>>> _type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Le
>>>>> t+Me+Down"}}]</anytag>'
>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-i
>>>>> ncubating/api2/user/messages
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <api><message>
>>>>> <id>29</id>
>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date>
>>>>> <source>api2</source>
>>>>> <body>test201</body>
>>>>>
>>>>>
>>>>>
>>>>> <metadata>&lt;metadata&gt;&lt;anytag&gt;&amp;quot;meta&amp;quot;:[{&amp;quot;p
>>>>>
>>>>>
>>>>> lace&amp;quot;:{&amp;quot;place_type&amp;quot;:&amp;quot;city&amp;quot;,&amp;quo
>>>>> t;region&amp;quot;:&amp;quot;CA
>>>>> &amp;quot;}},{&amp;quot;song&amp;quot;:{&amp;quo
>>>>>
>>>>>
>>>>> t;artist&amp;quot;:&amp;quot;Prince&amp;quot;,&amp;quot;songtitle&amp;quot;:&amp
>>>>> ;quot;Never Let Me
>>>>> Down&amp;quot;}}]&lt;/anytag&gt;&lt;/metadata&gt;</metadata>
>>>>>
>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>> </message></api>
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $ curl -b headers -d
>>>>> 'message=test202&metadata="meta":[{"place":{"place_type":"
>>>>>
>>>>>
>>>>> city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Dow
>>>>> n"}}]'
>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/us
>>>>> er/messages
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <api><message>
>>>>> <id>30</id>
>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date>
>>>>> <source>api2</source>
>>>>> <body>test202</body>
>>>>> <metadata></metadata>
>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>> </message></api>
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $ curl -b headers
>>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubat
>>>>> ing/api2/user/messages
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <api><messages><message>
>>>>> <id>30</id>
>>>>> <date>Wed, 21 Jul 2010 04:58:02 UTC</date>
>>>>> <source>api2</source>
>>>>> <body>test202</body>
>>>>> <metadata></metadata>
>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>> </message><message>
>>>>> <id>29</id>
>>>>> <date>Wed, 21 Jul 2010 04:57:39 UTC</date>
>>>>> <source>api2</source>
>>>>> <body>test201</body>
>>>>>
>>>>>
>>>>>
>>>>> <metadata>&lt;metadata&gt;&lt;anytag&gt;&amp;quot;meta&amp;quot;:[{&amp;quot;p
>>>>>
>>>>>
>>>>> lace&amp;quot;:{&amp;quot;place_type&amp;quot;:&amp;quot;city&amp;quot;,&amp;quo
>>>>> t;region&amp;quot;:&amp;quot;CA
>>>>> &amp;quot;}},{&amp;quot;song&amp;quot;:{&amp;quo
>>>>>
>>>>>
>>>>> t;artist&amp;quot;:&amp;quot;Prince&amp;quot;,&amp;quot;songtitle&amp;quot;:&amp
>>>>> ;quot;Never Let Me
>>>>> Down&amp;quot;}}]&lt;/anytag&gt;&lt;/metadata&gt;</metadata>
>>>>>
>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>> </message><message>
>>>>> <id>28</id>
>>>>> <date>Wed, 21 Jul 2010 04:57:05 UTC</date>
>>>>> <source>api2</source>
>>>>> <body>test200</body>
>>>>>
>>>>>
>>>>>
>>>>> <metadata>&lt;metadata&gt;&lt;outer&gt;&lt;meta&gt;&lt;metameta&gt;Hello&lt;/m
>>>>>
>>>>>
>>>>> etameta&gt;&lt;/meta&gt;&lt;onlymeta&gt;Meta&lt;/onlymeta&gt;&lt;/outer&gt;&lt;/
>>>>> metadata&gt;</metadata>
>>>>> <author><nickname>imtiaz2</nickname><id>3</id></author>
>>>>> <tags></tags><replyto></replyto><conversation></conversation>
>>>>> </message></messages></api>
>>>>>
>>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp
>>>>> $
>>>>>
>>>>> ----- Original Message ----- From: "Vassil Dichev" 
>>>>> <vdichev@apache.org>
>>>>> To: <esme-dev@incubator.apache.org>
>>>>> Sent: Tuesday, July 20, 2010 11:27 AM
>>>>> Subject: Re: Metadata handling (was "Release planning")
>>>>>
>>>>>
>>>>>>> I'd prefer option 1 (separate attribute from text). Within this
>>>>>>> separate attribute there is the question of how data is
>>>>>>> stored/represented. I'm ok with either raw string or a tuple-based
>>>>>>> structure like Twitter's. I kind of like the tuple (key-value)
>>>>>>> approach.
>>>>>>
>>>>>> I'm not too interested in how the data is stored, because it's fairly
>>>>>> trivial to implement either way. It's currently not yet clear to
me
>>>>>> what the requirements for the output format are.
>>>>>>
>>>>>>> What I'm insisting on and what I was saying we got wrong is that

>>>>>>> what
>>>>>>> goes in needs to be the same as what comes out. If it's tuple-based
>>>>>>> and I send in a tuple, then I should get that tuple (key and
value)
>>>>>>> back out when I request the metadata for a message. Right now
I 
>>>>>>> think
>>>>>>> we only get a concatenated list of values from the metadata and
>>>>>>> metaData methods and we're bound to an XML format.
>>>>>>
>>>>>> I don't get it. A tuple is an abstraction which might be expressed
in
>>>>>> a specific format. So what goes in is not what comes out depending
on
>>>>>> the format. Let me quote the specific example Twitter provides.
>>>>>>
>>>>>> This comes in:
>>>>>>
>>>>>> "annotations":
>>>>>> [{"type":{"another_attribute":"value", "attribute":"value"}}]
>>>>>>
>>>>>> This comes out:
>>>>>>
>>>>>> <annotations type="array">
>>>>>> <annotation>
>>>>>> <type>foo</type>
>>>>>> <attributes>
>>>>>> <attribute>
>>>>>> <name>bar</name>
>>>>>> <value>baz</value>
>>>>>> </attribute>
>>>>>> </attributes>
>>>>>> </annotation>
>>>>>> </annotations>
>>>>>>
>>>>>>
>>>>>>> As far as requiring a particular format, I think the internal
format
>>>>>>> should be either a raw string or a immutable hashmap with raw

>>>>>>> strings
>>>>>>> as keys and values. We can handle converting this to XML or JSON
in
>>>>>>> the API or view code.
>>>>>>
>>>>>> Again, it's not so interesting what's internally there, let's just
>>>>>> treat it as a black box. What I want to know is, do we want to have
>>>>>> for instance XML in a JSON reply returned:
>>>>>>
>>>>>> "annotations":
>>>>>> [{"type":{"<attributes> <attribute> <name>bar</name>
>>>>>> <value>baz</value> </attribute> </attributes>"}}]
>>>>>>
>>>>>> or, inversely, do we want JSON inside an XML reply? Something like:
>>>>>>
>>>>>> <annotations type="array">
>>>>>> <annotation>
>>>>>> <type>foo</type>
>>>>>> <attributes>
>>>>>> {"another_attribute":"value", "attribute":"value"}
>>>>>> </attributes>
>>>>>> </annotation>
>>>>>> </annotations>
>>>>>>
>>>>>> because the latter will need to be escaped.
>>>>>>
>>>>>> Sorry for being too dense, but in any case, we either have to escape
>>>>>> the metadata or we have to transform the structure to XML/JSON when

>>>>>> we
>>>>>> return it back to the user. None of these is "what goes in needs
to 
>>>>>> be
>>>>>> the same as what comes out"
>>>>>
>>>>
>>>>
>>>
>>
>> 


Mime
View raw message