Return-Path: Delivered-To: apmail-incubator-esme-dev-archive@minotaur.apache.org Received: (qmail 21110 invoked from network); 23 Jul 2010 08:54:44 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 23 Jul 2010 08:54:44 -0000 Received: (qmail 50086 invoked by uid 500); 23 Jul 2010 08:54:44 -0000 Delivered-To: apmail-incubator-esme-dev-archive@incubator.apache.org Received: (qmail 50026 invoked by uid 500); 23 Jul 2010 08:54:43 -0000 Mailing-List: contact esme-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: esme-dev@incubator.apache.org Delivered-To: mailing list esme-dev@incubator.apache.org Received: (qmail 50017 invoked by uid 99); 23 Jul 2010 08:54:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 08:54:41 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of in.imtiaz@gmail.com designates 209.85.214.175 as permitted sender) Received: from [209.85.214.175] (HELO mail-iw0-f175.google.com) (209.85.214.175) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 08:54:31 +0000 Received: by iwn34 with SMTP id 34so8990298iwn.6 for ; Fri, 23 Jul 2010 01:54:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to:references :subject:date:mime-version:content-type:content-transfer-encoding :x-priority:x-msmail-priority:x-mailer:x-mimeole; bh=/onKRSsuDb3aeSZg5tGtNRqCwRXxAXLcmE97W3s7KAg=; b=OXtyAtLa7z5WnfFgt2DFLJRflj3AXKzeCECxQj7Bef6yBmwo+4+QAokR1oIKRvKSv8 2EGOhdRQpp+mPV3r+YQZp/nphXIRpd7POoZHq6UajWqEhQYnjERvWfZTvwH19TU8chyk 2O4br3puapwOgoVuXzqxg002OiYmhTsSiGh8Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:references:subject:date:mime-version :content-type:content-transfer-encoding:x-priority:x-msmail-priority :x-mailer:x-mimeole; b=UicmpEkA+Bg9UXwwhRPPqVidskG1tv4hjTVn0Q9Hibq/oBia2tmNwnc4jVGj6Gfc3V xxdG9eqXYMAMZz381SKb4mGe0g/BK728Poiqj8fdJr0nuKCJLOC/bue8sX/Yzr4ggmRD YjTp3eUP7Mdu+KTfmtN/b4zGSutyv3E4btHRQ= Received: by 10.231.145.16 with SMTP id b16mr3252439ibv.198.1279875250805; Fri, 23 Jul 2010 01:54:10 -0700 (PDT) Received: from imtiaz20100131 ([122.167.121.108]) by mx.google.com with ESMTPS id r3sm18090ibk.1.2010.07.23.01.54.08 (version=SSLv3 cipher=RC4-MD5); Fri, 23 Jul 2010 01:54:10 -0700 (PDT) Message-ID: <8F1AB60E70BE4479BACA3FD4EB3318C7@imtiaz20100131> From: "Imtiaz Ahmed H E" To: References: <9F2AD1E468B34A0696D5BD539B43E04C@imtiaz20100131> Subject: Re: Metadata handling (was "Release planning") Date: Fri, 23 Jul 2010 14:24:04 +0530 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931 X-Virus-Checked: Checked by ClamAV on apache.org I have attached a fix (Esme_branches_metadata_Jira242Fix.diff) to Jira ticket 242, with the following change The line, lazy val metadata: String = originalXml \ "metadata" text commented and replaced by the line lazy val metadata: String = (originalXml \ "metadata").toString See my earlier mail to esme-dev for results from the fix. The fix is only for the metadata branch in the SVN repo directory 'branches'. This is pending further evaluation by esme-dev and improvements etc... Imtiaz ----- Original Message ----- From: "Ethan Jewett" To: Sent: Wednesday, July 21, 2010 7:51 PM Subject: Re: Metadata handling (was "Release planning") > Sounds good to me, at least until we can figure out what the next > step/problem is on the metadata front. Just ping the list when you've > uploaded the patch. > > You don't happen to have any JDBC/JNDI and/or Lucene experience do > you? I'm totally stumped on the issue with search when we have it run > using the database instead of the filesystem (ESME-205). > > Ethan > > On Wed, Jul 21, 2010 at 4:06 PM, Imtiaz Ahmed H E > wrote: >> Oops! I think I wrote that in a hurry, not even thinking!! >> >> Actually, I think I should just submit a patch to the metadata branch >> with >> my current change and move on to other Jira tickets while we wait for >> consensus/finalization. >> >> Imtiaz >> >> ----- Original Message ----- From: "Imtiaz Ahmed H E" >> >> To: >> Sent: Wednesday, July 21, 2010 7:30 PM >> Subject: Re: Metadata handling (was "Release planning") >> >> >>> How about it if I just store the metadata as a literal string and return >>> it as such for now and submit a patch to the metadata branch with that >>> and >>> move on to another Jira ticket ? >>> >>> Then, when we all agree on something final for the output format and >>> representation of message metadata we can look at further work on >>> ESME-242. >>> >>> Imtiaz >>> >>> ----- Original Message ----- From: "Ethan Jewett" >>> To: >>> Sent: Wednesday, July 21, 2010 7:01 PM >>> Subject: Re: Metadata handling (was "Release planning") >>> >>> >>> Imtiaz, >>> >>> Excellent! That's good progress. >>> >>> Any idea what the API response is still getting HTML entity-encoded? >>> That's what all of those < and > things are. Does the API do >>> this for all of its responses or is it just the metadata ones? We >>> probably don't need to be entity-encoding since we're returning XML, >>> not HTML. >>> >>> Also, I just created a branch called "metadata". Can you work against >>> this branch and feel free to submit patches against that branch to the >>> Jira ticket. Vassil, Anne, or I can then apply the patches directly to >>> the branch without the level of review that would be needed for a >>> patch against trunk. This should help allow us to try out your changes >>> and come to consensus and understanding more quickly. >>> >>> Thanks! >>> Ethan >>> >>> On Wed, Jul 21, 2010 at 7:20 AM, Imtiaz Ahmed H E >>> wrote: >>>> >>>> By the way, all I did was, >>>> >>>> in Message.scala, >>>> replace, >>>> >>>> lazy val metadata: String = originalXml \ "metadata" text >>>> >>>> by... >>>> >>>> lazy val metadata: String = (originalXml \ "metadata").toString >>>> >>>> to get the results you see in my previous mail... >>>> >>>> Imtiaz >>>> >>>> ----- Original Message ----- From: "Imtiaz Ahmed H E" >>>> >>>> To: >>>> Sent: Wednesday, July 21, 2010 10:34 AM >>>> Subject: Re: Metadata handling (was "Release planning") >>>> >>>> >>>>> Hi, >>>>> >>>>> In the absence of decisive consensus yet, I made a small change in >>>>> Message.scala as a first step in fixing ESME-242. >>>>> Here's curl output for the Jira ticket examples given there by >>>>> Ethan...There is maybe obvious unacceptable stuff here, but let me >>>>> know... >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ curl --dump-header headers -d >>>>> "token=HEZTQKM525SAMIPN4EDVRUOGHI40AKBL" >>>>> http:/ >>>>> /localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/session >>>>> >>>>> >>>>> >>>>> 3imtiaz2None>>>> ole_name>I A 2 H E >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ cat headers >>>>> HTTP/1.1 200 OK >>>>> Server: Apache-Coyote/1.1 >>>>> Set-Cookie: JSESSIONID=C7B688BAF8E99B2A638EF432885E310E; >>>>> Path=/esme-server-apach >>>>> e-esme-1.0-RC1-incubating >>>>> Expires: Wed, 21 Jul 2010 04:56:51 UTC >>>>> Date: Wed, 21 Jul 2010 04:56:51 GMT >>>>> Pragma: no-cache >>>>> Cache-Control: no-cache; private; no-store >>>>> X-Lift-Version: 2.0-SNAPSHOT >>>>> Content-Type: text/xml;charset=utf-8 >>>>> Content-Length: 178 >>>>> >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ curl -b headers -d >>>>> 'message=test200&metadata=Hello>>>> tameta>Meta' >>>>> http://localhost:8080/esme-ser >>>>> ver-apache-esme-1.0-RC1-incubating/api2/user/messages >>>>> >>>>> >>>>> 28 >>>>> Wed, 21 Jul 2010 04:57:05 UTC >>>>> api2 >>>>> test200 >>>>> >>>>> >>>>> >>>>> <metadata><outer><meta><metameta>Hello</m >>>>> >>>>> >>>>> etameta></meta><onlymeta>Meta</onlymeta></outer></ >>>>> metadata> >>>>> imtiaz23 >>>>> >>>>> >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ curl -b headers -d >>>>> 'message=test201&metadata="meta":[{"place":{"place >>>>> >>>>> >>>>> _type":"city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Le >>>>> t+Me+Down"}}]' >>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-i >>>>> ncubating/api2/user/messages >>>>> >>>>> >>>>> 29 >>>>> Wed, 21 Jul 2010 04:57:39 UTC >>>>> api2 >>>>> test201 >>>>> >>>>> >>>>> >>>>> <metadata><anytag>&quot;meta&quot;:[{&quot;p >>>>> >>>>> >>>>> lace&quot;:{&quot;place_type&quot;:&quot;city&quot;,&quo >>>>> t;region&quot;:&quot;CA >>>>> &quot;}},{&quot;song&quot;:{&quo >>>>> >>>>> >>>>> t;artist&quot;:&quot;Prince&quot;,&quot;songtitle&quot;:& >>>>> ;quot;Never Let Me >>>>> Down&quot;}}]</anytag></metadata> >>>>> >>>>> imtiaz23 >>>>> >>>>> >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ curl -b headers -d >>>>> 'message=test202&metadata="meta":[{"place":{"place_type":" >>>>> >>>>> >>>>> city","region":"CA+"}},{"song":{"artist":"Prince","songtitle":"Never+Let+Me+Dow >>>>> n"}}]' >>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubating/api2/us >>>>> er/messages >>>>> >>>>> >>>>> 30 >>>>> Wed, 21 Jul 2010 04:58:02 UTC >>>>> api2 >>>>> test202 >>>>> >>>>> imtiaz23 >>>>> >>>>> >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ curl -b headers >>>>> http://localhost:8080/esme-server-apache-esme-1.0-RC1-incubat >>>>> ing/api2/user/messages >>>>> >>>>> >>>>> 30 >>>>> Wed, 21 Jul 2010 04:58:02 UTC >>>>> api2 >>>>> test202 >>>>> >>>>> imtiaz23 >>>>> >>>>> >>>>> 29 >>>>> Wed, 21 Jul 2010 04:57:39 UTC >>>>> api2 >>>>> test201 >>>>> >>>>> >>>>> >>>>> <metadata><anytag>&quot;meta&quot;:[{&quot;p >>>>> >>>>> >>>>> lace&quot;:{&quot;place_type&quot;:&quot;city&quot;,&quo >>>>> t;region&quot;:&quot;CA >>>>> &quot;}},{&quot;song&quot;:{&quo >>>>> >>>>> >>>>> t;artist&quot;:&quot;Prince&quot;,&quot;songtitle&quot;:& >>>>> ;quot;Never Let Me >>>>> Down&quot;}}]</anytag></metadata> >>>>> >>>>> imtiaz23 >>>>> >>>>> >>>>> 28 >>>>> Wed, 21 Jul 2010 04:57:05 UTC >>>>> api2 >>>>> test200 >>>>> >>>>> >>>>> >>>>> <metadata><outer><meta><metameta>Hello</m >>>>> >>>>> >>>>> etameta></meta><onlymeta>Meta</onlymeta></outer></ >>>>> metadata> >>>>> imtiaz23 >>>>> >>>>> >>>>> >>>>> imtiaz@imtiaz-20100131 /cygdrive/d/temp >>>>> $ >>>>> >>>>> ----- Original Message ----- From: "Vassil Dichev" >>>>> >>>>> To: >>>>> Sent: Tuesday, July 20, 2010 11:27 AM >>>>> Subject: Re: Metadata handling (was "Release planning") >>>>> >>>>> >>>>>>> I'd prefer option 1 (separate attribute from text). Within this >>>>>>> separate attribute there is the question of how data is >>>>>>> stored/represented. I'm ok with either raw string or a tuple-based >>>>>>> structure like Twitter's. I kind of like the tuple (key-value) >>>>>>> approach. >>>>>> >>>>>> I'm not too interested in how the data is stored, because it's fairly >>>>>> trivial to implement either way. It's currently not yet clear to me >>>>>> what the requirements for the output format are. >>>>>> >>>>>>> What I'm insisting on and what I was saying we got wrong is that >>>>>>> what >>>>>>> goes in needs to be the same as what comes out. If it's tuple-based >>>>>>> and I send in a tuple, then I should get that tuple (key and value) >>>>>>> back out when I request the metadata for a message. Right now I >>>>>>> think >>>>>>> we only get a concatenated list of values from the metadata and >>>>>>> metaData methods and we're bound to an XML format. >>>>>> >>>>>> I don't get it. A tuple is an abstraction which might be expressed in >>>>>> a specific format. So what goes in is not what comes out depending on >>>>>> the format. Let me quote the specific example Twitter provides. >>>>>> >>>>>> This comes in: >>>>>> >>>>>> "annotations": >>>>>> [{"type":{"another_attribute":"value", "attribute":"value"}}] >>>>>> >>>>>> This comes out: >>>>>> >>>>>> >>>>>> >>>>>> foo >>>>>> >>>>>> >>>>>> bar >>>>>> baz >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> As far as requiring a particular format, I think the internal format >>>>>>> should be either a raw string or a immutable hashmap with raw >>>>>>> strings >>>>>>> as keys and values. We can handle converting this to XML or JSON in >>>>>>> the API or view code. >>>>>> >>>>>> Again, it's not so interesting what's internally there, let's just >>>>>> treat it as a black box. What I want to know is, do we want to have >>>>>> for instance XML in a JSON reply returned: >>>>>> >>>>>> "annotations": >>>>>> [{"type":{" bar >>>>>> baz "}}] >>>>>> >>>>>> or, inversely, do we want JSON inside an XML reply? Something like: >>>>>> >>>>>> >>>>>> >>>>>> foo >>>>>> >>>>>> {"another_attribute":"value", "attribute":"value"} >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> because the latter will need to be escaped. >>>>>> >>>>>> Sorry for being too dense, but in any case, we either have to escape >>>>>> the metadata or we have to transform the structure to XML/JSON when >>>>>> we >>>>>> return it back to the user. None of these is "what goes in needs to >>>>>> be >>>>>> the same as what comes out" >>>>> >>>> >>>> >>> >> >>