Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1C0C51151B for ; Tue, 6 May 2014 17:01:11 +0000 (UTC) Received: (qmail 33581 invoked by uid 500); 6 May 2014 16:10:49 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 33533 invoked by uid 500); 6 May 2014 16:10:48 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 33514 invoked by uid 99); 6 May 2014 16:10:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 May 2014 16:10:47 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of lewis.mcgibbney@gmail.com designates 209.85.216.47 as permitted sender) Received: from [209.85.216.47] (HELO mail-qa0-f47.google.com) (209.85.216.47) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 May 2014 16:10:44 +0000 Received: by mail-qa0-f47.google.com with SMTP id s7so3115404qap.20 for ; Tue, 06 May 2014 09:10:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=az7dXYeirfAjgTdCRqBtKYvBAo6LkhiCvwc65gIjabU=; b=vkUAjPSEaBSS/tpYqjZRt6zdOyfYdLd0OTo5FrYJwlOgO3NAe2nGg2n6LQc2/yHrwU jo2IOiiFYPQxIdbyUyvDAL6Qc1V+mPefHSyUU5xgwJmYHQo1P9GPAnCyUAs7Db4Jq4wb 88klPorcjWV1/frVsDhOlBTzbHPH7V+xTJkj7bl+cf8dJTBKgApMhScJ82QOvzQofPOg 11TBeyKpHVdR0RcAOJmzihDvlZAH5BmKFCDNV81ikcJo3i0flCoegd6HT59pbA8V8quS HWpa8trUQcGcWMYCcotNIJGL8/Lis1lvghbQyXvT6beoOozHVzLgb9z7OqZwhWMYN988 WpWw== MIME-Version: 1.0 X-Received: by 10.224.172.2 with SMTP id j2mr56960668qaz.83.1399392621114; Tue, 06 May 2014 09:10:21 -0700 (PDT) Received: by 10.96.224.99 with HTTP; Tue, 6 May 2014 09:10:21 -0700 (PDT) Received: by 10.96.224.99 with HTTP; Tue, 6 May 2014 09:10:21 -0700 (PDT) In-Reply-To: References: <6E6EB827-E481-427A-B01C-52456411D6C8@disney.com> Date: Tue, 6 May 2014 09:10:21 -0700 Message-ID: Subject: Re: Dynamic Package/namespace naming From: Lewis John Mcgibbney To: user@avro.apache.org Content-Type: multipart/alternative; boundary=001a11c2b73e153edd04f8bd7cc5 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2b73e153edd04f8bd7cc5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Sean thanks for the additional info. This is really helpful. Some of these constraints are not what I would prefer... and of course mean more work. Powers that be request them ;) Thanks On May 6, 2014 9:06 AM, "Sean Busbey" wrote: > Hi Lewis, > > Well, I'm not sure what you're going to gain by isolating each file into > its own namespace. > > However, one way you can achieve that is to take the information you want > included from before and then create a hash of it. You can use the hex > representation of the hash to give you something that > > * meets Avro's requirements for namespace names > * uniquely identifies each namespace based on experiment, channel, and > timestamp > > If you need the file to also contain the information that went into makin= g > that hash, you can attach it as Avro-ignored metadata on the schema. > > -Sean > > > On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney < > lewis.mcgibbney@gmail.com> wrote: > >> Hi Sean, >> Arggggh... >> Maybe I can propose a change to the namespace semantics in my project. >> I've been given the following brief. >> Data files should have a logical namespace structure which can be >> globally unified across multiple instruments and receiver platforms into= a >> common namespace on the filesystem. >> I have a whole list of other data representation constraints and Avro >> satisfies them all. This one is the only 'issue'. >> Thanks for your help. >> Lewis >> On May 5, 2014 4:21 PM, "Sean Busbey" wrote: >> >>> Hi Lewis! >>> >>> Avro namespaces don't allow the characters '/', ':', or '-'. So your >>> specific example would not work. The allowed characters for a namespac= e >>> are defined in the Avro spec[1]. >>> >>> It would help if you could clarify what purpose namespacing serves in >>> the system. >>> >>> [1]: http://avro.apache.org/docs/current/spec.html#Names >>> >>> >>> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney < >>> lewis.mcgibbney@gmail.com> wrote: >>> >>>> Hi Charles >>>> Thanks for reply. >>>> Re: more info >>>> Say I want my namespace to be >>>> >>>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/ >>>> >>>> Where all constituent parts of namespace separated by '/' are >>>> dynamic... is this possible via builder api? >>>> Thanks again >>>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" < >>>> Charles.X.Pritchard.-ND@disney.com> wrote: >>>> >>>>> Need more info =E2=80=94 you can use the Schema builder to do anythin= g you >>>>> like at runtime, >>>>> Schema.createRecord and setFields. >>>>> >>>>> >>>>> >>>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney < >>>>> lewis.mcgibbney@gmail.com> wrote: >>>>> >>>>> Hi Folks, >>>>> I'm trying to propose Afro for a project I've been drafted on to. >>>>> Namespace declarations are important... and a requirement is that >>>>> namespaces are dynamic in nature... preferably even decided at runtim= e. >>>>> Is this possible in Avro? >>>>> Thanks >>>>> Lewis >>>>> >>>>> >>>>> >>> > > > -- > Sean > --001a11c2b73e153edd04f8bd7cc5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hi Sean thanks for the additional info. This is really helpf= ul.
Some of these constraints are not what I would prefer... and of course mean= more work. Powers that be request them ;)
Thanks

On May 6, 2014 9:06 AM, "Sean Busbey" = <busbey@cloudera.com> wrot= e:
Hi Lewis,

Well, I'm not sure what y= ou're going to gain by isolating each file into its own namespace.

However, one way you can achieve that is to take the i= nformation you want included from before and then create a hash of it. You = can use the hex representation of the hash to give you something that

* meets Avro's requirements for namespace names
* uniquely identifies each namespace based on experiment, channel, a= nd timestamp

If you need the file to also contain = the information that went into making that hash, you can attach it as Avro-= ignored metadata on the schema.

-Sean


On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com> wrote:

Hi Sean,
Arggggh...
Maybe I can propose a change to the namespace semantics in my project.
I've been given the following brief.
Data files should have a logical namespace structure which can be globally = unified across multiple instruments and receiver platforms into a common na= mespace on the filesystem.
I have a whole list of other data representation constraints and Avro sati= sfies them all. This one is the only 'issue'.
Thanks for your help.
Lewis

On May 5, 2014 4:21 PM, "Sean Busbey" = <busbey= +lists@cloudera.com> wrote:
Hi Lewis!

Avro namespaces don't all= ow the characters '/', ':', or '-'. So your specifi= c example would not work. =C2=A0The allowed characters for a namespace are = defined in the Avro spec[1].

It would help if you could clarify what purpose namespa= cing serves in the system.



On Mon,= May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <lewis.mcgibbney@gma= il.com> wrote:

Hi Charles
Thanks for reply.
Re: more info
Say I want my namespace to be

experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29= /

Where all constituent parts of namespace separated by '/= ' are dynamic... is this possible via builder api?
Thanks again

On May 5, 2014 4:04 PM, "Pritchard, Charles= X. -ND" <Charles.X.Pritchard.-ND@disney.com> wrote:
Need more info =E2=80=94 you can use th= e Schema builder to do anything you like at runtime,
Schema.createRecor= d and setFields.



On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <lewis.mcgibbney@gmail.com> = wrote:

Hi Folks,
I'm trying to propose Afro for a project I've been drafted on to. Namespace declarations are important... and a requirement is that namespace= s are dynamic in nature... preferably even decided at runtime.
Is this possible in Avro?
Thanks
Lewis






--
=
Sean
--001a11c2b73e153edd04f8bd7cc5--