avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Dynamic Package/namespace naming
Date Tue, 06 May 2014 16:10:21 GMT
Hi Sean thanks for the additional info. This is really helpful.
Some of these constraints are not what I would prefer... and of course mean
more work. Powers that be request them ;)
Thanks
On May 6, 2014 9:06 AM, "Sean Busbey" <busbey@cloudera.com> wrote:

> Hi Lewis,
>
> Well, I'm not sure what you're going to gain by isolating each file into
> its own namespace.
>
> However, one way you can achieve that is to take the information you want
> included from before and then create a hash of it. You can use the hex
> representation of the hash to give you something that
>
> * meets Avro's requirements for namespace names
> * uniquely identifies each namespace based on experiment, channel, and
> timestamp
>
> If you need the file to also contain the information that went into making
> that hash, you can attach it as Avro-ignored metadata on the schema.
>
> -Sean
>
>
> On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> Hi Sean,
>> Arggggh...
>> Maybe I can propose a change to the namespace semantics in my project.
>> I've been given the following brief.
>> Data files should have a logical namespace structure which can be
>> globally unified across multiple instruments and receiver platforms into a
>> common namespace on the filesystem.
>> I have a whole list of other data representation constraints and Avro
>> satisfies them all. This one is the only 'issue'.
>> Thanks for your help.
>> Lewis
>> On May 5, 2014 4:21 PM, "Sean Busbey" <busbey+lists@cloudera.com> wrote:
>>
>>> Hi Lewis!
>>>
>>> Avro namespaces don't allow the characters '/', ':', or '-'. So your
>>> specific example would not work.  The allowed characters for a namespace
>>> are defined in the Avro spec[1].
>>>
>>> It would help if you could clarify what purpose namespacing serves in
>>> the system.
>>>
>>> [1]: http://avro.apache.org/docs/current/spec.html#Names
>>>
>>>
>>> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
>>> lewis.mcgibbney@gmail.com> wrote:
>>>
>>>> Hi Charles
>>>> Thanks for reply.
>>>> Re: more info
>>>> Say I want my namespace to be
>>>>
>>>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>>>>
>>>> Where all constituent parts of namespace separated by '/' are
>>>> dynamic... is this possible via builder api?
>>>> Thanks again
>>>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
>>>> Charles.X.Pritchard.-ND@disney.com> wrote:
>>>>
>>>>> Need more info — you can use the Schema builder to do anything you
>>>>> like at runtime,
>>>>> Schema.createRecord and setFields.
>>>>>
>>>>>
>>>>>
>>>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>>
>>>>> Hi Folks,
>>>>> I'm trying to propose Afro for a project I've been drafted on to.
>>>>> Namespace declarations are important... and a requirement is that
>>>>> namespaces are dynamic in nature... preferably even decided at runtime.
>>>>> Is this possible in Avro?
>>>>> Thanks
>>>>> Lewis
>>>>>
>>>>>
>>>>>
>>>
>
>
> --
> Sean
>

Mime
View raw message