avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Dynamic Package/namespace naming
Date Tue, 06 May 2014 15:33:29 GMT
Hi Lewis,

Well, I'm not sure what you're going to gain by isolating each file into
its own namespace.

However, one way you can achieve that is to take the information you want
included from before and then create a hash of it. You can use the hex
representation of the hash to give you something that

* meets Avro's requirements for namespace names
* uniquely identifies each namespace based on experiment, channel, and
timestamp

If you need the file to also contain the information that went into making
that hash, you can attach it as Avro-ignored metadata on the schema.

-Sean


On Mon, May 5, 2014 at 6:49 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi Sean,
> Arggggh...
> Maybe I can propose a change to the namespace semantics in my project.
> I've been given the following brief.
> Data files should have a logical namespace structure which can be globally
> unified across multiple instruments and receiver platforms into a common
> namespace on the filesystem.
> I have a whole list of other data representation constraints and Avro
> satisfies them all. This one is the only 'issue'.
> Thanks for your help.
> Lewis
> On May 5, 2014 4:21 PM, "Sean Busbey" <busbey+lists@cloudera.com> wrote:
>
>> Hi Lewis!
>>
>> Avro namespaces don't allow the characters '/', ':', or '-'. So your
>> specific example would not work.  The allowed characters for a namespace
>> are defined in the Avro spec[1].
>>
>> It would help if you could clarify what purpose namespacing serves in the
>> system.
>>
>> [1]: http://avro.apache.org/docs/current/spec.html#Names
>>
>>
>> On Mon, May 5, 2014 at 6:14 PM, Lewis John Mcgibbney <
>> lewis.mcgibbney@gmail.com> wrote:
>>
>>> Hi Charles
>>> Thanks for reply.
>>> Re: more info
>>> Say I want my namespace to be
>>>
>>> experiment_name-{timestamp}/channel_name/2014-05-05T16:13:29/
>>>
>>> Where all constituent parts of namespace separated by '/' are dynamic...
>>> is this possible via builder api?
>>> Thanks again
>>> On May 5, 2014 4:04 PM, "Pritchard, Charles X. -ND" <
>>> Charles.X.Pritchard.-ND@disney.com> wrote:
>>>
>>>> Need more info — you can use the Schema builder to do anything you like
>>>> at runtime,
>>>> Schema.createRecord and setFields.
>>>>
>>>>
>>>>
>>>> On May 5, 2014, at 3:54 PM, Lewis John Mcgibbney <
>>>> lewis.mcgibbney@gmail.com> wrote:
>>>>
>>>> Hi Folks,
>>>> I'm trying to propose Afro for a project I've been drafted on to.
>>>> Namespace declarations are important... and a requirement is that
>>>> namespaces are dynamic in nature... preferably even decided at runtime.
>>>> Is this possible in Avro?
>>>> Thanks
>>>> Lewis
>>>>
>>>>
>>>>
>>


-- 
Sean

Mime
View raw message