hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Khalil Honsali" <k.hons...@gmail.com>
Subject Re: simple architectural question
Date Wed, 26 Sep 2007 22:26:11 GMT
Thanks for the feedback.

I think that the FSNameSystem is the object used by NameNode to store
metadata information, which are as straighforward as the ones used by the
direct alternative to hadoop (PVFS2) [see 1]
Though I think that being -Java- OOP, hadoop can serve higher level and more
complex frameworks.
My idea for metadata is not clear yet, but agrees with Mrs. Kate's
viewpoint, hopefully to support for security/permissions, maybe consistency
too, and why not some more advanced/experimental features.
As a research subject, I am interested in the possibility of a metadata DBMS

Your suggestions/opinions are very much welcome.

K. Honsali

[1: PVFS2 metadata] from docs:
*metadata*data about data. in the UNIX sense, such things as owner, group,
permissions, timestamps, sizes. in the PVFS sense, also distribution

On 27/09/2007, kate rhodes <masukomi@gmail.com > wrote:
> I was assuming Khalil was speaking of more user specified meta-data
> not your standard simple file/permissions meta-data that it would be
> nigh-impossible to have a filesystem without.
> I could see great value in having each file contain info about the
> process that generated, or a hint as to what it contains for mappers
> to use. Without meta-data like that there's no way for a mapper to
> differentiate one file in a folder from another.... the filesystem
> becomes a defacto metadata store ... you have to describe/infer the
> file's contents by where you put them.  There's no way to inspect a
> file's metadata to see if it's from a "foo" type MR or a "bar" type
> MR. Does this file contain data about users or data about logs or data
> about data? Only way to tell is where it was stuck in the filesystem.
> Wouldn't it be much better if you could just store some meta-data with
> the file? Even if it was totally simplistic like e-mail headers...
> - kate = masukomi
> http://weblog.masukomi.org/
> On 9/26/07, Ted Dunning <tdunning@veoh.com> wrote:
> >
> > The blocks in a file are stored on the namenode (clearly these are file
> > meta-data).  The namenode also stores the number of replications.
> >
> > There is a CRC for each block that is stored by the datanodes.
> >
> > Based on IRC chatter, there will soon be user and group permission meta
> > data.
> >
> >
> > On 9/26/07 2:01 PM, "kate rhodes" <masukomi@gmail.com> wrote:
> >
> > > My understanding is that there is no file meta-data beyond it's name
> > > and the directory that it lives in.
> >
> >

Honsali Khalil − 本査理 カリル
Academic>Japan>NIT>Grad. Sc. Eng.>Dept. CS>Hiroshi&Tsumura Lab.
+81 (zero-)eight-zero 5134 8119
k.honsali@ezweb.ne.jp (instant reply mail)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message