distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Derrick <jonathan.derri...@gmail.com>
Subject Re: question about DL namespace
Date Mon, 12 Sep 2016 09:46:06 GMT
Sijie, thank you for your comments.

I'd like to make a proposal by introducing a `NamespaceResolver`.

What does a namespace resolver do? A namespace resolver is basically
resolving the log stream name into a metadata location path. Then DL knows
where to locate the metadata of a log stream. The resolver also takes the
responsibility of validating the stream name and managing the hierarchical
of streams.

A NamespaceResolver interface will look like as below:

public interface NamespaceResolver {

/** validate if the stream name is okay */

boolean validateStreamName(String streamName);

/** resolve the stream name into the location path of the metadata */

String resolveStreamPath(String streamName);


}

So a filesystem-like namespace resolver will only accept the absolute
file-like paths as the stream names and a kafka-like (what Khurrum
mentioned) namespace resolver will probably accept names like
'<stream>/<partition>'.

A namespace resolver will be added to the namespace metadata binding and
loaded via reflection.

Any thoughts? I will send out a pull request soon.

- jd


On Tue, Aug 23, 2016 at 9:07 AM, Khurrum Nasim <khurrumnasimm@gmail.com>
wrote:

> On Thu, Aug 18, 2016 at 2:30 AM, Sijie Guo <sijie@apache.org> wrote:
>
> > Jon,
> >
> > Sorry for late response. This is a very good question. Comments in line.
> >
> > Sijie
> >
> > On Monday, August 15, 2016, Jon Derrick <jonathan.derrickk@gmail.com>
> > wrote:
> >
> > > Hello all,
> > >
> > > I read the distributed log code closely. I found that the DL namespace
> > is a
> > > flat namespace. There will be a potential issue if there are a lot of
> > > streams created under a same namespace. I am very curious what are the
> > > thoughts behind that. Here are some questions:
> > >
> > > - How many streams that a namespace can support?
> >
> >
> > The maximum number of streams we have had for a single namespace is more
> > than 30k. But yup, you are right. It is limited by the number of children
> > that a znode can have.
> >
> > >
> > >
> > > It seems to be bound with
> > > the limitation on the number of children that a zookeeper znode can
> have.
> > > What's the maximum number of logs do you guys have?
> > > - Why not choose a tree representation? Then it might be easier to
> > organize
> > > streams. For example, if I want to use multiple dl streams as
> > partitions, I
> > > can just easily organize them together under same znode.
> >
> >
> > We don't want to DL to focus on partitions. We let applications decide
> how
> > to partition. So we choose a simple way to start. However, I don't think
> it
> > is necessary to be just a flat namespace. You probably already noticed
> that
> > there is another namespace implementation to support hierarchy.
> >
> > If you do like to support filesystem like namespace, I would suggest
> adding
> > a namespace type on metadata binding. So it can support different types
> of
> > namespaces. Does that meet your requirements?
> >
>
> +1 for supporting different types of namespaces. I want to organize a kafka
> topic in following format:
>
> namespace/topic/partitions : storing all the partitions
> namespace/topic/partitions/N : storing the given partition `N`
> namespace/topic/subscriptions : storing all the subscriptions
> namespace/topic/subscriptions/S : storing the information of subscription
> `S`
>
> both `namespace/topic/partitions/N` and `namespace/topic/subscriptions/S`
> are DL streams.
>
> So it would make me easier to manage the streams if I can customize
> namespace layout.
>
> - KN
>
>
> >
> >
> > > - Also if it is a tree-like namespace, it might be easier to implement
> a
> > > filesystem over the streams. Each file can be backed by one dl stream.
> In
> > > that way, I can also use DL as long term storage.
> > >
> > > Any thoughts? Appreciate your comments.
> > >
> > >
> > > --
> > > - jderrick
> > >
> >
>



-- 
- jderrick

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message