distributedlog-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Juma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DL-58) Be able to configure data placement policy per log stream
Date Wed, 19 Oct 2016 08:03:58 GMT
Jay Juma created DL-58:

             Summary: Be able to configure data placement policy per log stream
                 Key: DL-58
                 URL: https://issues.apache.org/jira/browse/DL-58
             Project: DistributedLog
          Issue Type: Improvement
          Components: distributedlog-core
            Reporter: Jay Juma


My question is "is it possible to create a log stream on a global cluster
that just write to bookies within same datacenter".

I had two use cases, one is for database replication. for example, if there
are 2 datacenters A and B. A global dl cluster is setup over A and B. The
database cluster in A will write updates into a (some) global log
stream(s), and the database in B will tail-read those streams and apply
changes. I think this is a very typical use case of DistributedLog, right?

There is another use case, it is just used for replication within one
datacenter. It doesn't need to replicate to the other datacenter. We want
to share the DL cluster for these two use cases. Is there a way to achieve
that? by tuning the data placement policy for individual streams?

Answers from [~sijie@apache.org]


In theory, yes. However currently the data placement policy is configured
per cluster.

We can consider pushing down the data placement policy as part of the log
segment metadata. So that bookie auto-recovery will be aware of what data
placement policy will be used for re-replicating/recovering a bookkeeper

This message was sent by Atlassian JIRA

View raw message