David McLaughlin created AURORA-1861:
----------------------------------------
Summary: Remove duplicate Snapshot fields for DB stores
Key: AURORA-1861
URL: https://issues.apache.org/jira/browse/AURORA-1861
Project: Aurora
Issue Type: Task
Components: Scheduler
Reporter: David McLaughlin
Currently we double-write any DB-backed stores into a Snapshot struct when creating a Snapshot.
This inflates the size of the Snapshot, which is already a problem for large production clusters
(see AURORA-74).
Example for LockStore from https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java:
{code}
new SnapshotField() {
// It's important for locks to be replayed first, since there are relations that expect
// references to be valid on insertion.
@Override
public void saveToSnapshot(MutableStoreProvider store, Snapshot snapshot) {
snapshot.setLocks(ILock.toBuildersSet(store.getLockStore().fetchLocks()));
}
@Override
public void restoreFromSnapshot(MutableStoreProvider store, Snapshot snapshot) {
if (hasDbSnapshot(snapshot)) {
LOG.info("Deferring lock restore to dbsnapshot");
return;
}
store.getLockStore().deleteLocks();
if (snapshot.isSetLocks()) {
for (Lock lock : snapshot.getLocks()) {
store.getLockStore().saveLock(ILock.build(lock));
}
}
}
},
{code}
Note: one major side-effect here is if anyone is trying to read these snapshots and utilize
the data outside of Java - they'll lose the ability to process the data without being able
to apply the DB script.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
|