jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Dürig <mdue...@apache.org>
Subject [j3 microkernel] support for large child node lists (svn commit: r1214921 - /jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java)
Date Thu, 15 Dec 2011 21:40:09 GMT

Hi,

Starting with this revision there is some experimental support for large 
child node lists in spi2microkernel and jcr2spi (jackrabbit-mk branch in 
the sandbox). The Microkernel itself is not yet optimized for large 
child node list but AFAIK Stefan is working on this.

To support large child node lists, I basically defer loading of child 
entries as much as possible. Furthermore child nodes are loaded in 
chunks of max. 512 entries at a time. Loaded child nodes are kept 
internally by the ChildNodeEntries class until cleared by the garbage 
collector. Using the garbage collector to control the child entries is 
not optimal and may lead to memory thrashing when clients fully iterate 
over large child node lists. Within the current design of the jcr2spi 
layer there does not seem to be a better solution though.

Disclaimer: while the test cases pass, I did not yet test the 
effectiveness of this solution.

Michael


-------- Original Message --------
Subject: svn commit: r1214921 - 
/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
Date: Thu, 15 Dec 2011 19:25:18 -0000
From: mduerig@apache.org
Reply-To: dev@jackrabbit.apache.org
To: commits@jackrabbit.apache.org

Author: mduerig
Date: Thu Dec 15 19:25:17 2011
New Revision: 1214921

URL: http://svn.apache.org/viewvc?rev=1214921&view=rev
Log:
Microkernel based Jackrabbit prototype (WIP)
batch loading of child nodes

Modified:

jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java

Modified: 
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
URL: 
http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java?rev=1214921&r1=1214920&r2=1214921&view=diff
==============================================================================
--- 
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java

(original)
+++ 
jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java

Thu Dec 15 19:25:17 2011
@@ -301,7 +301,7 @@ public class RepositoryServiceImpl exten

      @Override
      protected void checkWorkspace(String workspaceName) throws 
NoSuchWorkspaceException {
-        if (workspaceName == null || "".equals(workspaceName)) {
+        if (workspaceName == null || workspaceName.isEmpty()) {
              workspaceName = defaultWorkspace;
          }
          else if (workspaceName.contains("/")) {
@@ -339,7 +339,7 @@ public class RepositoryServiceImpl exten
                  throw new PathNotFoundException(itemPath.toString());
              }

-            String json = microKernel.getNodes(mkPath, rev);
+            String json = microKernel.getNodes(mkPath, rev, 0, 0, 0);
              return buildItemInfos(nodePath, itemPath, json, 
readFromDataStore);
          }
          catch (MicroKernelException e) {
@@ -352,18 +352,65 @@ public class RepositoryServiceImpl exten
      public Iterator<ChildInfo> getChildInfos(SessionInfo sessionInfo, 
NodeId parentId) throws RepositoryException {
          try {
              String wspName = sessionInfo.getWorkspaceName();
-            String rev = getRevision(sessionInfo);
-            Path path = getPath(parentId, wspName, rev);
-            String mkPath = Paths.pathToString(wspName, path);
+            final String rev = getRevision(sessionInfo);
+            final Path path = getPath(parentId, wspName, rev);
+            final String mkPath = Paths.pathToString(wspName, path);

              if (!microKernel.nodeExists(mkPath, rev)) {
                  throw new PathNotFoundException(path.toString());
              }

-            String json = microKernel.getNodes(mkPath, rev);
-            Iterator<? extends ItemInfo> infos = buildItemInfos(path, 
path, json, readFromDataStore);
-            NodeInfo info = (NodeInfo) infos.next();
-            return info.getChildInfos();
+            return new Iterator<ChildInfo>() {
+                private static final int chunkSize = 512;
+
+                private int pos;
+                private Iterator<ChildInfo> currentChunk;
+
+                {
+                    nextChunk();
+                }
+
+                @Override
+                public boolean hasNext() {
+                    if (currentChunk.hasNext()) {
+                        return true;
+                    }
+                    else {
+                        nextChunk();
+                        return currentChunk.hasNext();
+                    }
+                }
+
+                @Override
+                public ChildInfo next() {
+                    if (currentChunk.hasNext()) {
+                        return currentChunk.next();
+                    }
+                    else {
+                        nextChunk();
+                        return currentChunk.next();
+                    }
+                }
+
+                @Override
+                public void remove() {
+                    throw new UnsupportedOperationException("remove");
+                }
+
+                private void nextChunk() {
+                    try {
+                        String json = microKernel.getNodes(mkPath, rev, 
0, pos, chunkSize);
+                        pos += chunkSize;
+                        Iterator<? extends ItemInfo> infos = 
buildItemInfos(path, path, json, readFromDataStore);
+                        NodeInfo info = (NodeInfo) infos.next();
+                        currentChunk = info.getChildInfos();
+                    }
+                    catch (RepositoryException e) {
+                        // fixme think of a better way to handle this
+                        log.error(e.getMessage(), e);
+                    }
+                }
+            };
          }
          catch (MicroKernelException e) {
              log.error(e.getMessage(), e);




Mime
View raw message