jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Mueller <muel...@adobe.com>
Subject Re: [j3 microkernel] support for large child node lists (svn commit: r1214921 - /jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java)
Date Wed, 21 Dec 2011 09:04:40 GMT
Hi,

By the way, the MemoryKernelImpl supports an optimization for large child
node lists, but it is disabled by default. To enable it, call:

    mk.commit("/:root/head/config", "^ \"maxMemoryChildren\":" + max,
head, "");


See also org.apache.jackrabbit.mk.large.LargeNodeTest

Regards,
Thomas

On 12/15/11 10:40 PM, "Michael Dürig" <mduerig@apache.org> wrote:

>
>Hi,
>
>Starting with this revision there is some experimental support for large
>child node lists in spi2microkernel and jcr2spi (jackrabbit-mk branch in
>the sandbox). The Microkernel itself is not yet optimized for large
>child node list but AFAIK Stefan is working on this.
>
>To support large child node lists, I basically defer loading of child
>entries as much as possible. Furthermore child nodes are loaded in
>chunks of max. 512 entries at a time. Loaded child nodes are kept
>internally by the ChildNodeEntries class until cleared by the garbage
>collector. Using the garbage collector to control the child entries is
>not optimal and may lead to memory thrashing when clients fully iterate
>over large child node lists. Within the current design of the jcr2spi
>layer there does not seem to be a better solution though.
>
>Disclaimer: while the test cases pass, I did not yet test the
>effectiveness of this solution.
>
>Michael
>
>
>-------- Original Message --------
>Subject: svn commit: r1214921 -
>/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java
>/org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>Date: Thu, 15 Dec 2011 19:25:18 -0000
>From: mduerig@apache.org
>Reply-To: dev@jackrabbit.apache.org
>To: commits@jackrabbit.apache.org
>
>Author: mduerig
>Date: Thu Dec 15 19:25:17 2011
>New Revision: 1214921
>
>URL: http://svn.apache.org/viewvc?rev=1214921&view=rev
>Log:
>Microkernel based Jackrabbit prototype (WIP)
>batch loading of child nodes
>
>Modified:
>
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>
>Modified: 
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>URL: 
>http://svn.apache.org/viewvc/jackrabbit/sandbox/jackrabbit-mk/jackrabbit-s
>pi2microkernel/src/main/java/org/apache/jackrabbit/spi2microkernel/Reposit
>oryServiceImpl.java?rev=1214921&r1=1214920&r2=1214921&view=diff
>==========================================================================
>====
>--- 
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>(original)
>+++ 
>jackrabbit/sandbox/jackrabbit-mk/jackrabbit-spi2microkernel/src/main/java/
>org/apache/jackrabbit/spi2microkernel/RepositoryServiceImpl.java
>Thu Dec 15 19:25:17 2011
>@@ -301,7 +301,7 @@ public class RepositoryServiceImpl exten
>
>      @Override
>      protected void checkWorkspace(String workspaceName) throws
>NoSuchWorkspaceException {
>-        if (workspaceName == null || "".equals(workspaceName)) {
>+        if (workspaceName == null || workspaceName.isEmpty()) {
>              workspaceName = defaultWorkspace;
>          }
>          else if (workspaceName.contains("/")) {
>@@ -339,7 +339,7 @@ public class RepositoryServiceImpl exten
>                  throw new PathNotFoundException(itemPath.toString());
>              }
>
>-            String json = microKernel.getNodes(mkPath, rev);
>+            String json = microKernel.getNodes(mkPath, rev, 0, 0, 0);
>              return buildItemInfos(nodePath, itemPath, json,
>readFromDataStore);
>          }
>          catch (MicroKernelException e) {
>@@ -352,18 +352,65 @@ public class RepositoryServiceImpl exten
>      public Iterator<ChildInfo> getChildInfos(SessionInfo sessionInfo,
>NodeId parentId) throws RepositoryException {
>          try {
>              String wspName = sessionInfo.getWorkspaceName();
>-            String rev = getRevision(sessionInfo);
>-            Path path = getPath(parentId, wspName, rev);
>-            String mkPath = Paths.pathToString(wspName, path);
>+            final String rev = getRevision(sessionInfo);
>+            final Path path = getPath(parentId, wspName, rev);
>+            final String mkPath = Paths.pathToString(wspName, path);
>
>              if (!microKernel.nodeExists(mkPath, rev)) {
>                  throw new PathNotFoundException(path.toString());
>              }
>
>-            String json = microKernel.getNodes(mkPath, rev);
>-            Iterator<? extends ItemInfo> infos = buildItemInfos(path,
>path, json, readFromDataStore);
>-            NodeInfo info = (NodeInfo) infos.next();
>-            return info.getChildInfos();
>+            return new Iterator<ChildInfo>() {
>+                private static final int chunkSize = 512;
>+
>+                private int pos;
>+                private Iterator<ChildInfo> currentChunk;
>+
>+                {
>+                    nextChunk();
>+                }
>+
>+                @Override
>+                public boolean hasNext() {
>+                    if (currentChunk.hasNext()) {
>+                        return true;
>+                    }
>+                    else {
>+                        nextChunk();
>+                        return currentChunk.hasNext();
>+                    }
>+                }
>+
>+                @Override
>+                public ChildInfo next() {
>+                    if (currentChunk.hasNext()) {
>+                        return currentChunk.next();
>+                    }
>+                    else {
>+                        nextChunk();
>+                        return currentChunk.next();
>+                    }
>+                }
>+
>+                @Override
>+                public void remove() {
>+                    throw new UnsupportedOperationException("remove");
>+                }
>+
>+                private void nextChunk() {
>+                    try {
>+                        String json = microKernel.getNodes(mkPath, rev,
>0, pos, chunkSize);
>+                        pos += chunkSize;
>+                        Iterator<? extends ItemInfo> infos =
>buildItemInfos(path, path, json, readFromDataStore);
>+                        NodeInfo info = (NodeInfo) infos.next();
>+                        currentChunk = info.getChildInfos();
>+                    }
>+                    catch (RepositoryException e) {
>+                        // fixme think of a better way to handle this
>+                        log.error(e.getMessage(), e);
>+                    }
>+                }
>+            };
>          }
>          catch (MicroKernelException e) {
>              log.error(e.getMessage(), e);
>
>
>


Mime
View raw message