Mailing-List: contact users-help@ace.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@ace.apache.org
Received-SPF: pass (athena.apache.org: local policy)
In-Reply-To: <713BF914-3CA4-4B36-9D13-00CFC7B37BB4@luminis.eu>
References: 
 <OF4C9BC931.570E361B-ONC1257D6A.0056A5B1-C1257D6A.00598FCC@gebit.de>
 <713BF914-3CA4-4B36-9D13-00CFC7B37BB4@luminis.eu>
To: users@ace.apache.org
MIME-Version: 1.0
Subject: Antwort: Re: Poor performance of the REST Client API with hundreds of
 targets.
Message-ID: 
 <OF63C5E1BD.43CEABA9-ONC1257D6B.0035A63F-C1257D6B.00379C2B@gebit.de>
From: Klaus.Meyer@Gebit.de
Date: Wed, 8 Oct 2014 12:07:22 +0200
Content-Type: multipart/alternative;
 boundary="=_alternative 00379C2BC1257D6B_="

--=_alternative 00379C2BC1257D6B_=
Content-Type: text/plain; charset="US-ASCII"

Hello Marcel,

Thank you for the very fast answer. First I want to say that I'm very 
impressed about the concepts, design and quality of ACE and I really want 
to use it in a productional context.
I did some further investigations of my performance issue and found out 
that the remote access is not the problem. The checkout of the 
RepositoryAdmin is the method were the time gets lost. So the problem 
seems to be in the core of the ACE persistence layer ? This seems to be 
reasonable because the checkout makes a local copy of all data as you 
said, and when I have 1000 targets and some distributions with 100 
artifacts, there is some work to do. Do see the chance to optimize it or 
do you think there must be another problem ? Some thousands of targets is 
a very typical requirement in our POS domain. The performance problem in 
the moment is just in the area of management of the repositories not in 
the area of downloading artifacts from targets. In the latter case I could 
optimize it with the concept of relay servers if I understood right. 
Another problem is that sometimes are  exceptions thrown from the 
Preference Service:

org.osgi.service.prefs.BackingStoreException: Unable to load preferences.
        at org.apache.felix.prefs.impl.DataFileBackingStoreImpl.load(
DataFileBackingStoreImpl.java:164)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.update(
StreamBackingStoreImpl.java:102)
        at org.apache.felix.prefs.PreferencesImpl.sync(
PreferencesImpl.java:588)
        at org.apache.felix.prefs.PreferencesImpl.childrenNames(
PreferencesImpl.java:375)
        at 
org.apache.ace.client.repository.impl.RepositorySet.loadPreferences(
RepositorySet.java:159)
        at 
org.apache.ace.client.repository.impl.RepositoryAdminImpl.login(
RepositoryAdminImpl.java:404)
        at 
org.apache.ace.client.repository.impl.RepositoryAdminImpl.login(
RepositoryAdminImpl.java:385)
        at org.apache.ace.client.workspace.impl.WorkspaceImpl.login(
WorkspaceImpl.java:151)
        at 
org.apache.ace.client.workspace.impl.WorkspaceManagerImpl.createWorkspace(
WorkspaceManagerImpl.java:160)
        at org.apache.ace.client.rest.RESTClientServlet.doPost(
RESTClientServlet.java:264)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at 
org.apache.felix.http.base.internal.handler.ServletHandler.doHandle(
ServletHandler.java:96)
        at 
org.apache.felix.http.base.internal.handler.ServletHandler.handle(
ServletHandler.java:79)
        at 
org.apache.felix.http.base.internal.dispatch.ServletPipeline.handle(
ServletPipeline.java:42)
        at 
org.apache.felix.http.base.internal.dispatch.InvocationFilterChain.doFilter(
InvocationFilterChain.java:49)
        at 
org.apache.felix.http.base.internal.dispatch.HttpFilterChain.doFilter(
HttpFilterChain.java:33)
        at 
org.apache.felix.http.base.internal.dispatch.FilterPipeline.dispatch(
FilterPipeline.java:48)
        at 
org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(
Dispatcher.java:39)
        at org.apache.felix.http.base.internal.DispatcherServlet.service(
DispatcherServlet.java:67)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.eclipse.jetty.servlet.ServletHolder.handle(
ServletHolder.java:654)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHandler.java:445)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(
SessionHandler.java:225)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
ContextHandler.java:1044)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(
ServletHandler.java:372)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(
SessionHandler.java:189)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(
ContextHandler.java:978)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(
ScopedHandler.java:135)
        at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
ContextHandlerCollection.java:255)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:369)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(
AbstractHttpConnection.java:486)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(
AbstractHttpConnection.java:933)
        at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(
AbstractHttpConnection.java:995)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644
)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(
HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(
AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(
SelectChannelEndPoint.java:668)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(
SelectChannelEndPoint.java:52)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
QueuedThreadPool.java:543)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.io.StreamCorruptedException: unexpected EOF in middle of 
data block
        at java.io.ObjectInputStream$BlockDataInputStream.refill(Unknown 
Source)
        at java.io.ObjectInputStream$BlockDataInputStream.read(Unknown 
Source)
        at java.io.DataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown 
Source)
        at java.io.ObjectInputStream.readInt(Unknown Source)
        at 
org.apache.felix.prefs.impl.StreamBackingStoreImpl.readPreferences(
StreamBackingStoreImpl.java:165)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:144)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.StreamBackingStoreImpl.read(
StreamBackingStoreImpl.java:152)
        at org.apache.felix.prefs.impl.DataFileBackingStoreImpl.load(
DataFileBackingStoreImpl.java:159)
        ... 42 more


Do you have any ideas ?

Thank you in advance and kind regards

Klaus


Von:
Marcel Offermans <marcel.offermans@luminis.eu>
An:
ACE-users Apache ACE users <users@ace.apache.org>
Datum:
07.10.2014 18:47
Betreff:
Re: Poor performance of the REST Client API with hundreds of targets.


Hello Klaus,

On 07 Oct 2014, at 18:18 pm, Klaus.Meyer@Gebit.de wrote:

> I want to use ACE (version 2.0.1) for deploying POS (Point of sale) java 

> clients. In my scenario I have hundreds or thousands of targets 
> (POS-clients). I developed an API around the REST-API for creation of 
> distributions which point to an associated feature which contains about 
> 100 bundles. On the other side I have developed an API (based on the 
> REST-API) which creates targets and connects them to the distributions. 
I 
> found out that when I import about 1000 targets then the allinone server 

> becomes extremely slow, when managing each kind of data (creating and 
> deleting distributions, targets). Allocating a working area needs about 
> one minute or so. Also sometimes I get exceptions from the Felix 
> preference service. Do you have already experiences whith a lot of 
> targets, are there any issues known about performance ? Can you give me 
> some ideas what I can do next to solve my problem.

Let me start by explaining a bit about how the ACE client code works, as 
that will give you more insight into this problem.

ACE consists of a server, a client and an OBR repository. Each can be 
deployed independently, but for convenience we also have an "all-in-one" 
version. The client is what you need to manipulate the repository, and 
fundamentally the client works by "checking out" a copy of the current 
configuration, manipulating it locally and, when done, "committing" the 
new configuration back to the server.

Now, the client has an OSGi API (services) you can talk to, and on top of 
that we built three different APIs:

1. The web UI based on Vaadin.
2. The REST API.
3. The Gogo Shell API.

The first two can work "over the network" with a browser or REST client 
being the primary users.
The last one works "on the client" and is therefore usually a lot faster. 
We use it for all kinds of scripted access such as in continuous 
integration scenarios.

That being said, I have a few recommendations:

First of all, if you build your own API, I would consider building it 
directly on top of the OSGi, service based API just like the 3 existing 
clients do. That will be a lot quicker than interfacing, over the network, 
with a remote client.

Second of all, you might want to consider the Gogo Shell API to script 
these things. We use this in continuous integration and if it helps I can 
provide some examples (but I probably need to know a bit more details 
about the deployment process you are using).

Finally, we have tested ACE with 100's to 1000's of targets. I would say 
that is possible. I currently would not go to 10.000's of targets, our 
benchmarks have shown that you are running into some limits there. 
However, we have options to partition the targets into smaller groups 
within ACE, so if necessary I can go into that a bit more and explore if 
that is an option that could help you. Also, one thing that slows ACE down 
is when you have "auto configuration" XML files which are templates that 
have parameters that need to be substituted for each target. We are still 
discussing ways to optimize that.

I hope this gives you some pointers/suggestions that help. Feel free to 
follow up if you have more questions.

Greetings, Marcel


--=_alternative 00379C2BC1257D6B_=--