ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya <iyakupo...@gmail.com>
Subject Re: Understanding the mechanics of peer class loading
Date Wed, 17 May 2017 08:39:08 GMT
Hi Dmitry!

Do I understand correct that GridDeploymentClassLoader is always on server
used if peer classloading is enabled?

I've did not reproduce the re-deployment issue, but maybe have found
another, which is related to deployment.
Here is the code:
https://github.com/yakupov/IgniteDeploymentTest

How to execute:
1) mvn package in the root;
2) execute JUnit test IgnitionRunner#testJetty (it's in the ITest module).

The result is:
SEVERE: Failed to process message
[senderId=98138e20-8fe4-4750-8281-a92b2067fdcb, messageType=class
o.a.i.i.processors.cache.query.GridCacheQueryRequest]
java.lang.ClassCastException: LocatorBiPredicate cannot be cast to
org.apache.ignite.lang.IgniteBiPredicate
    at
org.apache.ignite.internal.processors.cache.query.GridCacheQueryRequest.finishUnmarshal(GridCacheQueryRequest.java:324)
    at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1298)
    at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:364)
    at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
    at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
    at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
    at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
    at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
    at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
    at
org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
    at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
    at java.lang.Thread.run(Thread.java:745)

where
import org.apache.ignite.lang.IgniteBiPredicate;
public class LocatorBiPredicate implements IgniteBiPredicate<String,
Locator> {...}


On Tue, May 16, 2017 at 12:51 PM, Dmitry Pavlov [via Apache Ignite Users] <
ml+s70518n12881h47@n6.nabble.com> wrote:

> Hi Ilya,
>
> I supposed following: GridDeploymentClassLoader usage for loading message
> class on receiver side is mandatory condition for reproduction. But for now
> it is only hypothesis.
>
> In my test if all nodes have message payload class in its classpath there
> are no problems with messaging under high load (around 10^6 of messages in
> 32 threads). Same is true if application class loader is used in for
> loading message class (no WAR files & no Jetty).
>
> Could you please check class loaders used at sender(s) sides and on
> receiver side. Is GridDeploymentClassLoader used for loading message
> payload class at receiver?
>
> Sincerely,
> Dmitry Pavlov
>
>
>
> вт, 16 мая 2017 г. в 12:17, Ilya <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=12881&i=0>>:
>
>> Hi Dmitry,
>>
>> Unfortunately, I've did not yet manage to reproduce this issue outside of
>> our project.
>>
>> What do you mean by "GridDeploymentClassLoader is used for loading class
>> on server"? How can server classloaders be configured? How does a remote
>> node choose, in which classloader to deploy the received class?
>>
>> My test configuration is as follows:
>>
>>    - Web application that has a single cache and has two IgniteMessaging
>>    local listeners;
>>    - It's run from JUnit test code under Jetty;
>>    - The three instances form a cluster;
>>    - One client sends messages to one of the topics (to the random node)
>>    using IgniteMessaging.
>>
>> All of these works on a single JVM. I suggest that Jetty servers might
>> use dedicated classloaders for each web-app. I've tried to launch client
>> from both test code and another web-app under Jetty, but that did not
>> change anything.
>>
>> In fact, the failing test on our production application is launched in
>> the same manner: two exploded web-apps (client and server), three instances
>> of server app and all of these is run under Jettys in a single VM...
>>
>> On Sun, May 14, 2017 at 2:22 PM, Dmitry Pavlov [via Apache Ignite Users]
>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=12879&i=0>>
>> wrote:
>> Hi, Ilya,
>>
>> I've tried to reproduce deployment problem in standalone project
>> involving Ingnite.start() in several WAR files. But this test is still
>> passing.
>>
>> It is still possible deployment problem can be reprdoced only
>> when GridDeploymentClassLoader is used for loading class on server, and
>> several different Web App class loaders are enabled on clients.
>>
>> Do you have standalone reproducer you can share?
>>
>> Best Regards,
>> Dmitry Pavlov
>>
>>
>>
>> пт, 12 мая 2017 г. в 15:58, Ilya <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=12831&i=0>>:
> Hi all!
>
> The question was originally asked (but not answered) on SO:
> http://stackoverflow.com/questions/43803402/how-does-
> peer-classloading-work-in-apache-ignite
>
> In short, we have "Failed to deploy user message" exceptions under high
> load
> in our project.
>
> Here is an overview of our architecture:
> - Distributed cache on three nodes, all nodes run on a single workstation
> (in this test);
> - Workers on each node;
> - Messaging between workers is done using IgniteMessaging (topic has the
> type of String and I've tried both byte[] and ByteBuffer as a message
> class);
> - Client connects to the cluster and triggers some business logic, that
> causes cross-node messaging, scan queries and MR jobs (using
> IgniteCompute::broadcast). All of these may performed concurrently.
>
> I've tried both SHARED and CONTINUOUS deployment mode, but the result
> remains the same.
>
> I've noticed lots of similar messages in the logs:
> /2017-05-05 13:31:28 INFO   org.apache.ignite.logger.java.JavaLogger info
> Removed undeployed class: GridDeployment [ts=1493980288578,
> depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@38815daa,
> clsLdrId=36c3828db51-0d65e7d5-77bf-444d-9b8b-d18bde94ad13, userVer=0,
> loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
> undeployed=true, usage=0]
> ...
> 2017-05-05 13:31:29 INFO   org.apache.ignite.logger.java.JavaLogger info
> Removed undeployed class: GridDeployment [ts=1493980289125,
> depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@355f6680,
> clsLdrId=1dd3828db51-1b20df7a-a98d-45a3-8ab6-e5d229945830, userVer=0,
> loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
> undeployed=true, usage=0]
> .../
>
> This happens when I use ByteBuffer as message type. In case of byte[],
> class
> B[ is being constantly re-deployed.
>
> ScanQuery predicate and IgniteCompute caller are also being constantly
> re-deployed.
> If we disable ScanQueries and IgniteCompute broadcasts - all is fine, there
> are no re-deployments.
>
> For the further testing I've disabled MRs and kept ScanQueries. I've also
> added some debug output to a fresh snapshot of Ignite 2.1.0. Messages
> "Class
> locally deployed: <my ScanQuery predicate>" usually come from the following
> call stack:
> /org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.
> recordDeploy(GridDeploymentLocalStore.java:404)
>         at
> org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.
> deploy(GridDeploymentLocalStore.java:333)
>         at
> org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.
> getDeployment(GridDeploymentLocalStore.java:201)
>         at
> org.apache.ignite.internal.managers.deployment.GridDeploymentManager.
> getLocalDeployment(GridDeploymentManager.java:383)
>         at
> org.apache.ignite.internal.managers.deployment.GridDeploymentManager.
> getDeployment(GridDeploymentManager.java:345)
>         at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.
> injectResources(GridCacheQueryManager.java:918)
>         at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.
> scanIterator(GridCacheQueryManager.java:826)
>         at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.
> executeQuery(GridCacheQueryManager.java:611)
>         at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.
> queryResult(GridCacheQueryManager.java:1593)
>         at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.
> runQuery(GridCacheQueryManager.java:1164)
>         at
> org.apache.ignite.internal.processors.cache.query.
> GridCacheDistributedQueryManager.processQueryRequest(
> GridCacheDistributedQueryManager.java:231)
>         at
> org.apache.ignite.internal.processors.cache.query.
> GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManag
> er.java:109)
>         at
> org.apache.ignite.internal.processors.cache.query.
> GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManag
> er.java:107)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> processMessage(GridCacheIoManager.java:863)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(
> GridCacheIoManager.java:386)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> handleMessage(GridCacheIoManager.java:308)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(
> GridCacheIoManager.java:100)
>         at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.
> onMessage(GridCacheIoManager.java:253)
>         at
> org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1257)
>         at
> org.apache.ignite.internal.managers.communication.GridIoManager.
> processRegularMessage0(GridIoManager.java:885)
>         at
> org.apache.ignite.internal.managers.communication.
> GridIoManager.access$2100(GridIoManager.java:114)
>         at
> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(
> GridIoManager.java:802)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)/
>
> Messages like "removed undeployed class" usually come from the
> IgniteMessaging's call stack.
>
> I've analyzed the Ignite kernel a bit, and got a suspicion that undeploy is
> being triggered for all classes in a classloader, when at least one class
> that resides in that classloader was re-deployed in some other loader.
>
> It happens inside
> org.apache.ignite.spi.deployment.local.LocalDeploymentSpi#register
>
>     At first, we get a "Map of new resources added for registered class
> loader" using LocalDeploymentSpi#addResource.
>     Then we "Remove resources for all class loaders except {@code
> ignoreClsLdr}." using LocalDeploymentSpi#removeResources. Inside this
> method, it looks like we add all loaders that contain the old version of
> the
> new resource to a "doomed" collection.
>     Finally, we iterate this collection and call onClassLoaderReleased for
> each element. The latter action actually causes all the classes to be
> undeployed (finally causing the "Removed undeployed class" messages).
>
> I don't understand this concept. Why are there multiple classloaders? Why
> do
> we undeploy the whole classloader in such cases?
>
> I'd be grateful, if someone could explain, how does peer classloading work
> in Ignite "under the hood".
>
> P. S. I'm looking at the sources of a fresh snapshot of Ignite 2.1.0, but
> the behavior is the same with the standard Ignite 1.9.0.
>
> P. P. S. Unfortunately, I've did not manage to reproduce this issue outside
> of our project yet.
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Understanding-the-mechanics-
> of-peer-class-loading-tp12661.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/
> Understanding-the-mechanics-of-peer-class-loading-tp12661p12831.html
> To unsubscribe from Understanding the mechanics of peer class loading, click
> here.
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> ------------------------------
> View this message in context: Re: Understanding the mechanics of peer
> class loading
> <http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12879.html>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/
> Understanding-the-mechanics-of-peer-class-loading-tp12661p12881.html
> To unsubscribe from Understanding the mechanics of peer class loading, click
> here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=12661&code=aXlha3Vwb3Y5M0BnbWFpbC5jb218MTI2NjF8LTQ4NzEwMDkzOQ==>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12961.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Mime
View raw message