ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Pavlov <dpavlov....@gmail.com>
Subject Re: Understanding the mechanics of peer class loading
Date Wed, 17 May 2017 10:04:33 GMT
Hi Ilya,

To my mind in this case it's not a problem of Peer Class Loading (PCL).
Please note if you disable PCL in Ignite Configuration, test will fail
anyway.

In this test LocatorBiPredicate is taken by Jetty from the parent class
loader (test launcher class loader) and cast to IgnitePredicate can't
complete sucessfull because of different class loaders used.

May I ask to to provide log and/or stacktrace of original deployment
problem in the project? Additionally you may enable debug log level for
deployment.

Best Regards,
Dmitry Pavlov

ср, 17 мая 2017 г. в 11:39, Ilya <iyakupov93@gmail.com>:

> Hi Dmitry!
>
> Do I understand correct that GridDeploymentClassLoader is always on server
> used if peer classloading is enabled?
>
> I've did not reproduce the re-deployment issue, but maybe have found
> another, which is related to deployment.
> Here is the code:
> https://github.com/yakupov/IgniteDeploymentTest
>
> How to execute:
> 1) mvn package in the root;
> 2) execute JUnit test IgnitionRunner#testJetty (it's in the ITest module).
>
> The result is:
> SEVERE: Failed to process message
> [senderId=98138e20-8fe4-4750-8281-a92b2067fdcb, messageType=class
> o.a.i.i.processors.cache.query.GridCacheQueryRequest]
> java.lang.ClassCastException: LocatorBiPredicate cannot be cast to
> org.apache.ignite.lang.IgniteBiPredicate
>     at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryRequest.finishUnmarshal(GridCacheQueryRequest.java:324)
>     at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1298)
>     at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:364)
>     at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
>     at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
>     at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
>     at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
>     at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
>     at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
>     at
> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
>     at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
>     at java.lang.Thread.run(Thread.java:745)
>
> where
> import org.apache.ignite.lang.IgniteBiPredicate;
> public class LocatorBiPredicate implements IgniteBiPredicate<String,
> Locator> {...}
>
>
> On Tue, May 16, 2017 at 12:51 PM, Dmitry Pavlov [via Apache Ignite Users]
> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=12961&i=0>>
> wrote:
>
>> Hi Ilya,
>>
>> I supposed following: GridDeploymentClassLoader usage for loading message
>> class on receiver side is mandatory condition for reproduction. But for now
>> it is only hypothesis.
>>
>> In my test if all nodes have message payload class in its classpath there
>> are no problems with messaging under high load (around 10^6 of messages in
>> 32 threads). Same is true if application class loader is used in for
>> loading message class (no WAR files & no Jetty).
>>
>> Could you please check class loaders used at sender(s) sides and on
>> receiver side. Is GridDeploymentClassLoader used for loading message
>> payload class at receiver?
>>
>> Sincerely,
>> Dmitry Pavlov
>>
>>
>>
>> вт, 16 мая 2017 г. в 12:17, Ilya <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=12881&i=0>>:
>>
> Hi Dmitry,
>>>
>>> Unfortunately, I've did not yet manage to reproduce this issue outside
>>> of our project.
>>>
>>> What do you mean by "GridDeploymentClassLoader is used for loading
>>> class on server"? How can server classloaders be configured? How does a
>>> remote node choose, in which classloader to deploy the received class?
>>>
>>> My test configuration is as follows:
>>>
>>>    - Web application that has a single cache and has two
>>>    IgniteMessaging local listeners;
>>>    - It's run from JUnit test code under Jetty;
>>>    - The three instances form a cluster;
>>>    - One client sends messages to one of the topics (to the random
>>>    node) using IgniteMessaging.
>>>
>>> All of these works on a single JVM. I suggest that Jetty servers might
>>> use dedicated classloaders for each web-app. I've tried to launch client
>>> from both test code and another web-app under Jetty, but that did not
>>> change anything.
>>>
>>> In fact, the failing test on our production application is launched in
>>> the same manner: two exploded web-apps (client and server), three instances
>>> of server app and all of these is run under Jettys in a single VM...
>>>
>>> On Sun, May 14, 2017 at 2:22 PM, Dmitry Pavlov [via Apache Ignite Users]
>>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=12879&i=0>>
>>> wrote:
>>> Hi, Ilya,
>>>
>>> I've tried to reproduce deployment problem in standalone project
>>> involving Ingnite.start() in several WAR files. But this test is still
>>> passing.
>>>
>>> It is still possible deployment problem can be reprdoced only
>>> when GridDeploymentClassLoader is used for loading class on server, and
>>> several different Web App class loaders are enabled on clients.
>>>
>>> Do you have standalone reproducer you can share?
>>>
>>> Best Regards,
>>> Dmitry Pavlov
>>>
>>>
>>>
>>> пт, 12 мая 2017 г. в 15:58, Ilya <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=12831&i=0>>:
>> Hi all!
>>
>> The question was originally asked (but not answered) on SO:
>>
>> http://stackoverflow.com/questions/43803402/how-does-peer-classloading-work-in-apache-ignite
>>
>> In short, we have "Failed to deploy user message" exceptions under high
>> load
>> in our project.
>>
>> Here is an overview of our architecture:
>> - Distributed cache on three nodes, all nodes run on a single workstation
>> (in this test);
>> - Workers on each node;
>> - Messaging between workers is done using IgniteMessaging (topic has the
>> type of String and I've tried both byte[] and ByteBuffer as a message
>> class);
>> - Client connects to the cluster and triggers some business logic, that
>> causes cross-node messaging, scan queries and MR jobs (using
>> IgniteCompute::broadcast). All of these may performed concurrently.
>>
>> I've tried both SHARED and CONTINUOUS deployment mode, but the result
>> remains the same.
>>
>> I've noticed lots of similar messages in the logs:
>> /2017-05-05 13:31:28 INFO   org.apache.ignite.logger.java.JavaLogger info
>> Removed undeployed class: GridDeployment [ts=1493980288578,
>> depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@38815daa,
>> clsLdrId=36c3828db51-0d65e7d5-77bf-444d-9b8b-d18bde94ad13, userVer=0,
>> loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
>> undeployed=true, usage=0]
>> ...
>> 2017-05-05 13:31:29 INFO   org.apache.ignite.logger.java.JavaLogger info
>> Removed undeployed class: GridDeployment [ts=1493980289125,
>> depMode=CONTINUOUS, clsLdr=WebAppClassLoader=MyApp@355f6680,
>> clsLdrId=1dd3828db51-1b20df7a-a98d-45a3-8ab6-e5d229945830, userVer=0,
>> loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
>> undeployed=true, usage=0]
>> .../
>>
>> This happens when I use ByteBuffer as message type. In case of byte[],
>> class
>> B[ is being constantly re-deployed.
>>
>> ScanQuery predicate and IgniteCompute caller are also being constantly
>> re-deployed.
>> If we disable ScanQueries and IgniteCompute broadcasts - all is fine,
>> there
>> are no re-deployments.
>>
>> For the further testing I've disabled MRs and kept ScanQueries. I've also
>> added some debug output to a fresh snapshot of Ignite 2.1.0. Messages
>> "Class
>> locally deployed: <my ScanQuery predicate>" usually come from the
>> following
>> call stack:
>>
>> /org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.recordDeploy(GridDeploymentLocalStore.java:404)
>>         at
>>
>> org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.deploy(GridDeploymentLocalStore.java:333)
>>         at
>>
>> org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.getDeployment(GridDeploymentLocalStore.java:201)
>>         at
>>
>> org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getLocalDeployment(GridDeploymentManager.java:383)
>>         at
>>
>> org.apache.ignite.internal.managers.deployment.GridDeploymentManager.getDeployment(GridDeploymentManager.java:345)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.injectResources(GridCacheQueryManager.java:918)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.scanIterator(GridCacheQueryManager.java:826)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.executeQuery(GridCacheQueryManager.java:611)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.queryResult(GridCacheQueryManager.java:1593)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1164)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager.processQueryRequest(GridCacheDistributedQueryManager.java:231)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:109)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.query.GridCacheDistributedQueryManager$2.apply(GridCacheDistributedQueryManager.java:107)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100)
>>         at
>>
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253)
>>         at
>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
>>         at
>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885)
>>         at
>>
>> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:114)
>>         at
>>
>> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:802)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)/
>>
>> Messages like "removed undeployed class" usually come from the
>> IgniteMessaging's call stack.
>>
>> I've analyzed the Ignite kernel a bit, and got a suspicion that undeploy
>> is
>> being triggered for all classes in a classloader, when at least one class
>> that resides in that classloader was re-deployed in some other loader.
>>
>> It happens inside
>> org.apache.ignite.spi.deployment.local.LocalDeploymentSpi#register
>>
>>     At first, we get a "Map of new resources added for registered class
>> loader" using LocalDeploymentSpi#addResource.
>>     Then we "Remove resources for all class loaders except {@code
>> ignoreClsLdr}." using LocalDeploymentSpi#removeResources. Inside this
>> method, it looks like we add all loaders that contain the old version of
>> the
>> new resource to a "doomed" collection.
>>     Finally, we iterate this collection and call onClassLoaderReleased for
>> each element. The latter action actually causes all the classes to be
>> undeployed (finally causing the "Removed undeployed class" messages).
>>
>> I don't understand this concept. Why are there multiple classloaders? Why
>> do
>> we undeploy the whole classloader in such cases?
>>
>> I'd be grateful, if someone could explain, how does peer classloading work
>> in Ignite "under the hood".
>>
>> P. S. I'm looking at the sources of a fresh snapshot of Ignite 2.1.0, but
>> the behavior is the same with the standard Ignite 1.9.0.
>>
>> P. P. S. Unfortunately, I've did not manage to reproduce this issue
>> outside
>> of our project yet.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>>
>> http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12831.html
>> To unsubscribe from Understanding the mechanics of peer class loading, click
>> here.
>> NAML
>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>> ------------------------------
>> View this message in context: Re: Understanding the mechanics of peer
>> class loading
>> <http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12879.html>
>> Sent from the Apache Ignite Users mailing list archive
>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>>
>>
>> ------------------------------
>>
> If you reply to this email, your message will be added to the discussion
>> below:
>>
>
>> http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12881.html
>>
> To unsubscribe from Understanding the mechanics of peer class loading, click
>> here.
>> NAML
>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
> ------------------------------
> View this message in context: Re: Understanding the mechanics of peer
> class loading
> <http://apache-ignite-users.70518.x6.nabble.com/Understanding-the-mechanics-of-peer-class-loading-tp12661p12961.html>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>

Mime
View raw message