ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hueb1 <eric...@finra.org>
Subject AWS Java SDK marshalling errors, best practices?
Date Tue, 11 Aug 2015 17:51:12 GMT
I have two nodes on different EC2 machines. I'm trying to run a compute task
using ExecutorService framework. The compute task uses the AWS Java SDK to
perform AmazonS3 API calls.  I have Peer class loading enabled.  I
originally was getting Serialization exceptions saying that AmazonS3 is not
serializable.  

First question: What is the best practice to get around this Serialization
exception when using third party packages that are not trivial to make
serializable?

To get around this for now, I change the OptimizationMarshaller to not
require serialization. Now I'm getting

[17:49:49,861][SEVERE][main][GridExecutorService] Failed to get task result:
ComputeTaskInternalFuture [ses=GridTaskSessionImpl
[taskName=ignite_poc.aws.S3Util$BlockDownloader, dep=GridDeployment
[ts=1439315380598, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@30a4effe,
clsLdrId=8adb1ed1f41-df0e0a53-f6ff-4439-a6d8-a6014e38e031, userVer=0,
loc=true,
sampleClsName=o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap,
pendingUndeploy=false, undeployed=false, usage=0],
taskClsName=ignite_poc.aws.S3Util$BlockDownloader,
sesId=13eb1ed1f41-df0e0a53-f6ff-4439-a6d8-a6014e38e031,
startTime=1439315383145, endTime=9223372036854775807,
taskNodeId=df0e0a53-f6ff-4439-a6d8-a6014e38e031,
clsLdr=sun.misc.Launcher$AppClassLoader@30a4effe, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false,
subjId=df0e0a53-f6ff-4439-a6d8-a6014e38e031, mapFut=IgniteFuture
[orig=GridFutureAdapter [resFlag=2, res=null, startTime=1439315383145,
endTime=1439315383145, ignoreInterrupts=false, lsnr=null, state=DONE]]],
super=GridFutureAdapter [resFlag=1, res=class o.a.i.IgniteException: Remote
job threw user exception (override or implement ComputeTask.result(..)
method if you would like to have automatic failover for this exception).,
startTime=1439315383145, endTime=1439315383276, ignoreInterrupts=false,
lsnr=null, state=DONE]]
class org.apache.ignite.IgniteCheckedException: Failed to find class with
given class loader for unmarshalling (make sure same version of all classes
are available on all nodes or enable peer-class-loading):
sun.misc.Launcher$AppClassLoader@305f387c
        at
org.apache.ignite.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:242)
        at
org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:382)
        at
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1046)
        at
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1728)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:690)
        at
org.apache.ignite.internal.managers.communication.GridIoManager.access$1500(GridIoManager.java:58)
        at
org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:653)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.amazonaws.http.conn.$Proxy2
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)

I copied all dependent libraries (AWS included) into the libs folder on each
of the two nodes prior to starting them up, so each node should have all
required libs.  This works fine when executing in a single node, but the
classpath seems to get messed up when I move to multiple nodes.   



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/AWS-Java-SDK-marshalling-errors-best-practices-tp909.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message