predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shane Johnson <>
Subject Re: Issue with loading dependencies and jars
Date Fri, 09 Mar 2018 14:47:08 GMT
Thanks Donald and Mars,

I created a new distribution (
with the added CLASSPATH code and pointed to the distribution with
the PREDICTIONIO_DIST_URL variable within the engine app in Heroku.


It didn't seem to force the aws-java-sdk to load first as I reviewed the
release logs. Should the aws-java-sdk.jar show up as the first file within
the --jars section when this is added CLASSPATH="/app/PredictionIO-dist/

I'm still getting the NoSuchMethodError when the *aws-java-sdk.jar* loads
after the *pio-data-s3-assembly-0.12.0-incubating.jar**. *Do you have other
suggestions to try? I was also testing locally to change the order of the
--jars but changes to the didn't seem to change the
order of the jars in the logs.

Running train on releaseā€¦

Picked up JAVA_TOOL_OPTIONS: -Xmx12g -Dfile.encoding=UTF-8

[INFO] [Runner$] Submission command:
--driver-memory 13g --class
org.apache.predictionio.workflow.CreateWorkflow --jars
--files file:/app/PredictionIO-dist/conf/,file:/app/PredictionIO-dist/conf/core-site.xml
--driver-java-options -Dpio.log.dir=/app
--engine-id org.template.liftscoring.LiftScoringEngine
--engine-version 0c35eebf403cf91fe77a64921d76aa1ca6411d20
--engine-variant file:/app/engine.json --verbosity 0 --json-extractor
Both --env


Exception in thread "main" java.lang.NoSuchMethodError:<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V

	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(

	at org.apache.hadoop.fs.FileSystem.createFileSystem(

	at org.apache.hadoop.fs.FileSystem.access$200(

*Shane Johnson | LIFT IQ*
*Founder | CEO*

* <>* or *
mobile: (801) 360-3350
LinkedIn <>  |  Twitter
<> |  Facebook

On Wed, Mar 7, 2018 at 1:01 PM, Mars Hall <> wrote:

> Shane,
> On Wed, Mar 7, 2018 at 4:49 AM, Shane Johnson <> wrote:
>> Re: adding a line to ensure a jar is loaded first. Is this what you are
>> referring to...(line at the bottom in red)?
> I believe the code would need to look like this to effect the output
> classpath as intended:
>> CLASSPATH="/app/PredictionIO-dist/lib/spark/aws-java-sdk.jar:$CLASSPATH"
>> echo "$CLASSPATH"
> aws-java-sdk.jar is already in the CLASSPATH though, So, the script will
> need to be skip or remove it first.
> --
> *Mars Hall
> 415-818-7039 <(415)%20818-7039>
> Customer Facing Architect
> Salesforce Platform / Heroku
> San Francisco, California

View raw message