hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cs user <acldstk...@gmail.com>
Subject Re: Class org.apache.hadoop.fs.s3a.S3A not found
Date Fri, 16 Dec 2016 21:49:55 GMT
Hello,

Thank you both for getting back to me so quickly,

Can you explain how I can get this to work with the latest version? I'm
trying to submit a spark job with the following ( With the hadoop 3
release).

Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages"
"com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:3.0.0-alpha1"
"sparkr-shell"',

But then I get:

16/12/16 21:47:56 ERROR RBackendHandler: createSparkContext on
org.apache.spark.api.r.RRDD failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) :
  java.lang.IllegalAccessError: tried to access method
org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V
from class org.apache.hadoop.fs.s3a.S3AInstrumentation


How can I set my environment to pull down the hadoop 2.8 release?

Thank you again for your help.




On Fri, Dec 16, 2016 at 9:33 PM, Mingliang Liu <liuml07@gmail.com> wrote:

> You may need the hadoop-aws*-version*.jar (and aws-java-sdk.jar) in your
> classpath.
>
> For new project, have a look at the latest S3A progress:
> https://github.com/apache/hadoop/blob/release-3.
> 0.0-alpha1-RC0/hadoop-tools/hadoop-aws/src/site/markdown/
> tools/hadoop-aws/index.md
>
> Good luck,
>
> L
>
> On Dec 16, 2016, at 1:24 PM, cs user <acldstkusr@gmail.com> wrote:
>
> Hello,
>
> Can someone tell me which version of Hadoop I should be able to find this
> class in?
>
> Thanks!
>
>
>

Mime
View raw message