flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyle Hamlin <hamlin...@gmail.com>
Subject Re: Testing Flink 1.4 unable to write to s3 locally
Date Wed, 03 Jan 2018 16:26:47 GMT
Hello Stephan & Nico,

Here is the full stacktrace, its not much more than what I originally
posted. I remember seeing an XMLInputFactory input error at one point, but
I haven't seen that again. Is there any other information I can provide
that will help resolve?

[flink-1.4.0] ./bin/flink run ~/streaming.jar
Cluster configuration: Standalone cluster with JobManager at localhost/
127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8082
Starting execution of program
Submitting job with JobID: 022e4a310cd56ba4f7befc0286921663. Waiting for
job completion.
Connected to JobManager at
Actor[akka.tcp://flink@localhost:6123/user/jobmanager#-1321297259]
with leader session id 00000000-0000-0000-0000-000000000000.
01/03/2018 11:21:53 Job execution switched to status RUNNING.
01/03/2018 11:21:53 Source: Kafka -> Sink: Unnamed(1/1) switched to
SCHEDULED
01/03/2018 11:21:53 Source: Kafka -> Sink: Unnamed(1/1) switched to
DEPLOYING
01/03/2018 11:21:53 Source: Kafka -> Sink: Unnamed(1/1) switched to RUNNING
01/03/2018 11:21:54 Source: Kafka -> Sink: Unnamed(1/1) switched to FAILED
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.internal.S3ErrorResponseHandler
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:363)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:542)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.createAmazonS3Client(PrestoS3FileSystem.java:639)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.initialize(PrestoS3FileSystem.java:212)
at
org.apache.flink.fs.s3presto.S3FileSystemFactory.create(S3FileSystemFactory.java:132)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:397)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:748)

01/03/2018 11:21:54 Job execution switched to status FAILING.
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.internal.S3ErrorResponseHandler
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:363)
at
org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:542)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.createAmazonS3Client(PrestoS3FileSystem.java:639)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.initialize(PrestoS3FileSystem.java:212)
at
org.apache.flink.fs.s3presto.S3FileSystemFactory.create(S3FileSystemFactory.java:132)
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:397)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
at
org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
at
org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
at java.lang.Thread.run(Thread.java:748)
01/03/2018 11:21:54 Job execution switched to status RESTARTING.

On Wed, Jan 3, 2018 at 11:33 AM Stephan Ewen <sewen@apache.org> wrote:

> The error is not a the missing Class "S3ErrorResponseHandler", but the
> initialization of that class.
>
> It could be missing classes that are statically referenced by the
> "S3ErrorResponseHandler".
> The ones I see are "XMLInputFactory" (part of Java itself, should always
> be there) and "org.apache.commons.logging.Log" which should also be there.
> There may be other classes loaded during initialization. Knowing more
> about the exception would help, as we have not seen that error, yet.
>
>
> On Wed, Jan 3, 2018 at 11:39 AM, Nico Kruber <nico@data-artisans.com>
> wrote:
>
>> Hi Kyle,
>> except for putting the jar into the lib/ folder and setting up
>> credentials, nothing else should be required [1].
>>
>> The S3ErrorResponseHandler class itself is in the jar, as you can see with
>> jar tf flink-s3-fs-presto-1.4.0.jar | grep
>>
>> org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.internal.S3ErrorResponseHandler
>>
>> Therefore, the cause for this exception would be interesting (as Stephan
>> suggested).
>>
>>
>> Nico
>>
>> [1]
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#shaded-hadooppresto-s3-file-systems-recommended
>>
>> On 03/01/18 10:22, Stephan Ewen wrote:
>> > Hi Kyle!
>> >
>> > Is there more of the stack trace available, like an original exception
>> > cause?
>> >
>> > Best,
>> > Stephan
>> >
>> >
>> > On Sun, Dec 31, 2017 at 5:10 PM, Kyle Hamlin <hamlin.kn@gmail.com
>> > <mailto:hamlin.kn@gmail.com>> wrote:
>> >
>> >     Hi,
>> >
>> >     When testing Flink 1.4 locally the error below keeps getting thrown.
>> >     I've followed the setup by moving the flink-s3-fs-presto.jar from
>> >     the opt/ folder to the lib/ folder. Is there something additional I
>> >     need to do?
>> >
>> >     java.lang.NoClassDefFoundError: Could not initialize class
>> >
>>  org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.internal.S3ErrorResponseHandler
>> >     at
>> >
>>  org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:363)
>> >     at
>> >
>>  org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:542)
>> >     at
>> >
>>  org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.createAmazonS3Client(PrestoS3FileSystem.java:639)
>> >     at
>> >
>>  org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.initialize(PrestoS3FileSystem.java:212)
>> >     at
>> >
>>  org.apache.flink.fs.s3presto.S3FileSystemFactory.create(S3FileSystemFactory.java:132)
>> >     at
>> >
>>  org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:397)
>> >     at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
>> >     at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
>> >     at
>> >
>>  org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory.<init>(FsCheckpointStreamFactory.java:99)
>> >     at
>> >
>>  org.apache.flink.runtime.state.filesystem.FsStateBackend.createStreamFactory(FsStateBackend.java:277)
>> >     at
>> >
>>  org.apache.flink.streaming.runtime.tasks.StreamTask.createCheckpointStreamFactory(StreamTask.java:787)
>> >     at
>> >
>>  org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:247)
>> >     at
>> >
>>  org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:694)
>> >     at
>> >
>>  org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:682)
>> >     at
>> >
>>  org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
>> >     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
>> >
>> >
>>
>>
>

Mime
View raw message