spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bharath kumar avusherla (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-24476) java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured Streaming in 2.3.0
Date Wed, 06 Jun 2018 16:24:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

bharath kumar avusherla updated SPARK-24476:
--------------------------------------------
    Attachment: socket-timeout-exception

> java.net.SocketTimeoutException: Read timed out Exception while running the Spark Structured
Streaming in 2.3.0
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24476
>                 URL: https://issues.apache.org/jira/browse/SPARK-24476
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: bharath kumar avusherla
>            Priority: Major
>         Attachments: socket-timeout-exception
>
>
> We are working on spark streaming application using spark structured streaming with checkpointing
in s3. When we start the application, the application runs just fine for sometime  then
it crashes with the error mentioned below. The amount of time it will run successfully varies
from time to time, sometimes it will run for 2 days without any issues then crashes, sometimes
it will crash after 4hrs/ 24hrs. 
> Our streaming application joins(left and inner) multiple sources from kafka and also
s3 and aurora database.
> Can you please let us know how to solve this problem? Is it possible to increase the
timeout period?
> Here, I'm pasting the complete exception log below.
> *_Exception:_*
> *_Caused by: java.net.SocketTimeoutException: Read timed out_*
>         _at java.net.SocketInputStream.socketRead0(Native Method)_
>         _at java.net.SocketInputStream.read(SocketInputStream.java:150)_
>         _at java.net.SocketInputStream.read(SocketInputStream.java:121)_
>         _at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)_
>         _at sun.security.ssl.InputRecord.read(InputRecord.java:503)_
>         _at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)_
>         _at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)_
>         _at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371)_
>         _at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)_
>         _at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:553)_
>         _at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:412)_
>         _at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:179)_
>         _at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)_
>         _at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:134)_
>         _at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)_
>         _at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)_
>         _at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)_
>         _at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)_
>         _at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)_
>         _at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:334)_
>         _at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)_
>         _at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)_
>         _at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)_
>         _at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)_
>         _at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093)_
>         _at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548)_
>         _at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)_
>         _at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)_
>         _at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
>         _at java.lang.reflect.Method.invoke(Method.java:483)_
>         _at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)_
>         _at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)_
>         _at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)_
>         _at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)_
>         _at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)_
>         _at org.apache.hadoop.fs.s3native.$Proxy18.retrieveMetadata(Unknown Source)_
>         _at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:493)_
>         _at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1437)_
>         _at org.apache.spark.sql.execution.streaming.HDFSMetadataLog$FileSystemManager.exists(HDFSMetadataLog.scala:446)_
>         _at org.apache.spark.sql.execution.streaming.HDFSMetadataLog.get(HDFSMetadataLog.scala:195)_
>         _at org.apache.spark.sql.execution.streaming.HDFSMetadataLog.add(HDFSMetadataLog.scala:110)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1.apply$mcV$sp(MicroBatchExecution.scala:339)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1.apply(MicroBatchExecution.scala:338)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch$1.apply(MicroBatchExecution.scala:338)_
>         _at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:271)_
>         _at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution.org$apache$spark$sql$execution$streaming$MicroBatchExecution$$constructNextBatch(MicroBatchExecution.scala:338)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply$mcV$sp(MicroBatchExecution.scala:128)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:121)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1$$anonfun$apply$mcZ$sp$1.apply(MicroBatchExecution.scala:121)_
>         _at org.apache.spark.sql.execution.streaming.ProgressReporter$class.reportTimeTaken(ProgressReporter.scala:271)_
>         _at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:58)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$runActivatedStream$1.apply$mcZ$sp(MicroBatchExecution.scala:121)_
>         _at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:56)_
>         _at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:117)_
>         _at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:279)_



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message