cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuki Morishita (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8774) BulkOutputFormat never completes if streaming have errors
Date Thu, 12 Feb 2015 17:01:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318549#comment-14318549
] 

Yuki Morishita commented on CASSANDRA-8774:
-------------------------------------------

What I understand from that {{{try ... catch}}} clause is that it is using TimeoutException
to report progress every 1000 ms.
It looks to me that doing correct error handling when ExecutorException is thrown is the right
way to do here, instead of handling StreamingException individually.
WDYT [~alexliu68]?

> BulkOutputFormat never completes if streaming have errors
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-8774
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8774
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>            Reporter: Erik Forsberg
>             Fix For: 2.0.13
>
>         Attachments: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch
>
>
> With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, the hadoop
task would fail. This doesn't seem to happen with 2.0.12.
> I have a hadoop map task that use BulkoutputFormat. If one of the cassandra nodes I'm
writing to is down, I'm getting the following syslog output from the map task:
> {noformat}
> 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop
library
> 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=MAP, sessionId=
> 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit
code 0
> 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin
: org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e
> 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b11111e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462
> 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully
loaded & initialized native-zlib library
> 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: Unable to initialize
MemoryMeter (jamm not specified as javaagent).  This means Cassandra will be unable to measure
object sizes accurately and may consequently OOM.
> 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not found. Native
methods will be disabled.
> 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: Opening /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_000000_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1
(1018 bytes)
> 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load
> 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:7
> 2015-02-10 10:54:20,714 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:8
> 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:7
> 2015-02-10 10:54:20,730 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:4
> 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:3
> 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:8
> 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:4
> 2015-02-10 10:54:20,770 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:6
> 2015-02-10 10:54:20,778 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:5
> 2015-02-10 10:54:20,786 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:3
> 2015-02-10 10:54:20,790 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:6
> 2015-02-10 10:54:20,867 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:5
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(152 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(491 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(375 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(322 bytes)
> 2015-02-10 10:54:20,898 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(169 bytes)
> 2015-02-10 10:54:20,983 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:7 is complete
> 2015-02-10 10:54:20,984 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:8 is complete
> 2015-02-10 10:54:20,985 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:3 is complete
> 2015-02-10 10:54:20,997 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:4 is complete
> 2015-02-10 10:54:21,067 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:6 is complete
> 2015-02-10 10:55:24,027 WARN org.apache.cassandra.streaming.DefaultConnectionFactory:
Failed attempt 1 to connect to /ipv6:prefix:1:441:0:0:0:5. Retrying in 20000 ms. (java.net.ConnectException:
Connection timed out)
> 2015-02-10 10:56:54,344 WARN org.apache.cassandra.streaming.DefaultConnectionFactory:
Failed attempt 2 to connect to /ipv6:prefix:1:441:0:0:0:5. Retrying in 40000 ms. (java.net.ConnectException:
Connection timed out)
> 2015-02-10 10:58:37,493 ERROR org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Streaming error occurred
> java.net.ConnectException: Connection timed out
> 	at sun.nio.ch.Net.connect0(Native Method)
> 	at sun.nio.ch.Net.connect(Net.java:465)
> 	at sun.nio.ch.Net.connect(Net.java:457)
> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> 	at java.nio.channels.SocketChannel.open(SocketChannel.java:184)
> 	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:134)
> 	at org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:52)
> 	at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:235)
> 	at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:77)
> 	at org.apache.cassandra.streaming.StreamSession$1.run(StreamSession.java:221)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-02-10 10:58:37,502 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:5 is complete
> 2015-02-10 10:58:37,507 WARN org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Stream failed
> {noformat}
> The hadoop task will then just sit there and do nothing forever, which indicates that
it's getting progress updates from the cassandra code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message