cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8774) BulkOutputFormat never completes if streaming have errors
Date Thu, 12 Feb 2015 18:21:12 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318684#comment-14318684
] 

Alex Liu commented on CASSANDRA-8774:
-------------------------------------

Why not handle ExecutorException errors as a whole if there are no other none-critical root
causes?

> BulkOutputFormat never completes if streaming have errors
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-8774
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8774
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>            Reporter: Erik Forsberg
>             Fix For: 2.0.13
>
>         Attachments: 0001-CASSANDRA-8774-Handle-StreamException-when-bulkloadi.patch
>
>
> With BulkoutputFormat in Cassandra 1.2.18, if any streaming errors occured, the hadoop
task would fail. This doesn't seem to happen with 2.0.12.
> I have a hadoop map task that use BulkoutputFormat. If one of the cassandra nodes I'm
writing to is down, I'm getting the following syslog output from the map task:
> {noformat}
> 2015-02-10 10:54:15,162 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop
library
> 2015-02-10 10:54:15,601 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM
Metrics with processName=MAP, sessionId=
> 2015-02-10 10:54:15,901 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit
code 0
> 2015-02-10 10:54:15,907 INFO org.apache.hadoop.mapred.Task:  Using ResourceCalculatorPlugin
: org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4984451e
> 2015-02-10 10:54:16,110 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hdpmt01.osp-hadoop.osa:9000/user/jenkins/syst/5ef13_osp/tvstore/sumcombinations/hourly/2015021002/per_period-5ba2faa4b11111e4aa21fa163e82bc46-sumcombinations/0/data/part-00047:0+462
> 2015-02-10 10:54:16,739 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully
loaded & initialized native-zlib library
> 2015-02-10 10:54:16,740 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,741 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
> 2015-02-10 10:54:16,927 ERROR org.apache.cassandra.cql3.QueryProcessor: Unable to initialize
MemoryMeter (jamm not specified as javaagent).  This means Cassandra will be unable to measure
object sizes accurately and may consequently OOM.
> 2015-02-10 10:54:17,780 INFO org.apache.cassandra.utils.CLibrary: JNA not found. Native
methods will be disabled.
> 2015-02-10 10:54:19,446 INFO org.apache.cassandra.io.sstable.SSTableReader: Opening /opera/log1/hadoop/mapred/local/taskTracker/jenkins/jobcache/job_201502041226_13903/attempt_201502041226_13903_m_000000_0/work/tmp/syst5ef13osp/Data_hourly/syst5ef13osp-Data_hourly-jb-1
(1018 bytes)
> 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Executing streaming plan for Bulk Load
> 2015-02-10 10:54:20,713 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:7
> 2015-02-10 10:54:20,714 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:8
> 2015-02-10 10:54:20,715 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:7
> 2015-02-10 10:54:20,730 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:4
> 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:3
> 2015-02-10 10:54:20,731 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:8
> 2015-02-10 10:54:20,750 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:4
> 2015-02-10 10:54:20,770 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:6
> 2015-02-10 10:54:20,778 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Beginning stream session with /ipv6:prefix:1:441:0:0:0:5
> 2015-02-10 10:54:20,786 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:3
> 2015-02-10 10:54:20,790 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:6
> 2015-02-10 10:54:20,867 INFO org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Starting streaming to /ipv6:prefix:1:441:0:0:0:5
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(152 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(491 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(375 bytes)
> 2015-02-10 10:54:20,897 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(322 bytes)
> 2015-02-10 10:54:20,898 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Prepare completed. Receiving 0 files(0 bytes), sending
1 files(169 bytes)
> 2015-02-10 10:54:20,983 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:7 is complete
> 2015-02-10 10:54:20,984 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:8 is complete
> 2015-02-10 10:54:20,985 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:3 is complete
> 2015-02-10 10:54:20,997 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:4 is complete
> 2015-02-10 10:54:21,067 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:6 is complete
> 2015-02-10 10:55:24,027 WARN org.apache.cassandra.streaming.DefaultConnectionFactory:
Failed attempt 1 to connect to /ipv6:prefix:1:441:0:0:0:5. Retrying in 20000 ms. (java.net.ConnectException:
Connection timed out)
> 2015-02-10 10:56:54,344 WARN org.apache.cassandra.streaming.DefaultConnectionFactory:
Failed attempt 2 to connect to /ipv6:prefix:1:441:0:0:0:5. Retrying in 40000 ms. (java.net.ConnectException:
Connection timed out)
> 2015-02-10 10:58:37,493 ERROR org.apache.cassandra.streaming.StreamSession: [Stream #29f27cd0-b113-11e4-a465-91cc09fc46f1]
Streaming error occurred
> java.net.ConnectException: Connection timed out
> 	at sun.nio.ch.Net.connect0(Native Method)
> 	at sun.nio.ch.Net.connect(Net.java:465)
> 	at sun.nio.ch.Net.connect(Net.java:457)
> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> 	at java.nio.channels.SocketChannel.open(SocketChannel.java:184)
> 	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:134)
> 	at org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:52)
> 	at org.apache.cassandra.streaming.StreamSession.createConnection(StreamSession.java:235)
> 	at org.apache.cassandra.streaming.ConnectionHandler.initiate(ConnectionHandler.java:77)
> 	at org.apache.cassandra.streaming.StreamSession$1.run(StreamSession.java:221)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> 2015-02-10 10:58:37,502 INFO org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Session with /ipv6:prefix:1:441:0:0:0:5 is complete
> 2015-02-10 10:58:37,507 WARN org.apache.cassandra.streaming.StreamResultFuture: [Stream
#29f27cd0-b113-11e4-a465-91cc09fc46f1] Stream failed
> {noformat}
> The hadoop task will then just sit there and do nothing forever, which indicates that
it's getting progress updates from the cassandra code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message