flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "tycho_yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLUME-3021) flume Elasticsearch 5.0 not support
Date Mon, 20 Mar 2017 08:30:41 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932298#comment-15932298
] 

tycho_yang edited comment on FLUME-3021 at 3/20/17 8:30 AM:
------------------------------------------------------------

Use your binary package, not stable enough,
70000 data transmission through continuous testing, found that there are about 2 times in
the exception of the 5, the same configuration of the same data, configuration data with no
problem this can be determined before the same configuration in ES1.7 run OK

	at java.lang.Thread.run(Thread.java:745)
2017-03-20 15:08:32,799 (elasticsearch[_client_][generic][T#1]) [ERROR - org.elasticsearch.transport.netty4.Netty4InternalESLogger.error(Netty4InternalESLogger.java:184)]
Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
	at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113)
	at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
	at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1011)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
	at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:422)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:93)
	at org.elasticsearch.transport.TcpTransport$ScheduledPing.doRunInLifecycle(TcpTransport.java:315)
	at org.elasticsearch.common.util.concurrent.AbstractLifecycleRunnable.doRun(AbstractLifecycleRunnable.java:67)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-03-20 15:08:32,871 (elasticsearch[_client_][generic][T#1]) [ERROR - org.elasticsearch.transport.netty4.Netty4InternalESLogger.error(Netty4InternalESLogger.java:184)]
Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:423)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:93)
	at org.elasticsearch.transport.TcpTransport$ScheduledPing.doRunInLifecycle(TcpTransport.java:315)
	at org.elasticsearch.common.util.concurrent.AbstractLifecycleRunnable.doRun(AbstractLifecycleRunnable.java:67)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)





===========================================================================================================================================


The same test error
2017-03-20 16:05:14,900 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:221)]
Failed to commit transaction. Transaction rolled back.
NoNodeAvailableException[None of the configured nodes were available: [{node-1}{Xb_O5doNTMuS2u4pbC4ILQ}{O4cUzwIPQ9uVMWkygbwhtw}{192.168.3.63}{192.168.3.63:9300}]];
nested: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected];
	at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:288)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476)
	at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
	at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:356)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
	at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:184)
	at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected]
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590)
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113)
	at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472)
	... 13 more
2017-03-20 16:05:14,954 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)]
Unable to deliver event. Exception follows.
NoNodeAvailableException[None of the configured nodes were available: [{node-1}{Xb_O5doNTMuS2u4pbC4ILQ}{O4cUzwIPQ9uVMWkygbwhtw}{192.168.3.63}{192.168.3.63:9300}]];
nested: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected];
	at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:288)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476)
	at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
	at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:356)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
	at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:184)
	at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected]
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590)
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113)
	at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472)
	... 13 more




was (Author: ytc301):
Use your binary package, not stable enough,
70000 data transmission through continuous testing, found that there are about 2 times in
the exception of the 5, the same configuration of the same data, configuration data with no
problem this can be determined before the same configuration in ES1.7 run OK

	at java.lang.Thread.run(Thread.java:745)
2017-03-20 15:08:32,799 (elasticsearch[_client_][generic][T#1]) [ERROR - org.elasticsearch.transport.netty4.Netty4InternalESLogger.error(Netty4InternalESLogger.java:184)]
Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
	at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113)
	at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
	at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1011)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:826)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
	at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:422)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:93)
	at org.elasticsearch.transport.TcpTransport$ScheduledPing.doRunInLifecycle(TcpTransport.java:315)
	at org.elasticsearch.common.util.concurrent.AbstractLifecycleRunnable.doRun(AbstractLifecycleRunnable.java:67)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
2017-03-20 15:08:32,871 (elasticsearch[_client_][generic][T#1]) [ERROR - org.elasticsearch.transport.netty4.Netty4InternalESLogger.error(Netty4InternalESLogger.java:184)]
Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
	at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
	at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
	at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)
	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:423)
	at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:93)
	at org.elasticsearch.transport.TcpTransport$ScheduledPing.doRunInLifecycle(TcpTransport.java:315)
	at org.elasticsearch.common.util.concurrent.AbstractLifecycleRunnable.doRun(AbstractLifecycleRunnable.java:67)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:596)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)





==========================================================================


The same test error
2017-03-20 16:05:14,900 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:221)]
Failed to commit transaction. Transaction rolled back.
NoNodeAvailableException[None of the configured nodes were available: [{node-1}{Xb_O5doNTMuS2u4pbC4ILQ}{O4cUzwIPQ9uVMWkygbwhtw}{192.168.3.63}{192.168.3.63:9300}]];
nested: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected];
	at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:288)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476)
	at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
	at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:356)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
	at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:184)
	at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected]
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590)
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113)
	at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472)
	... 13 more
2017-03-20 16:05:14,954 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)]
Unable to deliver event. Exception follows.
NoNodeAvailableException[None of the configured nodes were available: [{node-1}{Xb_O5doNTMuS2u4pbC4ILQ}{O4cUzwIPQ9uVMWkygbwhtw}{192.168.3.63}{192.168.3.63:9300}]];
nested: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected];
	at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:288)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476)
	at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
	at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:356)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
	at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:184)
	at org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not connected]
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590)
	at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113)
	at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496)
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472)
	... 13 more



> flume Elasticsearch 5.0 not support
> -----------------------------------
>
>                 Key: FLUME-3021
>                 URL: https://issues.apache.org/jira/browse/FLUME-3021
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.7.0
>            Reporter: tycho_yang
>            Assignee: Yonghao Zou
>             Fix For: 1.8.0
>
>         Attachments: FLUME-3021-0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message