flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hui.pan" <hui....@wbkit.com>
Subject Re: A puzzy problem about flume Failover Sink Processor
Date Mon, 03 Apr 2017 10:24:05 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.5;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.5;'>
    <div>
<div>Dear&nbsp;<span style="font-size: 13px; line-height: 19.5px;">Chris:</span></div><div><span
style="font-size: 13px; line-height: 19.5px;">&nbsp; &nbsp; Thank you for your
reply.Do you mean how to maintain my namenodes when failover occurs?</span></div><div><span
style="font-size: 13px; line-height: 19.5px;">&nbsp; &nbsp; </span><font
size="2" style="font-size: 14px;"><span style="line-height: 19.5px;">&nbsp;I
use&nbsp;</span><span style="line-height: 1.5;">ZKFailoverController which
is&nbsp;</span><span style="line-height: 1.5;">is a new component in zookeeper
for my two namenodes,it is a &nbsp;</span></font><span style="line-height:
1.5;">Automatic Namenode failover process in Hadoop。</span></div><div><span
style="line-height: 1.5;">&nbsp; &nbsp;Then what's should I do?</span></div><div><span
style="line-height: 1.5;">&nbsp; &nbsp;Thanks.</span></div>
<div><span><br></span></div>
<div><br></div><div id="ntes-pcmail-signature" style="font-family:'微软雅黑'"><font
style="padding: 0; margin:0;">                </font>

</div>Hollis Pan
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">04/1/2017 19:15</span>,<a class="mail-to" style="text-decoration:none;color:#2a97ff;"
href="mailto:chris@hor.rocks">Chris Horrocks&lt;chris@hor.rocks&gt;</a> wrote:
</div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 13px; font-family:
'微软雅黑';">
The&nbsp;following error occurs&nbsp;when your flume agent tries to write to the standby
NameNode:
<div>
    <br>
</div>
<div>"<span style="background-color: rgba(255, 255, 255, 0);">Operation&nbsp;category&nbsp;WRITE&nbsp;is&nbsp;not&nbsp;supported&nbsp;in&nbsp;state&nbsp;standby"</span>
</div>
<div>
    <br>
</div>
<div>What failover mechanism are you using for your NameNodes?</div>
<div><span style="background-color: rgba(255, 255, 255, 0);"></span>
    <br>
    <div>
        <br>
    </div>
    <div>
        <br>
    </div>
    <div id="protonmail_mobile_signature_block">
        <div>--
            <br>
        </div>
        <div>Chris Horrocks</div>
    </div>
    <div>
        <br>
        <div>
            <div>
                <br>
            </div>On Sat, Apr 1, 2017 at 11:31 am, hui.pan@wbkit.com &lt;<a href="mailto:hui.pan@wbkit.com"
class="">hui.pan@wbkit.com</a>&gt; wrote:</div>
        <blockquote class="protonmail_quote" type="cite">
            <div><span></span><span style="color: rgb(0, 0, 0); background-color:
rgba(0, 0, 0, 0);">&nbsp; I&nbsp;have&nbsp;a&nbsp;puzzy&nbsp;problem&nbsp;about&nbsp;flume&nbsp;Failover&nbsp;Sink&nbsp;Processor,Description&nbsp;as&nbsp;below:<br>&nbsp;&nbsp;Enviroment&nbsp;is&nbsp;that&nbsp;we&nbsp;do&nbsp;HA&nbsp;configuration&nbsp;in&nbsp;HDFS,that&nbsp;is&nbsp;we&nbsp;have&nbsp;2&nbsp;namenode,1&nbsp;status&nbsp;for&nbsp;active(host&nbsp;name:n1)&nbsp;and&nbsp;another&nbsp;for&nbsp;standby(hostname:n2).<br>&nbsp;&nbsp;We&nbsp;configure&nbsp;our&nbsp;flume.conf&nbsp;like&nbsp;this:<br>&nbsp;&nbsp;&nbsp;#&nbsp;Flume&nbsp;agent&nbsp;config<br>&nbsp;&nbsp;&nbsp;c1.sources&nbsp;=&nbsp;source-cd&nbsp;<br>&nbsp;&nbsp;&nbsp;c1.channels&nbsp;=&nbsp;c_cd&nbsp;<br>&nbsp;&nbsp;&nbsp;c1.sinks&nbsp;=&nbsp;s_cd1&nbsp;&nbsp;s_cd2&nbsp;&nbsp;//s_cd1's&nbsp;hostname&nbsp;is&nbsp;n1,s_cd2's&nbsp;hotname&nbsp;is&nbsp;n2<br><br>&nbsp;&nbsp;&nbsp;c1.sinkgroups&nbsp;=&nbsp;g1&nbsp;<br>&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.sinks&nbsp;=&nbsp;s_cd1&nbsp;s_cd2&nbsp;<br>&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.type&nbsp;=&nbsp;failover<br>&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.priority.s_cd1&nbsp;=&nbsp;5<br>&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.priority.s_cd2&nbsp;=&nbsp;10<br>&nbsp;&nbsp;&nbsp;c1.sinkgroups.g1.processor.maxpenalty&nbsp;=&nbsp;10000<br><br>&nbsp;&nbsp;&nbsp;c1.sources.source-cd.type&nbsp;=&nbsp;org.apache.flume.source.taildir.TaildirSource<br>&nbsp;&nbsp;&nbsp;c1.sources.source-cd.channels&nbsp;=&nbsp;c_cd<br>&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;c1.channels.c_cd.type&nbsp;=&nbsp;memory<br>&nbsp;&nbsp;&nbsp;c1.channels.c_cd.capacity&nbsp;=&nbsp;5000000<br>&nbsp;&nbsp;&nbsp;c1.channels.c_cd.transactionCapacity&nbsp;=&nbsp;1000<br><br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.type&nbsp;=&nbsp;hdfs<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.path&nbsp;=&nbsp;hdfs://192.168.1.31:8020/PATH<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.channel&nbsp;=&nbsp;c_cd<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.batchSize&nbsp;=&nbsp;1000<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.useLocalTimeStamp&nbsp;=&nbsp;true<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.rollSize&nbsp;=&nbsp;0<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.rollCount&nbsp;=&nbsp;1000000<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd1.hdfs.rollInterval&nbsp;=&nbsp;600<br><br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.type&nbsp;=&nbsp;hdfs<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.path&nbsp;=&nbsp;hdfs://192.168.1.32:8020/PATH<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.channel&nbsp;=&nbsp;c_cd<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.batchSize&nbsp;=&nbsp;1000<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.useLocalTimeStamp&nbsp;=&nbsp;true&nbsp;<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.rollSize&nbsp;=&nbsp;0<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.rollCount&nbsp;=&nbsp;1000000<br>&nbsp;&nbsp;&nbsp;c1.sinks.s_cd2.hdfs.rollInterval&nbsp;=&nbsp;600<br>&nbsp;<br>&nbsp;&nbsp;&nbsp;In&nbsp;the&nbsp;flume.conf,we&nbsp;know&nbsp;that&nbsp;the&nbsp;priority&nbsp;of&nbsp;n2&nbsp;is&nbsp;higher&nbsp;than&nbsp;n1,so&nbsp;data&nbsp;should&nbsp;be&nbsp;put&nbsp;into&nbsp;n2.Unfortunately,n2&nbsp;is&nbsp;breakdown&nbsp;when&nbsp;run&nbsp;big&nbsp;data.according&nbsp;to&nbsp;flume&nbsp;<br>&nbsp;mechanism,unfinished&nbsp;data&nbsp;should&nbsp;be&nbsp;flow&nbsp;to&nbsp;n1,but&nbsp;the&nbsp;result&nbsp;is&nbsp;not&nbsp;like&nbsp;this.The&nbsp;log&nbsp;is&nbsp;like&nbsp;this:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br><br>2017-04-01&nbsp;15:31:28,285&nbsp;WARN&nbsp;org.apache.hadoop.hdfs.LeaseRenewer:&nbsp;Failed&nbsp;to&nbsp;renew&nbsp;lease&nbsp;for&nbsp;[DFSClient_NONMAPREDUCE_-1862131888_34]&nbsp;for&nbsp;525&nbsp;seconds.&nbsp;&nbsp;Will&nbsp;retry&nbsp;shortly&nbsp;...<br>java.net.ConnectException:&nbsp;Call&nbsp;From&nbsp;archive.cloudera.com/192.168.1.31&nbsp;to&nbsp;slave01:8020&nbsp;failed&nbsp;on&nbsp;connection&nbsp;exception:&nbsp;java.net.ConnectException:&nbsp;Connection&nbsp;refused;&nbsp;For&nbsp;more&nbsp;details&nbsp;see:&nbsp;&nbsp;http://wiki.apache.org/hadoop/ConnectionRefused<br><span
style="white-space: pre;">	</span>at&nbsp;sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown&nbsp;Source)
                <br><span style="white-space: pre;">	</span>at&nbsp;sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                <br><span style="white-space: pre;">	</span>at&nbsp;java.lang.reflect.Constructor.newInstance(Constructor.java:526)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client.call(Client.java:1475)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client.call(Client.java:1408)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
                <br><span style="white-space: pre;">	</span>at&nbsp;com.sun.proxy.$Proxy20.renewLease(Unknown&nbsp;Source)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:576)
                <br><span style="white-space: pre;">	</span>at&nbsp;sun.reflect.GeneratedMethodAccessor6.invoke(Unknown&nbsp;Source)
                <br><span style="white-space: pre;">	</span>at&nbsp;sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                <br><span style="white-space: pre;">	</span>at&nbsp;java.lang.reflect.Method.invoke(Method.java:606)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
                <br><span style="white-space: pre;">	</span>at&nbsp;com.sun.proxy.$Proxy21.renewLease(Unknown&nbsp;Source)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:922)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:423)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:448)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304)
                <br><span style="white-space: pre;">	</span>at&nbsp;java.lang.Thread.run(Thread.java:745)
                <br>Caused&nbsp;by:&nbsp;java.net.ConnectException:&nbsp;Connection&nbsp;refused
                <br><span style="white-space: pre;">	</span>at&nbsp;sun.nio.ch.SocketChannelImpl.checkConnect(Native&nbsp;Method)
                <br><span style="white-space: pre;">	</span>at&nbsp;sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Client.call(Client.java:1447)
                <br><span style="white-space: pre;">	</span>...&nbsp;16&nbsp;more
                <br>
                <br>&nbsp;According&nbsp;to&nbsp;this&nbsp;log,n2&nbsp;is&nbsp;breakdown&nbsp;meanwhile&nbsp;n1&nbsp;doesn't&nbsp;get&nbsp;any&nbsp;data.So&nbsp;we&nbsp;recovery&nbsp;n2.and&nbsp;the&nbsp;log&nbsp;is&nbsp;like&nbsp;this:
                <br>&nbsp;&nbsp;&nbsp;
                <br>
                <br>&nbsp;2017-03-31&nbsp;21:16:35,783&nbsp;WARN&nbsp;org.apache.flume.sink.hdfs.HDFSEventSink:&nbsp;HDFS&nbsp;IO&nbsp;error
                <br>org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):&nbsp;Operation&nbsp;category&nbsp;WRITE&nbsp;is&nbsp;not&nbsp;supported&nbsp;in&nbsp;state&nbsp;standby.&nbsp;Visit&nbsp;https://s.apache.org/sbnn-error
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1826)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1404)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:592)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:111)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:393)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
                <br><span style="white-space: pre;">	</span>at&nbsp;java.security.AccessController.doPrivileged(Native&nbsp;Method)
                <br><span style="white-space: pre;">	</span>at&nbsp;javax.security.auth.Subject.doAs(Subject.java:415)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
                <br><span style="white-space: pre;">	</span>at&nbsp;org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
                <br>&nbsp;
                <br>&nbsp;So&nbsp;how&nbsp;can&nbsp;we&nbsp;do&nbsp;to&nbsp;make&nbsp;n1&nbsp;get&nbsp;data&nbsp;when&nbsp;n2&nbsp;is&nbsp;breakdown.Is&nbsp;there&nbsp;any&nbsp;wrong&nbsp;about&nbsp;my&nbsp;configuration&nbsp;or&nbsp;this&nbsp;situation&nbsp;is&nbsp;unsuitable&nbsp;for&nbsp;this.Thanks&nbsp;a&nbsp;lot.
                <br>&nbsp; &nbsp;</span>
            </div>
            <div>
                <br>
            </div>
            <hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
            <div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE:
10pt"><div>Hollis @ Wbkit</div><div>Apr 1 .2017</div></div></span>
            </div>
        </blockquote>
    </div>
</div></blockquote><style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #3593db;
            cursor: pointer;
        }
    </style><!--😀-->
</div>
</body>
</html>

Mime
View raw message