flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Issue with HBase Sink in Flume ( 1.3.0)
Date Tue, 18 Feb 2014 19:41:28 GMT
Looks like you are using Flume 1.3.0. Is there a reason for not using a newer version? Flume 1.4.0 is not almost 6 months old. 1.3.0 did have a known performance issue which was the reason 1.3.1 was release almost immediately after. 


Thanks,
Hari


On Tuesday, February 18, 2014 at 11:23 AM, Kris Ogirri wrote:

> Hello Hari,
> 
> I didn't know it was a holiday in the US. 
> 
> Please see version information below:
> 
> Hbase:
> HBase Shell; enter 'help<RETURN>' for list of supported commands.
> Type "exit<RETURN>" to leave the HBase Shell
> Version 0.94.3, rab548827f0c52211c1d67437484fcba635072767, Wed Jul 31 18:13:25 PDT 2013
> 
> 
> Flume: 
> [biadmin@bivm bin]$ ./flume-ng version
> Flume 1.3.0
> Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
> Revision: abbccbd2ff14dd6fed2a8a3891eb51aff985e9f5
> Compiled by jenkins on Wed Jun 12 19:16:33 PDT 2013
> From source with checksum dce204011600e67e1455971266d3da07
> 
> 
> Thanks for all the assistance.
> 
> BR,
> 
> 
> 
> On 18 February 2014 20:14, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > Hi Kris, 
> > 
> > Please realize that people usually work on their own time on these mailing lists and since your first message was sent on a Monday early morning on a long weekend in the US, others may not have seen your message either.  
> > 
> > Are you running Apache Flume and Apache HBase? If yes, what versions (output of flume-ng version and hbase version)? 
> > 
> > 
> > Thanks,
> > Hari
> > 
> > 
> > On Tuesday, February 18, 2014 at 10:22 AM, Kris Ogirri wrote:
> > 
> > > Hi,
> > > 
> > > Cant anybody help with this? I am thinking its a small issue because everything seems to work fine but the data from the Channel never gets persisted into Hbase?
> > > 
> > > I have added the description of the Hbase tables:
> > > 
> > > hbase(main):005:0> describe 'telco_cdr_rec'
> > > DESCRIPTION                                          ENABLED                    
> > >  {NAME => 'telco_cdr_rec', FAMILIES => [{NAME => 'co true                       
> > >  lfam', REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS                            
> > >   => 'false', COMPRESSION => 'NONE', ENCODE_ON_DISK                             
> > >  => 'true', BLOCKCACHE => 'true', MIN_VERSIONS => '0                            
> > >  ', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'fal                            
> > >  se', BLOOMFILTER => 'NONE', TTL => '2147483647 (tel:2147483647)', VE                            
> > >  RSIONS => '3', BLOCKSIZE => '65536'}]}                                         
> > > 1 row(s) in 0.1600 seconds
> > > 
> > > 
> > > If no one can help with the problem, can anyone provide a link to the Flume -> Zookeeper -> Hbase Internal documentation so I can trace where the error lies.
> > > 
> > > Are there Zookeeper log files where I can analyse whether Flume actually sends the Txns to Hbase via Zookeeper?
> > > 
> > > 
> > > 
> > > On 17 February 2014 16:38, Kris Ogirri <kanirip@gmail.com (mailto:kanirip@gmail.com)> wrote:
> > > > Hello Jeff,
> > > > 
> > > > Please find below requested logs.. Initiation part of the logs were unfortunately not included. I can run these again if necessary but the Zookeeper connection is included in the logs.
> > > > 
> > > > 
> > > > 14/02/17 10:26:12 INFO properties.PropertiesFileConfigurationProvider: created channel ch2
> > > > 14/02/17 10:26:13 INFO sink.DefaultSinkFactory: Creating instance of sink: hbase-sink, type: org.apache.flume.sink.hbase.HBaseSink
> > > > 14/02/17 10:26:13 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: hdfs
> > > > 14/02/17 10:26:14 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
> > > > 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{exec-source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec-source,state:IDLE} }} sinkRunners:{hbase-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{ name:null counters:{} } }, hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{ name:null counters:{} } }} channels:{ch1=FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }, ch2=FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }} }
> > > > 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch1
> > > > 14/02/17 10:26:14 INFO file.FileChannel: Starting FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }...
> > > > 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel ch2
> > > > 14/02/17 10:26:14 INFO file.FileChannel: Starting FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }...
> > > > 14/02/17 10:26:14 INFO file.Log: Encryption is not enabled
> > > > 14/02/17 10:26:14 INFO file.Log: Replay started
> > > > 14/02/17 10:26:14 INFO file.Log: Encryption is not enabled
> > > > 14/02/17 10:26:14 INFO file.Log: Replay started
> > > > 14/02/17 10:26:14 INFO file.Log: Found NextFileID 7, from [/home/biadmin/.flume/file-channel/data/log-7, /home/biadmin/.flume/file-channel/data/log-6]
> > > > 14/02/17 10:26:14 INFO file.Log: Found NextFileID 6, from [/home/biadmin/.flume/file-channel2/data/log-6, /home/biadmin/.flume/file-channel2/data/log-4, /home/biadmin/.flume/file-channel2/data/log-5]
> > > > 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Starting up with /home/biadmin/.flume/file-channel2/checkpoint/checkpoint and /home/biadmin/.flume/file-channel2/checkpoint/checkpoint.meta
> > > > 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Reading checkpoint metadata from /home/biadmin/.flume/file-channel2/checkpoint/checkpoint.meta
> > > > 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Starting up with /home/biadmin/.flume/file-channel/checkpoint/checkpoint and /home/biadmin/.flume/file-channel/checkpoint/checkpoint.meta
> > > > 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Reading checkpoint metadata from /home/biadmin/.flume/file-channel/checkpoint/checkpoint.meta
> > > > 14/02/17 10:26:14 INFO file.Log: Last Checkpoint Mon Feb 17 10:21:35 EST 2014, queue depth = 0
> > > > 14/02/17 10:26:14 INFO file.Log: Last Checkpoint Mon Feb 17 10:21:31 EST 2014, queue depth = 0
> > > > 14/02/17 10:26:14 INFO file.Log: Replaying logs with v2 replay logic
> > > > 14/02/17 10:26:14 INFO file.Log: Replaying logs with v2 replay logic
> > > > 14/02/17 10:26:14 INFO file.ReplayHandler: Starting replay of [/home/biadmin/.flume/file-channel/data/log-6, /home/biadmin/.flume/file-channel/data/log-7]
> > > > 14/02/17 10:26:14 INFO file.ReplayHandler: Starting replay of [/home/biadmin/.flume/file-channel2/data/log-4, /home/biadmin/.flume/file-channel2/data/log-5, /home/biadmin/.flume/file-channel2/data/log-6]
> > > > 14/02/17 10:26:14 INFO file.ReplayHandler: Replaying /home/biadmin/.flume/file-channel/data/log-6
> > > > 14/02/17 10:26:14 INFO file.ReplayHandler: Replaying /home/biadmin/.flume/file-channel2/data/log-4
> > > > 14/02/17 10:26:14 INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
> > > > 14/02/17 10:26:14 INFO tools.DirectMemoryUtils: Direct Memory Allocation:  Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 20971520, Remaining = 20971520
> > > > 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position: 32040
> > > > 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying /home/biadmin/.flume/file-channel/data/log-7
> > > > 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position: 2496
> > > > 14/02/17 10:26:16 WARN file.LogFile: Checkpoint for file(/home/biadmin/.flume/file-channel2/data/log-4) is: 1392407375821, which is beyond the requested checkpoint time: 1392650490155 and position 0
> > > > 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying /home/biadmin/.flume/file-channel2/data/log-5
> > > > 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position: 22843
> > > > 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 22843 in /home/biadmin/.flume/file-channel2/data/log-5
> > > > 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying /home/biadmin/.flume/file-channel2/data/log-6
> > > > 14/02/17 10:26:16 WARN file.LogFile: Checkpoint for file(/home/biadmin/.flume/file-channel2/data/log-6) is: 1392650490155, which is beyond the requested checkpoint time: 1392650490155 and position 0
> > > > 14/02/17 10:26:16 INFO file.ReplayHandler: read: 0, put: 0, take: 0, rollback: 0, commit: 0, skip: 0, eventCount:0
> > > > 14/02/17 10:26:16 INFO file.Log: Rolling /home/biadmin/.flume/file-channel2/data
> > > > 14/02/17 10:26:16 INFO file.Log: Roll start /home/biadmin/.flume/file-channel2/data
> > > > 14/02/17 10:26:16 INFO file.LogFile: Opened /home/biadmin/.flume/file-channel2/data/log-7
> > > > 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 2496 in /home/biadmin/.flume/file-channel/data/log-7
> > > > 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 32071 in /home/biadmin/.flume/file-channel/data/log-6
> > > > 14/02/17 10:26:16 INFO file.ReplayHandler: read: 1, put: 0, take: 0, rollback: 0, commit: 0, skip: 1, eventCount:0
> > > > 14/02/17 10:26:16 INFO file.Log: Rolling /home/biadmin/.flume/file-channel/data
> > > > 14/02/17 10:26:16 INFO file.Log: Roll start /home/biadmin/.flume/file-channel/data
> > > > 14/02/17 10:26:16 INFO file.LogFile: Opened /home/biadmin/.flume/file-channel/data/log-8
> > > > 14/02/17 10:26:16 INFO file.Log: Roll end
> > > > 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 0
> > > > 14/02/17 10:26:16 INFO file.Log: Roll end
> > > > 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 0
> > > > 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650774387, queueSize: 0, queueHead: 10516
> > > > 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650774388, queueSize: 0, queueHead: 223682
> > > > 14/02/17 10:26:16 INFO file.LogFileV3: Updating log-7.meta currentPosition = 0, logWriteOrderID = 1392650774387
> > > > 14/02/17 10:26:16 INFO file.LogFileV3: Updating log-8.meta currentPosition = 0, logWriteOrderID = 1392650774388
> > > > 14/02/17 10:26:16 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 0 logWriteOrderID: 1392650774387
> > > > 14/02/17 10:26:16 INFO file.FileChannel: Queue Size after replay: 0 [channel=ch2]
> > > > 14/02/17 10:26:17 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 0 logWriteOrderID: 1392650774388
> > > > 14/02/17 10:26:17 INFO file.FileChannel: Queue Size after replay: 0 [channel=ch1]
> > > > 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: ch2, registered successfully.
> > > > 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch2 started
> > > > 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: ch1, registered successfully.
> > > > 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 started
> > > > 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink hbase-sink
> > > > 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink hdfs-sink
> > > > 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting Source exec-source
> > > > 14/02/17 10:26:17 INFO source.ExecSource: Exec source starting with command:tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> > > > 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: hdfs-sink, registered successfully.
> > > > 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink started
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5--1, built on 01/23/2013 14:29 GMT
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:host.name (http://host.name)=bivm
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.vendor=IBM Corporation
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/ibm/biginsights/jdk/jre
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.class.path=conf:/opt/ibm/biginsights/flume/lib/snappy-java-1.0.4.1.jar:/opt/ibm/biginsights/flume/lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/flume/lib/jackson-mapper-asl-1.9.3.jar:/opt/ibm/biginsights/flume/lib/flume-avro-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-jdbc-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/velocity-1.7.jar:/opt/ibm/biginsights/flume/lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/flume/lib/flume-ng-node-1.3.0.jar:/opt/ibm/biginsights/flume/lib/commons-dbcp-1.4.jar:/opt/ibm/biginsights/flume/lib/log4j-1.2.16.jar:/opt/ibm/biginsights/flume/lib/flume-hdfs-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/asynchbase-1.2.0.jar:/opt/ibm/biginsights/flume/lib/flume-recoverable-memory-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/async-1.3.1.jar:/opt/ibm/biginsights/flume/lib/slf4j-log4j12-1.6.1.jar:/opt/ibm/biginsights/flume/lib/flume-thrift-source-1.3.0.jar:/opt/ibm/biginsight
s/flume/lib/flume-file-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/libthrift-0.6.1.jar:/opt/ibm/biginsights/flume/lib/avro-1.7.2.jar:/opt/ibm/biginsights/flume/lib/jetty-6.1.26.jar:/opt/ibm/biginsights/flume/lib/jackson-core-asl-1.9.3.jar:/opt/ibm/biginsights/flume/lib/servlet-api-2.5-20110124.jar:/opt/ibm/biginsights/flume/lib/flume-ng-elasticsearch-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-ng-configuration-1.3.0.jar:/opt/ibm/biginsights/flume/lib/jsr305-1.3.9.jar:/opt/ibm/biginsights/flume/lib/irclib-1.10.jar:/opt/ibm/biginsights/flume/lib/commons-cli-1.2.jar:/opt/ibm/biginsights/flume/lib/derby-10.8.3.1.jar:/opt/ibm/biginsights/flume/lib/flume-ng-log4jappender-1.3.0.jar:/opt/ibm/biginsights/flume/lib/netty-3.4.0.Final.jar:/opt/ibm/biginsights/flume/lib/flume-irc-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/jcl-over-slf4j-1.7.2.jar:/opt/ibm/biginsights/flume/lib/slf4j-api-1.6.1.jar:/opt/ibm/biginsights/flume/lib/joda-time-2.1.jar:/opt/ibm/biginsights/flume/lib/commo
ns-lang-2.5.jar:/opt/ibm/biginsights/flume/lib/commons-io-2.1.jar:/opt/ibm/biginsights/flume/lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/flume/lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/flume/lib/commons-pool-1.5.4.jar:/opt/ibm/biginsights/flume/lib/flume-ng-hbase-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/protobuf-java-2.4.1.jar:/opt/ibm/biginsights/flume/lib/flume-scribe-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-ng-core-1.3.0.jar:/opt/ibm/biginsights/flume/lib/gson-2.2.2.jar:/opt/ibm/biginsights/flume/lib/flume-ng-sdk-1.3.0.jar:/opt/ibm/biginsights/flume/lib/avro-ipc-1.7.2.jar:/opt/ibm/biginsights/flume/lib/guava-10.0.1.jar:/opt/ibm/biginsights/flume/lib/paranamer-2.3.jar:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/IHC/libexec/..:/opt/ibm/biginsights/IHC/libexec/../hadoop-core-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/adaptive-mr.jar:/opt/ibm/biginsights/IHC/libexec/../lib/asm-3.2.jar:/opt/ibm/bigi
nsights/IHC/libexec/../lib/aspectjrt-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjtools-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/biginsights-sftpfs-1.0.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-cli-1.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-codec-1.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-daemon-1.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-digester-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-el-1.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-io-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-lang-2.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/com
mons-logging-api-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-math-2.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-net-3.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/core-3.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftplet-api-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftpserver-core-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/guardium-proxy.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ibm-compression.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/ibm
/biginsights/IHC/libexec/../lib/jdeb-0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-core-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-json-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-server-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jets3t-0.6.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.42.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.43.jar:/opt/ibm/biginsights/IHC/libexec/../lib/junit-4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/log4j-1.2.16.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mockito-all-1.8.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/oro-2.0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/IHC/libexec/../lib/workflowScheduler.jar:/opt/ibm/biginsights/IHC/libexec/../lib/xmlenc-0.52.jar:/opt/ibm/
biginsights/IHC/libexec/../lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/IHC/:/opt/ibm/biginsights/IHC/:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/IHC/lib/biginsights-gpfs-1.1.1.jar:/opt/ibm/biginsights/IHC/hadoop-core.jar:/opt/ibm/biginsights/IHC/lib/biginsights-gpfs-1.1.1.jar:/opt/ibm/biginsights/IHC/hadoop-core.jar:/home/biadmin/twitter4j/lib/twitter4j-media-support-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-core-3.0.3.jar:home/biadmin/twitter4j/lib/twitter4j-async-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-stream-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-media-support-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-core-3.0.3.jar:home/biadmin/twitter4j/lib/twitter4j-async-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-stream-3.0.3.jar:/opt/ibm/biginsights/jdk/lib/tools
.jar:/opt/ibm/biginsights/hbase:/opt/ibm/biginsights/hbase/hbase-0.94.3-security.jar:/opt/ibm/biginsights/hbase/hbase-0.94.3-security-tests.jar:/opt/ibm/biginsights/hbase/hbase.jar:/opt/ibm/biginsights/hbase/lib/activation-1.1.jar:/opt/ibm/biginsights/hbase/lib/asm-3.1.jar:/opt/ibm/biginsights/hbase/lib/avro-1.7.2.jar:/opt/ibm/biginsights/hbase/lib/avro-ipc-1.7.2.jar:/opt/ibm/biginsights/hbase/lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/hbase/lib/commons-cli-1.2.jar:/opt/ibm/biginsights/hbase/lib/commons-codec-1.4.jar:/opt/ibm/biginsights/hbase/lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/hbase/lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/hbase/lib/commons-digester-1.8.jar:/opt/ibm/biginsights/hbase/lib/commons-el-1.0.jar:/opt/ibm/biginsights/hbase/lib/commons-httpclient-3.1.jar:/opt/ibm/biginsights/hbase/lib/commons-io-2.1.jar:/opt/ibm/biginsights/hbase/lib/commons-lang-2.5.jar:/opt/ibm/biginsights/hbase/lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights
/hbase/lib/commons-math-2.2.jar:/opt/ibm/biginsights/hbase/lib/commons-net-3.1.jar:/opt/ibm/biginsights/hbase/lib/core-3.1.1.jar:/opt/ibm/biginsights/hbase/lib/guardium-proxy.jar:/opt/ibm/biginsights/hbase/lib/guava-11.0.2.jar:/opt/ibm/biginsights/hbase/lib/hadoop-core.jar:/opt/ibm/biginsights/hbase/lib/hadoop-tools-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/high-scale-lib-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/httpclient-4.1.2.jar:/opt/ibm/biginsights/hbase/lib/httpcore-4.1.3.jar:/opt/ibm/biginsights/hbase/lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-jaxrs-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-xc-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jamon-runtime-2.3.1.jar:/opt/ibm/biginsights/hbase/lib/jasper-compiler-5.5.23.jar:/opt/ibm/biginsights/hbase/lib/jasper-runtime-5.5.23.jar:/opt/ibm/biginsights/hbase/lib/jaxb-api-2.1.jar:/opt/ibm/biginsights/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/ibm/biginsi
ghts/hbase/lib/jersey-core-1.8.jar:/opt/ibm/biginsights/hbase/lib/jersey-json-1.8.jar:/opt/ibm/biginsights/hbase/lib/jersey-server-1.8.jar:/opt/ibm/biginsights/hbase/lib/jettison-1.1.jar:/opt/ibm/biginsights/hbase/lib/jetty-6.1.26.jar:/opt/ibm/biginsights/hbase/lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/hbase/lib/jruby-complete-1.6.5.1.jar:/opt/ibm/biginsights/hbase/lib/jsp-2.1-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/lib/jsr305-1.3.9.jar:/opt/ibm/biginsights/hbase/lib/junit-4.10-HBASE-1.jar:/opt/ibm/biginsights/hbase/lib/libthrift-0.8.0.jar:/opt/ibm/biginsights/hbase/lib/log4j-1.2.16.jar:/opt/ibm/biginsights/hbase/lib/metrics-core-2.1.2.jar:/opt/ibm/biginsights/hbase/lib/netty-3.2.4.Final.jar:/opt/ibm/biginsights/hbase/lib/netty-3.4.0.Final.jar:/opt/ibm/biginsights/hbase/lib/protobuf-java-2.4.0a.jar:/opt/ibm/biginsights/hbase/lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/hba
se/lib/servlet-api-2.5-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/snappy-java-1.0.4.1.jar:/opt/ibm/biginsights/hbase/lib/stax-api-1.0.1.jar:/opt/ibm/biginsights/hbase/lib/velocity-1.7.jar:/opt/ibm/biginsights/hbase/lib/xmlenc-0.52.jar:/opt/ibm/biginsights/hbase/lib/xml-ibm.jar:/opt/ibm/biginsights/hbase/lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/hbase/lib/zookeeper.jar:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/IHC/libexec/..:/opt/ibm/biginsights/IHC/libexec/../hadoop-core-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/adaptive-mr.jar:/opt/ibm/biginsights/IHC/libexec/../lib/asm-3.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjrt-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjtools-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/biginsights-sftpfs-1.0.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-cli-1.2.jar:/opt/ibm/biginsights/IHC/libe
xec/../lib/commons-codec-1.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-daemon-1.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-digester-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-el-1.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-io-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-lang-2.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-api-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-math-2.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-net-3.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/core-3.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftplet-api-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftpserver-core-1.0.6.jar:/opt/ibm/
biginsights/IHC/libexec/../lib/guardium-proxy.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ibm-compression.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jdeb-0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-core-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-json-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-server-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jets3t-0.6.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-6.1.26.j
ar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.42.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.43.jar:/opt/ibm/biginsights/IHC/libexec/../lib/junit-4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/log4j-1.2.16.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mockito-all-1.8.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/oro-2.0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/IHC/libexec/../lib/workflowScheduler.jar:/opt/ibm/biginsights/IHC/libexec/../lib/xmlenc-0.52.jar:/opt/ibm/biginsights/IHC/libexec/../lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/hbase/conf
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/opt/ibm/biginsights/IHC/libexec/../lib/native/Linux-amd64-64:/opt/ibm/biginsights/IHC/libexec/../lib/native/Linux-amd64-64
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:java.compiler=j9jit24
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:os.name (http://os.name)=Linux
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.18-194.17.4.el5
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:user.name (http://user.name)=biadmin
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/biadmin
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/ibm/biginsights/flume/bin
> > > > 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=bivm:2181 sessionTimeout=180000 watcher=hconnection
> > > > 14/02/17 10:26:17 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 20984@bivm
> > > > 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Opening socket connection to server bivm/192.168.37.128:2181 (http://192.168.37.128:2181). Will not attempt to authenticate using SASL (Unable to locate a login configuration)
> > > > 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Socket connection established to bivm/192.168.37.128:2181 (http://192.168.37.128:2181), initiating session
> > > > 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Session establishment complete on server bivm/192.168.37.128:2181 (http://192.168.37.128:2181), sessionid = 0x144401355b4001d, negotiated timeout = 60000
> > > > 14/02/17 10:29:56 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 60
> > > > 14/02/17 10:29:56 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650774536, queueSize: 60, queueHead: 10514
> > > > 14/02/17 10:29:56 INFO file.LogFileV3: Updating log-7.meta currentPosition = 32036, logWriteOrderID = 1392650774536
> > > > 14/02/17 10:29:57 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 32036 logWriteOrderID: 1392650774536
> > > > 14/02/17 10:29:57 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel2/data/log-4
> > > > 14/02/17 10:29:57 INFO file.Log: Removing old log /home/biadmin/.flume/file-channel2/data/log-4, result = true, minFileID 7
> > > > 14/02/17 10:29:57 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel2/data/log-5
> > > > 14/02/17 10:29:57 INFO file.Log: Removing old log /home/biadmin/.flume/file-channel2/data/log-5, result = true, minFileID 7
> > > > 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 460
> > > > 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650775504, queueSize: 520, queueHead: 10514
> > > > 14/02/17 10:29:58 INFO file.LogFileV3: Updating log-7.meta currentPosition = 277565, logWriteOrderID = 1392650775504
> > > > 14/02/17 10:29:58 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 277565 logWriteOrderID: 1392650775504
> > > > 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 540
> > > > 14/02/17 10:29:59 INFO hdfs.BucketWriter: Creating hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998182.tmp
> > > > 14/02/17 10:29:59 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 423
> > > > 14/02/17 10:30:00 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650775933, queueSize: 137, queueHead: 10917
> > > > 14/02/17 10:30:00 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650775934, queueSize: 539, queueHead: 223681
> > > > 14/02/17 10:30:01 INFO file.LogFileV3: Updating log-7.meta currentPosition = 304892, logWriteOrderID = 1392650775933
> > > > 14/02/17 10:30:01 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 304892 logWriteOrderID: 1392650775933
> > > > 14/02/17 10:30:02 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 137
> > > > 14/02/17 10:30:02 INFO file.LogFileV3: Updating log-8.meta currentPosition = 288266, logWriteOrderID = 1392650775934
> > > > 14/02/17 10:30:02 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776074, queueSize: 0, queueHead: 11054
> > > > 14/02/17 10:30:04 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 288266 logWriteOrderID: 1392650775934
> > > > 14/02/17 10:30:04 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel/data/log-6
> > > > 14/02/17 10:30:04 INFO file.Log: Removing old log /home/biadmin/.flume/file-channel/data/log-6, result = true, minFileID 8
> > > > 14/02/17 10:30:05 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 29
> > > > 14/02/17 10:30:06 INFO file.LogFileV3: Updating log-7.meta currentPosition = 310581, logWriteOrderID = 1392650776074
> > > > 14/02/17 10:30:13 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776105, queueSize: 550, queueHead: 223690
> > > > 14/02/17 10:30:19 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 310581 logWriteOrderID: 1392650776074
> > > > 14/02/17 10:30:21 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 20
> > > > 14/02/17 10:30:29 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776127, queueSize: 20, queueHead: 11052
> > > > 14/02/17 10:30:29 INFO file.LogFileV3: Updating log-8.meta currentPosition = 299362, logWriteOrderID = 1392650776105
> > > > 14/02/17 10:30:30 INFO file.LogFileV3: Updating log-7.meta currentPosition = 321308, logWriteOrderID = 1392650776127
> > > > 14/02/17 10:30:30 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 299362 logWriteOrderID: 1392650776105
> > > > 14/02/17 10:30:30 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 321308 logWriteOrderID: 1392650776127
> > > > 14/02/17 10:30:31 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 21
> > > > 14/02/17 10:30:32 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 38
> > > > 14/02/17 10:30:34 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776192, queueSize: 569, queueHead: 223691
> > > > 14/02/17 10:30:34 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776193, queueSize: 20, queueHead: 11070
> > > > 14/02/17 10:30:34 INFO file.LogFileV3: Updating log-8.meta currentPosition = 310040, logWriteOrderID = 1392650776192
> > > > 14/02/17 10:30:34 INFO file.LogFileV3: Updating log-7.meta currentPosition = 332801, logWriteOrderID = 1392650776193
> > > > 14/02/17 10:30:34 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 310040 logWriteOrderID: 1392650776192
> > > > 14/02/17 10:30:35 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 332801 logWriteOrderID: 1392650776193
> > > > 14/02/17 10:30:37 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 20
> > > > 14/02/17 10:30:39 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 20
> > > > 14/02/17 10:30:39 INFO hdfs.BucketWriter: Renaming hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998182.tmp to hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998182
> > > > 14/02/17 10:30:40 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776236, queueSize: 0, queueHead: 11090
> > > > 14/02/17 10:30:40 INFO hdfs.BucketWriter: Creating hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998183.tmp
> > > > 14/02/17 10:30:42 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776237, queueSize: 589, queueHead: 223691
> > > > 14/02/17 10:30:58 INFO file.LogFileV3: Updating log-7.meta currentPosition = 333657, logWriteOrderID = 1392650776236
> > > > 14/02/17 10:30:59 INFO file.LogFileV3: Updating log-8.meta currentPosition = 320738, logWriteOrderID = 1392650776237
> > > > 14/02/17 10:31:01 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 333657 logWriteOrderID: 1392650776236
> > > > 14/02/17 10:31:03 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 320738 logWriteOrderID: 1392650776237
> > > > 14/02/17 10:31:04 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 125
> > > > 14/02/17 10:31:05 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 20
> > > > 14/02/17 10:31:07 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776384, queueSize: 464, queueHead: 223816
> > > > 14/02/17 10:31:07 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776385, queueSize: 20, queueHead: 11088
> > > > 14/02/17 10:31:19 INFO file.LogFileV3: Updating log-7.meta currentPosition = 344355, logWriteOrderID = 1392650776385
> > > > 14/02/17 10:31:19 INFO file.LogFileV3: Updating log-8.meta currentPosition = 325863, logWriteOrderID = 1392650776384
> > > > 14/02/17 10:31:20 INFO hdfs.BucketWriter: Renaming hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998183.tmp to hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998183
> > > > 14/02/17 10:31:22 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 325863 logWriteOrderID: 1392650776384
> > > > 14/02/17 10:31:22 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 344355 logWriteOrderID: 1392650776385
> > > > 14/02/17 10:31:23 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 20
> > > > 14/02/17 10:31:23 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 1
> > > > 14/02/17 10:31:23 INFO hdfs.BucketWriter: Creating hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998184.tmp
> > > > 14/02/17 10:31:24 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776427, queueSize: 0, queueHead: 11108
> > > > 14/02/17 10:31:24 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776428, queueSize: 463, queueHead: 223817
> > > > 14/02/17 10:31:25 INFO file.LogFileV3: Updating log-8.meta currentPosition = 335946, logWriteOrderID = 1392650776428
> > > > 14/02/17 10:31:26 INFO file.LogFileV3: Updating log-7.meta currentPosition = 345211, logWriteOrderID = 1392650776427
> > > > 14/02/17 10:31:26 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 345211 logWriteOrderID: 1392650776427
> > > > 14/02/17 10:31:26 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 335946 logWriteOrderID: 1392650776428
> > > > 14/02/17 10:31:27 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 40
> > > > 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 70
> > > > 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776540, queueSize: 473, queueHead: 223847
> > > > 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650776541, queueSize: 40, queueHead: 11106
> > > > 14/02/17 10:31:28 INFO file.LogFileV3: Updating log-8.meta currentPosition = 356818, logWriteOrderID = 1392650776540
> > > > 14/02/17 10:31:28 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 356818 logWriteOrderID: 1392650776540
> > > > 14/02/17 10:31:28 INFO file.LogFileV3: Updating log-7.meta currentPosition = 366536, logWriteOrderID = 1392650776541
> > > > 14/02/17 10:31:30 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 366536 logWriteOrderID: 1392650776541
> > > > 14/02/17 10:31:31 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 493
> > > > 14/02/17 10:31:32 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 40
> > > > 14/02/17 10:31:34 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650777082, queueSize: 0, queueHead: 11146
> > > > 14/02/17 10:31:35 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650777083, queueSize: 0, queueHead: 224340
> > > > 14/02/17 10:31:38 INFO file.LogFileV3: Updating log-7.meta currentPosition = 368733, logWriteOrderID = 1392650777082
> > > > 14/02/17 10:31:38 INFO file.LogFileV3: Updating log-8.meta currentPosition = 379163, logWriteOrderID = 1392650777083
> > > > 14/02/17 10:31:38 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 368733 logWriteOrderID: 1392650777082
> > > > 14/02/17 10:31:38 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 379163 logWriteOrderID: 1392650777083
> > > > 14/02/17 10:31:39 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 920
> > > > 14/02/17 10:31:39 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 900
> > > > 14/02/17 10:31:40 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650778995, queueSize: 900, queueHead: 224338
> > > > 14/02/17 10:31:40 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650778996, queueSize: 920, queueHead: 11144
> > > > 14/02/17 10:31:49 INFO file.LogFileV3: Updating log-7.meta currentPosition = 859009, logWriteOrderID = 1392650778996
> > > > 14/02/17 10:31:49 INFO file.LogFileV3: Updating log-8.meta currentPosition = 859505, logWriteOrderID = 1392650778995
> > > > 14/02/17 10:31:49 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 859009 logWriteOrderID: 1392650778996
> > > > 14/02/17 10:31:50 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 920
> > > > 14/02/17 10:31:53 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 859505 logWriteOrderID: 1392650778995
> > > > 14/02/17 10:31:53 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650779929, queueSize: 0, queueHead: 12064
> > > > 14/02/17 10:31:54 INFO hdfs.BucketWriter: Renaming hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998184.tmp to hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998184
> > > > 14/02/17 10:31:54 INFO hdfs.BucketWriter: Creating hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998185.tmp
> > > > 14/02/17 10:31:54 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 22
> > > > 14/02/17 10:31:55 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650779951, queueSize: 918, queueHead: 224340
> > > > 14/02/17 10:31:56 INFO file.LogFileV3: Updating log-7.meta currentPosition = 897089, logWriteOrderID = 1392650779929
> > > > 14/02/17 10:31:56 INFO file.LogFileV3: Updating log-8.meta currentPosition = 870220, logWriteOrderID = 1392650779951
> > > > 14/02/17 10:31:56 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 870220 logWriteOrderID: 1392650779951
> > > > 14/02/17 10:31:56 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 897089 logWriteOrderID: 1392650779929
> > > > 14/02/17 10:31:57 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 300
> > > > 14/02/17 10:32:00 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650781760, queueSize: 300, queueHead: 12062
> > > > 14/02/17 10:32:00 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 1198
> > > > 14/02/17 10:32:01 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650781761, queueSize: 0, queueHead: 225538
> > > > 14/02/17 10:32:02 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1057180, logWriteOrderID = 1392650781760
> > > > 14/02/17 10:32:03 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1057180 logWriteOrderID: 1392650781760
> > > > 14/02/17 10:32:03 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1068832, logWriteOrderID = 1392650781761
> > > > 14/02/17 10:32:03 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1068832 logWriteOrderID: 1392650781761
> > > > 14/02/17 10:32:04 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 798
> > > > 14/02/17 10:32:07 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650783137, queueSize: 500, queueHead: 12360
> > > > 14/02/17 10:32:07 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 520
> > > > 14/02/17 10:32:08 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650783138, queueSize: 519, queueHead: 225537
> > > > 14/02/17 10:32:12 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1336479, logWriteOrderID = 1392650783137
> > > > 14/02/17 10:32:14 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1346456, logWriteOrderID = 1392650783138
> > > > 14/02/17 10:32:14 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1336479 logWriteOrderID: 1392650783137
> > > > 14/02/17 10:32:15 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 100
> > > > 14/02/17 10:32:16 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1346456 logWriteOrderID: 1392650783138
> > > > 14/02/17 10:32:17 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650783761, queueSize: 400, queueHead: 12460
> > > > 14/02/17 10:32:17 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 519
> > > > 14/02/17 10:32:20 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650783762, queueSize: 0, queueHead: 226056
> > > > 14/02/17 10:32:21 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1341143, logWriteOrderID = 1392650783761
> > > > 14/02/17 10:32:23 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1367771, logWriteOrderID = 1392650783762
> > > > 14/02/17 10:32:23 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1341143 logWriteOrderID: 1392650783761
> > > > 14/02/17 10:32:24 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1367771 logWriteOrderID: 1392650783762
> > > > 14/02/17 10:32:24 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 300
> > > > 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 100
> > > > 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650784174, queueSize: 300, queueHead: 12660
> > > > 14/02/17 10:32:25 INFO hdfs.BucketWriter: Renaming hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998185.tmp to hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998185
> > > > 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650784175, queueSize: 100, queueHead: 226054
> > > > 14/02/17 10:32:25 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1402287, logWriteOrderID = 1392650784174
> > > > 14/02/17 10:32:26 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1402287 logWriteOrderID: 1392650784174
> > > > 14/02/17 10:32:26 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1421128, logWriteOrderID = 1392650784175
> > > > 14/02/17 10:32:26 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1421128 logWriteOrderID: 1392650784175
> > > > 14/02/17 10:32:27 INFO hdfs.BucketWriter: Creating hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> > > > 14/02/17 10:32:27 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 480
> > > > 14/02/17 10:32:28 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 278
> > > > 14/02/17 10:32:28 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650785222, queueSize: 98, queueHead: 13042
> > > > 14/02/17 10:32:32 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650785223, queueSize: 0, queueHead: 226332
> > > > 14/02/17 10:32:33 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1514767, logWriteOrderID = 1392650785222
> > > > 14/02/17 10:32:34 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1514767 logWriteOrderID: 1392650785222
> > > > 14/02/17 10:32:35 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 118
> > > > 14/02/17 10:32:38 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1528845, logWriteOrderID = 1392650785223
> > > > 14/02/17 10:32:38 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650785364, queueSize: 0, queueHead: 13160
> > > > 14/02/17 10:32:40 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1528845 logWriteOrderID: 1392650785223
> > > > 14/02/17 10:32:41 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1529781, logWriteOrderID = 1392650785364
> > > > 14/02/17 10:32:42 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1529781 logWriteOrderID: 1392650785364
> > > > 14/02/17 10:32:43 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to sync = 500
> > > > 14/02/17 10:32:44 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 500
> > > > 14/02/17 10:32:45 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650786415, queueSize: 500, queueHead: 13158
> > > > 14/02/17 10:32:47 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650786416, queueSize: 500, queueHead: 226330
> > > > 14/02/17 10:32:53 INFO node.FlumeNode: Flume node stopping - agent
> > > > 14/02/17 10:32:53 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
> > > > 14/02/17 10:32:53 INFO properties.PropertiesFileConfigurationProvider: Configuration provider stopping
> > > > 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Node manager stopping
> > > > 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Shutting down configuration: { sourceRunners:{exec-source=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec-source,state:START} }} sinkRunners:{hbase-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{ name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=59} } }, hdfs-sink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{ name:null counters:{runner.backoffs.consecutive=3, runner.backoffs=53} } }} channels:{ch1=FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }, ch2=FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }} }
> > > > 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Stopping Source exec-source
> > > > 14/02/17 10:32:53 INFO lifecycle.LifecycleSupervisor: Stopping component: EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec-source,state:START} }
> > > > 14/02/17 10:32:53 INFO source.ExecSource: Stopping exec source with command:tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> > > > 14/02/17 10:32:54 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1795949, logWriteOrderID = 1392650786416
> > > > 14/02/17 10:32:54 INFO file.LogFileV3: Updating log-7.meta currentPosition = 1796885, logWriteOrderID = 1392650786415
> > > > 14/02/17 10:32:57 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1795949 logWriteOrderID: 1392650786416
> > > > 14/02/17 10:32:57 ERROR source.ExecSource: Failed while running command: tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> > > > java.io.IOException: Pipe closed
> > > >         at java.io.PipedInputStream.read(PipedInputStream.java:302)
> > > >         at java.lang.ProcessPipedInputStream.read(UNIXProcess.java:412)
> > > >         at java.io.PipedInputStream.read(PipedInputStream.java:372)
> > > >         at java.lang.ProcessInputStream.read(UNIXProcess.java:471)
> > > >         at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
> > > >         at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
> > > >         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
> > > >         at java.io.InputStreamReader.read(InputStreamReader.java:188)
> > > >         at java.io.BufferedReader.fill(BufferedReader.java:147)
> > > >         at java.io.BufferedReader.readLine(BufferedReader.java:310)
> > > >         at java.io.BufferedReader.readLine(BufferedReader.java:373)
> > > >         at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:272)
> > > >         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:452)
> > > >         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> > > >         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 INFO source.ExecSource: Command [tail -F /home/biadmin/bigdemo/data/rec_telco.cdr] exited with 130
> > > > 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping Sink hbase-sink
> > > > 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{ name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=59} } }
> > > > 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Component has already been stopped EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:exec-source,state:STOP} }
> > > > 14/02/17 10:32:58 WARN file.Log: Interrupted while waiting for log shared lock
> > > > java.lang.InterruptedException
> > > >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1035)
> > > >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1314)
> > > >         at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:839)
> > > >         at org.apache.flume.channel.file.Log.tryLockShared(Log.java:599)
> > > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:446)
> > > >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > >         at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:190)
> > > >         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
> > > > org.apache.flume.ChannelException: Failed to obtain lock for writing to the log. Try increasing the log write timeout value. [channel=ch2]
> > > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
> > > >         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
> > > >         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
> > > >         at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:190)
> > > >         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 INFO client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x144401355b4001d
> > > > 14/02/17 10:32:58 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel2/data/log-7 position: 1796885 logWriteOrderID: 1392650786415
> > > > 14/02/17 10:32:57 WARN hdfs.BucketWriter: Caught IOException writing to HDFSWriter (Filesystem closed). Closing file (hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp) and rethrowing exception.
> > > > 14/02/17 10:32:58 WARN hdfs.BucketWriter: Caught IOException while closing file (hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp). Exception follows.
> > > > java.io.IOException: Filesystem closed
> > > >         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
> > > >         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
> > > >         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
> > > >         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:382)
> > > >         at org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
> > > >         at org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
> > > >         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> > > >         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 INFO file.EventQueueBackingStoreFile: Start checkpoint for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to sync = 1
> > > > 14/02/17 10:32:58 INFO hdfs.BucketWriter: HDFSWriter is already closed: hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> > > > 14/02/17 10:32:58 ERROR hdfs.BucketWriter: Unexpected error
> > > > java.io.IOException: Filesystem closed
> > > >         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
> > > >         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
> > > >         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
> > > >         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:237)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:232)
> > > >         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> > > >         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> > > >         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:109)
> > > >         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:217)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 INFO file.EventQueueBackingStoreFile: Updating checkpoint metadata: logWriteOrderID: 1392650786418, queueSize: 499, queueHead: 226331
> > > > 14/02/17 10:32:58 INFO zookeeper.ZooKeeper: Session: 0x144401355b4001d closed
> > > > 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping Sink hdfs-sink
> > > > 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component: SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{ name:null counters:{runner.backoffs.consecutive=3, runner.backoffs=53} } }
> > > > 14/02/17 10:32:58 WARN file.Log: Interrupted while waiting for log shared lock
> > > > java.lang.InterruptedException
> > > >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1035)
> > > >         at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1314)
> > > >         at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:839)
> > > >         at org.apache.flume.channel.file.Log.tryLockShared(Log.java:599)
> > > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:536)
> > > >         at org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> > > >         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)
> > > >         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
> > > > org.apache.flume.ChannelException: Failed to obtain lock for writing to the log. Try increasing the log write timeout value. [channel=ch1]
> > > >         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:539)
> > > >         at org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
> > > >         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)
> > > >         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> > > >         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 INFO hdfs.HDFSEventSink: Closing hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec
> > > > 14/02/17 10:32:58 INFO zookeeper.ClientCnxn: EventThread shut down
> > > > 14/02/17 10:32:58 INFO file.LogFileV3: Updating log-8.meta currentPosition = 1795990, logWriteOrderID = 1392650786418
> > > > 14/02/17 10:32:58 INFO hdfs.BucketWriter: HDFSWriter is already closed: hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> > > > 14/02/17 10:32:58 WARN hdfs.HDFSEventSink: Exception while closing hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec. Exception follows.
> > > > java.io.IOException: Filesystem closed
> > > >         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
> > > >         at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
> > > >         at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
> > > >         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
> > > >         at org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
> > > >         at org.apache.flume.sink.hdfs.HDFSEventSink$4.call(HDFSEventSink.java:757)
> > > >         at org.apache.flume.sink.hdfs.HDFSEventSink$4.call(HDFSEventSink.java:755)
> > > >         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
> > > >         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
> > > >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
> > > >         at java.lang.Thread.run(Thread.java:738)
> > > > 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: hdfs-sink stopped
> > > > 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping Channel ch1
> > > > 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component: FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }
> > > > 14/02/17 10:32:58 INFO file.FileChannel: Stopping FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }...
> > > > 14/02/17 10:32:58 INFO file.Log: Updated checkpoint for file: /home/biadmin/.flume/file-channel/data/log-8 position: 1795990 logWriteOrderID: 1392650786418
> > > > 14/02/17 10:32:58 INFO file.LogFile: Closing /home/biadmin/.flume/file-channel/data/log-8
> > > > 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel/data/log-7
> > > > 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel/data/log-8
> > > > 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch1 stopped
> > > > 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping Channel ch2
> > > > 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component: FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }
> > > > 14/02/17 10:32:58 INFO file.FileChannel: Stopping FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }...
> > > > 14/02/17 10:32:58 INFO file.LogFile: Closing /home/biadmin/.flume/file-channel2/data/log-7
> > > > 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel2/data/log-6
> > > > 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader /home/biadmin/.flume/file-channel2/data/log-7
> > > > 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: ch2 stopped
> > > > 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 9
> > > > 
> > > > 
> > > > 
> > > > On 17 February 2014 16:38, Kris Ogirri <kanirip@gmail.com (mailto:kanirip@gmail.com)> wrote:
> > > > > Hello Jeff,
> > > > > 
> > > > > Please find below requested logs.. Initiation part of the logs were unfortunately not included. I can run these again if necessary but the Zookeeper connection is included in the logs.
> > > > > 
> > > > > 
> > > > > 
> > > > > On 17 February 2014 16:05, Jeff Lord <jlord@cloudera.com (mailto:jlord@cloudera.com)> wrote:
> > > > > > Logs ?
> > > > > > 
> > > > > > On Mon, Feb 17, 2014 at 5:51 AM, Kris Ogirri <kanirip@gmail.com (mailto:kanirip@gmail.com)> wrote:
> > > > > > > Dear Mailing Group,
> > > > > > >
> > > > > > > I am currently having issues with the Hbase sink function. I have developed
> > > > > > > an agent with a fanout channel setup ( single source, multiple channels,
> > > > > > > multiple sinks) sinking to a HDFS cluster and Hbase deployment.
> > > > > > >
> > > > > > >  The issue is that although the HDFS is working well, the Hbase flow is
> > > > > > > simply not working. There are no errors being reported by Flume for the
> > > > > > > Hbase channel but there are never any records being written to the HBase
> > > > > > > store. The Hbase table as stipulated in the config always remains empty.
> > > > > > > Studying the Flume startup logs I observe that the session connection to
> > > > > > > Zookeeper is always successfully established
> > > > > > >
> > > > > > > Are there any special configurations I am missing out?
> > > > > > >
> > > > > > > I am using the Async Event Serializer to persist the txns.
> > > > > > >
> > > > > > > Any assistance will be greatly appreciated.
> > > > > > >
> > > > > > >
> > > > > > > Please see below for the flume configuration:
> > > > > > >
> > > > > > > [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo
> > > > > > > agent.sources=exec-source
> > > > > > > agent.sinks=hdfs-sink hbase-sink
> > > > > > > agent.channels=ch1 ch2
> > > > > > >
> > > > > > > agent.sources.exec-source.type=exec
> > > > > > > agent.sources.exec-source.command=tail -F
> > > > > > > /home/biadmin/bigdemo/data/rec_telco.cdr
> > > > > > >
> > > > > > > agent.sinks.hdfs-sink.type=hdfs
> > > > > > > agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/
> > > > > > > agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec
> > > > > > > # File size to trigger roll, in bytes (0: never roll based on file size)
> > > > > > > agent.sinks.hdfs-sink.hdfs.rollSize = 134217728
> > > > > > > agent.sinks.hdfs-sink.hdfs.rollCount = 0
> > > > > > > # number of events written to file before it flushed to HDFS
> > > > > > > agent.sinks.hdfs-sink.hdfs.batchSize = 10000
> > > > > > > agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000
> > > > > > >
> > > > > > >
> > > > > > > agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
> > > > > > > agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
> > > > > > > agent.sinks.hbase-sink.table=telco_cdr_rec
> > > > > > > agent.sinks.hbase-sink.columnFamily = colfam
> > > > > > > agent.sinks.hbase-sink.channels = ch2
> > > > > > > #agent.sinks.hbase-sink.hdfs.batchSize = 10000
> > > > > > > #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000
> > > > > > >
> > > > > > >
> > > > > > > agent.channels.ch1.type=file
> > > > > > > agent.channels.ch1.checkpointInterval=3000
> > > > > > > agent.channels.ch1.transactionCapacity=10000
> > > > > > > agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint
> > > > > > > agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data
> > > > > > > agent.channels.ch1.write-timeout=30
> > > > > > > agent.channels.ch1.keep-alive=30
> > > > > > > #agent.channels.ch1.capacity=1000
> > > > > > >
> > > > > > > agent.channels.ch2.type=file
> > > > > > > agent.channels.ch2.checkpointInterval=300
> > > > > > > agent.channels.ch2.transactionCapacity=10000
> > > > > > > agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint
> > > > > > > agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data
> > > > > > > agent.channels.ch2.write-timeout=30
> > > > > > > agent.channels.ch2.keep-alive=30
> > > > > > > #agent.channels.ch2.capacity=1000
> > > > > > >
> > > > > > >
> > > > > > > agent.sources.exec-source.channels=ch1 ch2
> > > > > > > agent.sinks.hdfs-sink.channel=ch1
> > > > > > > agent.sinks.hbase-sink.channel=ch2
> > > > > > >
> > > > > 
> > > > 
> > > 
> > 
> 


Mime
View raw message