flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kris Ogirri <kani...@gmail.com>
Subject Re: Issue with HBase Sink in Flume ( 1.3.0)
Date Tue, 18 Feb 2014 19:23:15 GMT
Hello Hari,

I didn't know it was a holiday in the US.

Please see version information below:

Hbase:
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.94.3, rab548827f0c52211c1d67437484fcba635072767, Wed Jul 31
18:13:25 PDT 2013


Flume:
[biadmin@bivm bin]$ ./flume-ng version
Flume 1.3.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: abbccbd2ff14dd6fed2a8a3891eb51aff985e9f5
Compiled by jenkins on Wed Jun 12 19:16:33 PDT 2013
>From source with checksum dce204011600e67e1455971266d3da07


Thanks for all the assistance.

BR,



On 18 February 2014 20:14, Hari Shreedharan <hshreedharan@cloudera.com>wrote:

>  Hi Kris,
>
> Please realize that people usually work on their own time on these mailing
> lists and since your first message was sent on a Monday early morning on a
> long weekend in the US, others may not have seen your message either.
>
> Are you running Apache Flume and Apache HBase? If yes, what versions
> (output of flume-ng version and hbase version)?
>
>
> Thanks,
> Hari
>
> On Tuesday, February 18, 2014 at 10:22 AM, Kris Ogirri wrote:
>
> Hi,
>
> Cant anybody help with this? I am thinking its a small issue because
> everything seems to work fine but the data from the Channel never gets
> persisted into Hbase?
>
> I have added the description of the Hbase tables:
>
> hbase(main):005:0> describe 'telco_cdr_rec'
> DESCRIPTION
> ENABLED
>  {NAME => 'telco_cdr_rec', FAMILIES => [{NAME => 'co
> true
>  lfam', REPLICATION_SCOPE => '0',
> KEEP_DELETED_CELLS
>   => 'false', COMPRESSION => 'NONE',
> ENCODE_ON_DISK
>  => 'true', BLOCKCACHE => 'true', MIN_VERSIONS =>
> '0
>  ', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY =>
> 'fal
>  se', BLOOMFILTER => 'NONE', TTL => '2147483647',
> VE
>  RSIONS => '3', BLOCKSIZE =>
> '65536'}]}
> 1 row(s) in 0.1600 seconds
>
>
> If no one can help with the problem, can anyone provide a link to the
> Flume -> Zookeeper -> Hbase Internal documentation so I can trace where the
> error lies.
>
> Are there Zookeeper log files where I can analyse whether Flume actually
> sends the Txns to Hbase via Zookeeper?
>
>
>
> On 17 February 2014 16:38, Kris Ogirri <kanirip@gmail.com> wrote:
>
> Hello Jeff,
>
> Please find below requested logs.. Initiation part of the logs were
> unfortunately not included. I can run these again if necessary but the
> Zookeeper connection is included in the logs.
>
>
> 14/02/17 10:26:12 INFO properties.PropertiesFileConfigurationProvider:
> created channel ch2
> 14/02/17 10:26:13 INFO sink.DefaultSinkFactory: Creating instance of sink:
> hbase-sink, type: org.apache.flume.sink.hbase.HBaseSink
> 14/02/17 10:26:13 INFO sink.DefaultSinkFactory: Creating instance of sink:
> hdfs-sink, type: hdfs
> 14/02/17 10:26:14 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
> 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting new
> configuration:{ sourceRunners:{exec-source=EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:IDLE} }}
> sinkRunners:{hbase-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{
> name:null counters:{} } }, hdfs-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{
> name:null counters:{} } }} channels:{ch1=FileChannel ch1 { dataDirs:
> [/home/biadmin/.flume/file-channel/data] }, ch2=FileChannel ch2 { dataDirs:
> [/home/biadmin/.flume/file-channel2/data] }} }
> 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel ch1
> 14/02/17 10:26:14 INFO file.FileChannel: Starting FileChannel ch1 {
> dataDirs: [/home/biadmin/.flume/file-channel/data] }...
> 14/02/17 10:26:14 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel ch2
> 14/02/17 10:26:14 INFO file.FileChannel: Starting FileChannel ch2 {
> dataDirs: [/home/biadmin/.flume/file-channel2/data] }...
> 14/02/17 10:26:14 INFO file.Log: Encryption is not enabled
> 14/02/17 10:26:14 INFO file.Log: Replay started
> 14/02/17 10:26:14 INFO file.Log: Encryption is not enabled
> 14/02/17 10:26:14 INFO file.Log: Replay started
> 14/02/17 10:26:14 INFO file.Log: Found NextFileID 7, from
> [/home/biadmin/.flume/file-channel/data/log-7,
> /home/biadmin/.flume/file-channel/data/log-6]
> 14/02/17 10:26:14 INFO file.Log: Found NextFileID 6, from
> [/home/biadmin/.flume/file-channel2/data/log-6,
> /home/biadmin/.flume/file-channel2/data/log-4,
> /home/biadmin/.flume/file-channel2/data/log-5]
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Starting up with
> /home/biadmin/.flume/file-channel2/checkpoint/checkpoint and
> /home/biadmin/.flume/file-channel2/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Reading
> checkpoint metadata from
> /home/biadmin/.flume/file-channel2/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Starting up with
> /home/biadmin/.flume/file-channel/checkpoint/checkpoint and
> /home/biadmin/.flume/file-channel/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.EventQueueBackingStoreFileV3: Reading
> checkpoint metadata from
> /home/biadmin/.flume/file-channel/checkpoint/checkpoint.meta
> 14/02/17 10:26:14 INFO file.Log: Last Checkpoint Mon Feb 17 10:21:35 EST
> 2014, queue depth = 0
> 14/02/17 10:26:14 INFO file.Log: Last Checkpoint Mon Feb 17 10:21:31 EST
> 2014, queue depth = 0
> 14/02/17 10:26:14 INFO file.Log: Replaying logs with v2 replay logic
> 14/02/17 10:26:14 INFO file.Log: Replaying logs with v2 replay logic
> 14/02/17 10:26:14 INFO file.ReplayHandler: Starting replay of
> [/home/biadmin/.flume/file-channel/data/log-6,
> /home/biadmin/.flume/file-channel/data/log-7]
> 14/02/17 10:26:14 INFO file.ReplayHandler: Starting replay of
> [/home/biadmin/.flume/file-channel2/data/log-4,
> /home/biadmin/.flume/file-channel2/data/log-5,
> /home/biadmin/.flume/file-channel2/data/log-6]
> 14/02/17 10:26:14 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel/data/log-6
> 14/02/17 10:26:14 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel2/data/log-4
> 14/02/17 10:26:14 INFO tools.DirectMemoryUtils: Unable to get
> maxDirectMemory from VM: NoSuchMethodException:
> sun.misc.VM.maxDirectMemory(null)
> 14/02/17 10:26:14 INFO tools.DirectMemoryUtils: Direct Memory Allocation:
> Allocation = 1048576, Allocated = 0, MaxDirectMemorySize = 20971520,
> Remaining = 20971520
> 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position:
> 32040
> 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel/data/log-7
> 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position:
> 2496
> 14/02/17 10:26:16 WARN file.LogFile: Checkpoint for
> file(/home/biadmin/.flume/file-channel2/data/log-4) is: 1392407375821,
> which is beyond the requested checkpoint time: 1392650490155 and position 0
> 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel2/data/log-5
> 14/02/17 10:26:16 INFO file.LogFile: fast-forward to checkpoint position:
> 22843
> 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 22843 in
> /home/biadmin/.flume/file-channel2/data/log-5
> 14/02/17 10:26:16 INFO file.ReplayHandler: Replaying
> /home/biadmin/.flume/file-channel2/data/log-6
> 14/02/17 10:26:16 WARN file.LogFile: Checkpoint for
> file(/home/biadmin/.flume/file-channel2/data/log-6) is: 1392650490155,
> which is beyond the requested checkpoint time: 1392650490155 and position 0
> 14/02/17 10:26:16 INFO file.ReplayHandler: read: 0, put: 0, take: 0,
> rollback: 0, commit: 0, skip: 0, eventCount:0
> 14/02/17 10:26:16 INFO file.Log: Rolling
> /home/biadmin/.flume/file-channel2/data
> 14/02/17 10:26:16 INFO file.Log: Roll start
> /home/biadmin/.flume/file-channel2/data
> 14/02/17 10:26:16 INFO file.LogFile: Opened
> /home/biadmin/.flume/file-channel2/data/log-7
> 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 2496 in
> /home/biadmin/.flume/file-channel/data/log-7
> 14/02/17 10:26:16 INFO file.LogFile: Encountered EOF at 32071 in
> /home/biadmin/.flume/file-channel/data/log-6
> 14/02/17 10:26:16 INFO file.ReplayHandler: read: 1, put: 0, take: 0,
> rollback: 0, commit: 0, skip: 1, eventCount:0
> 14/02/17 10:26:16 INFO file.Log: Rolling
> /home/biadmin/.flume/file-channel/data
> 14/02/17 10:26:16 INFO file.Log: Roll start
> /home/biadmin/.flume/file-channel/data
> 14/02/17 10:26:16 INFO file.LogFile: Opened
> /home/biadmin/.flume/file-channel/data/log-8
> 14/02/17 10:26:16 INFO file.Log: Roll end
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 0
> 14/02/17 10:26:16 INFO file.Log: Roll end
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 0
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650774387, queueSize: 0,
> queueHead: 10516
> 14/02/17 10:26:16 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650774388, queueSize: 0,
> queueHead: 223682
> 14/02/17 10:26:16 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 0, logWriteOrderID = 1392650774387
> 14/02/17 10:26:16 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 0, logWriteOrderID = 1392650774388
> 14/02/17 10:26:16 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 0 logWriteOrderID:
> 1392650774387
> 14/02/17 10:26:16 INFO file.FileChannel: Queue Size after replay: 0
> [channel=ch2]
> 14/02/17 10:26:17 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 0 logWriteOrderID:
> 1392650774388
> 14/02/17 10:26:17 INFO file.FileChannel: Queue Size after replay: 0
> [channel=ch1]
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried
> counter group for type: CHANNEL, name: ch2, registered successfully.
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch2 started
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried
> counter group for type: CHANNEL, name: ch1, registered successfully.
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch1 started
> 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Sink hbase-sink
> 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Sink hdfs-sink
> 14/02/17 10:26:17 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Source exec-source
> 14/02/17 10:26:17 INFO source.ExecSource: Exec source starting with
> command:tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Monitoried
> counter group for type: SINK, name: hdfs-sink, registered successfully.
> 14/02/17 10:26:17 INFO instrumentation.MonitoredCounterGroup: Component
> type: SINK, name: hdfs-sink started
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5--1, built on 01/23/2013 14:29 GMT
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:host.name
> =bivm
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=IBM Corporation
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/opt/ibm/biginsights/jdk/jre
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=conf:/opt/ibm/biginsights/flume/lib/snappy-java-1.0.4.1.jar:/opt/ibm/biginsights/flume/lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/flume/lib/jackson-mapper-asl-1.9.3.jar:/opt/ibm/biginsights/flume/lib/flume-avro-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-jdbc-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/velocity-1.7.jar:/opt/ibm/biginsights/flume/lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/flume/lib/flume-ng-node-1.3.0.jar:/opt/ibm/biginsights/flume/lib/commons-dbcp-1.4.jar:/opt/ibm/biginsights/flume/lib/log4j-1.2.16.jar:/opt/ibm/biginsights/flume/lib/flume-hdfs-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/asynchbase-1.2.0.jar:/opt/ibm/biginsights/flume/lib/flume-recoverable-memory-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/async-1.3.1.jar:/opt/ibm/biginsights/flume/lib/slf4j-log4j12-1.6.1.jar:/opt/ibm/biginsights/flume/lib/flume-thrift-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-file-channel-1.3.0.jar:/opt/ibm/biginsights/flume/lib/libthrift-0.6.1.jar:/opt/ibm/biginsights/flume/lib/avro-1.7.2.jar:/opt/ibm/biginsights/flume/lib/jetty-6.1.26.jar:/opt/ibm/biginsights/flume/lib/jackson-core-asl-1.9.3.jar:/opt/ibm/biginsights/flume/lib/servlet-api-2.5-20110124.jar:/opt/ibm/biginsights/flume/lib/flume-ng-elasticsearch-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-ng-configuration-1.3.0.jar:/opt/ibm/biginsights/flume/lib/jsr305-1.3.9.jar:/opt/ibm/biginsights/flume/lib/irclib-1.10.jar:/opt/ibm/biginsights/flume/lib/commons-cli-1.2.jar:/opt/ibm/biginsights/flume/lib/derby-10.8.3.1.jar:/opt/ibm/biginsights/flume/lib/flume-ng-log4jappender-1.3.0.jar:/opt/ibm/biginsights/flume/lib/netty-3.4.0.Final.jar:/opt/ibm/biginsights/flume/lib/flume-irc-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/jcl-over-slf4j-1.7.2.jar:/opt/ibm/biginsights/flume/lib/slf4j-api-1.6.1.jar:/opt/ibm/biginsights/flume/lib/joda-time-2.1.jar:/opt/ibm/biginsights/flume/lib/commons-lang-2.5.jar:/opt/ibm/biginsights/flume/lib/commons-io-2.1.jar:/opt/ibm/biginsights/flume/lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/flume/lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/flume/lib/commons-pool-1.5.4.jar:/opt/ibm/biginsights/flume/lib/flume-ng-hbase-sink-1.3.0.jar:/opt/ibm/biginsights/flume/lib/protobuf-java-2.4.1.jar:/opt/ibm/biginsights/flume/lib/flume-scribe-source-1.3.0.jar:/opt/ibm/biginsights/flume/lib/flume-ng-core-1.3.0.jar:/opt/ibm/biginsights/flume/lib/gson-2.2.2.jar:/opt/ibm/biginsights/flume/lib/flume-ng-sdk-1.3.0.jar:/opt/ibm/biginsights/flume/lib/avro-ipc-1.7.2.jar:/opt/ibm/biginsights/flume/lib/guava-10.0.1.jar:/opt/ibm/biginsights/flume/lib/paranamer-2.3.jar:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/IHC/libexec/..:/opt/ibm/biginsights/IHC/libexec/../hadoop-core-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/adaptive-mr.jar:/opt/ibm/biginsights/IHC/libexec/../lib/asm-3.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjrt-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjtools-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/biginsights-sftpfs-1.0.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-cli-1.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-codec-1.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-daemon-1.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-digester-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-el-1.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-io-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-lang-2.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-api-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-math-2.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-net-3.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/core-3.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftplet-api-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftpserver-core-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/guardium-proxy.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ibm-compression.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jdeb-0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-core-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-json-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-server-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jets3t-0.6.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.42.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.43.jar:/opt/ibm/biginsights/IHC/libexec/../lib/junit-4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/log4j-1.2.16.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mockito-all-1.8.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/oro-2.0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/IHC/libexec/../lib/workflowScheduler.jar:/opt/ibm/biginsights/IHC/libexec/../lib/xmlenc-0.52.jar:/opt/ibm/biginsights/IHC/libexec/../lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/IHC/:/opt/ibm/biginsights/IHC/:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/IHC/lib/biginsights-gpfs-1.1.1.jar:/opt/ibm/biginsights/IHC/hadoop-core.jar:/opt/ibm/biginsights/IHC/lib/biginsights-gpfs-1.1.1.jar:/opt/ibm/biginsights/IHC/hadoop-core.jar:/home/biadmin/twitter4j/lib/twitter4j-media-support-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-core-3.0.3.jar:home/biadmin/twitter4j/lib/twitter4j-async-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-stream-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-media-support-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-core-3.0.3.jar:home/biadmin/twitter4j/lib/twitter4j-async-3.0.3.jar:/home/biadmin/twitter4j/lib/twitter4j-stream-3.0.3.jar:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/hbase:/opt/ibm/biginsights/hbase/hbase-0.94.3-security.jar:/opt/ibm/biginsights/hbase/hbase-0.94.3-security-tests.jar:/opt/ibm/biginsights/hbase/hbase.jar:/opt/ibm/biginsights/hbase/lib/activation-1.1.jar:/opt/ibm/biginsights/hbase/lib/asm-3.1.jar:/opt/ibm/biginsights/hbase/lib/avro-1.7.2.jar:/opt/ibm/biginsights/hbase/lib/avro-ipc-1.7.2.jar:/opt/ibm/biginsights/hbase/lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/hbase/lib/commons-cli-1.2.jar:/opt/ibm/biginsights/hbase/lib/commons-codec-1.4.jar:/opt/ibm/biginsights/hbase/lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/hbase/lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/hbase/lib/commons-digester-1.8.jar:/opt/ibm/biginsights/hbase/lib/commons-el-1.0.jar:/opt/ibm/biginsights/hbase/lib/commons-httpclient-3.1.jar:/opt/ibm/biginsights/hbase/lib/commons-io-2.1.jar:/opt/ibm/biginsights/hbase/lib/commons-lang-2.5.jar:/opt/ibm/biginsights/hbase/lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/commons-math-2.2.jar:/opt/ibm/biginsights/hbase/lib/commons-net-3.1.jar:/opt/ibm/biginsights/hbase/lib/core-3.1.1.jar:/opt/ibm/biginsights/hbase/lib/guardium-proxy.jar:/opt/ibm/biginsights/hbase/lib/guava-11.0.2.jar:/opt/ibm/biginsights/hbase/lib/hadoop-core.jar:/opt/ibm/biginsights/hbase/lib/hadoop-tools-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/high-scale-lib-1.1.1.jar:/opt/ibm/biginsights/hbase/lib/httpclient-4.1.2.jar:/opt/ibm/biginsights/hbase/lib/httpcore-4.1.3.jar:/opt/ibm/biginsights/hbase/lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-jaxrs-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jackson-xc-1.8.8.jar:/opt/ibm/biginsights/hbase/lib/jamon-runtime-2.3.1.jar:/opt/ibm/biginsights/hbase/lib/jasper-compiler-5.5.23.jar:/opt/ibm/biginsights/hbase/lib/jasper-runtime-5.5.23.jar:/opt/ibm/biginsights/hbase/lib/jaxb-api-2.1.jar:/opt/ibm/biginsights/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/ibm/biginsights/hbase/lib/jersey-core-1.8.jar:/opt/ibm/biginsights/hbase/lib/jersey-json-1.8.jar:/opt/ibm/biginsights/hbase/lib/jersey-server-1.8.jar:/opt/ibm/biginsights/hbase/lib/jettison-1.1.jar:/opt/ibm/biginsights/hbase/lib/jetty-6.1.26.jar:/opt/ibm/biginsights/hbase/lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/hbase/lib/jruby-complete-1.6.5.1.jar:/opt/ibm/biginsights/hbase/lib/jsp-2.1-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/lib/jsr305-1.3.9.jar:/opt/ibm/biginsights/hbase/lib/junit-4.10-HBASE-1.jar:/opt/ibm/biginsights/hbase/lib/libthrift-0.8.0.jar:/opt/ibm/biginsights/hbase/lib/log4j-1.2.16.jar:/opt/ibm/biginsights/hbase/lib/metrics-core-2.1.2.jar:/opt/ibm/biginsights/hbase/lib/netty-3.2.4.Final.jar:/opt/ibm/biginsights/hbase/lib/netty-3.4.0.Final.jar:/opt/ibm/biginsights/hbase/lib/protobuf-java-2.4.0a.jar:/opt/ibm/biginsights/hbase/lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/hbase/lib/servlet-api-2.5-6.1.14.jar:/opt/ibm/biginsights/hbase/lib/snappy-java-1.0.4.1.jar:/opt/ibm/biginsights/hbase/lib/stax-api-1.0.1.jar:/opt/ibm/biginsights/hbase/lib/velocity-1.7.jar:/opt/ibm/biginsights/hbase/lib/xmlenc-0.52.jar:/opt/ibm/biginsights/hbase/lib/xml-ibm.jar:/opt/ibm/biginsights/hbase/lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/hbase/lib/zookeeper.jar:/opt/ibm/biginsights/hadoop-conf:/opt/ibm/biginsights/jdk/lib/tools.jar:/opt/ibm/biginsights/IHC/libexec/..:/opt/ibm/biginsights/IHC/libexec/../hadoop-core-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/adaptive-mr.jar:/opt/ibm/biginsights/IHC/libexec/../lib/asm-3.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjrt-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/aspectjtools-1.6.11.jar:/opt/ibm/biginsights/IHC/libexec/../lib/biginsights-sftpfs-1.0.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-beanutils-1.8.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-cli-1.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-codec-1.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-collections-3.2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-configuration-1.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-daemon-1.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-digester-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-el-1.0.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-io-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-lang-2.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-logging-api-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-math-2.2.jar:/opt/ibm/biginsights/IHC/libexec/../lib/commons-net-3.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/core-3.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftplet-api-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ftpserver-core-1.0.6.jar:/opt/ibm/biginsights/IHC/libexec/../lib/guardium-proxy.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-capacity-scheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-fairscheduler-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hadoop-thriftfs-1.1.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/ibm/biginsights/IHC/libexec/../lib/ibm-compression.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jdeb-0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-core-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-json-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jersey-server-1.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jets3t-0.6.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jetty-util-6.1.26.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.42.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsch-0.1.43.jar:/opt/ibm/biginsights/IHC/libexec/../lib/junit-4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/log4j-1.2.16.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mina-core-2.0.4.jar:/opt/ibm/biginsights/IHC/libexec/../lib/mockito-all-1.8.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/oro-2.0.8.jar:/opt/ibm/biginsights/IHC/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/ibm/biginsights/IHC/libexec/../lib/workflowScheduler.jar:/opt/ibm/biginsights/IHC/libexec/../lib/xmlenc-0.52.jar:/opt/ibm/biginsights/IHC/libexec/../lib/zookeeper-3.4.5.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/ibm/biginsights/IHC/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/opt/ibm/biginsights/hbase/conf:/opt/ibm/biginsights/hbase/conf
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=:/opt/ibm/biginsights/IHC/libexec/../lib/native/Linux-amd64-64:/opt/ibm/biginsights/IHC/libexec/../lib/native/Linux-amd64-64
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=j9jit24
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.18-194.17.4.el5
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client environment:user.name
> =biadmin
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/home/biadmin
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/ibm/biginsights/flume/bin
> 14/02/17 10:26:17 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=bivm:2181 sessionTimeout=180000 watcher=hconnection
> 14/02/17 10:26:17 INFO zookeeper.RecoverableZooKeeper: The identifier of
> this process is 20984@bivm
> 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Opening socket connection to
> server bivm/192.168.37.128:2181. Will not attempt to authenticate using
> SASL (Unable to locate a login configuration)
> 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Socket connection established
> to bivm/192.168.37.128:2181, initiating session
> 14/02/17 10:26:17 INFO zookeeper.ClientCnxn: Session establishment
> complete on server bivm/192.168.37.128:2181, sessionid =
> 0x144401355b4001d, negotiated timeout = 60000
> 14/02/17 10:29:56 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 60
> 14/02/17 10:29:56 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650774536, queueSize: 60,
> queueHead: 10514
> 14/02/17 10:29:56 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 32036, logWriteOrderID = 1392650774536
> 14/02/17 10:29:57 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 32036
> logWriteOrderID: 1392650774536
> 14/02/17 10:29:57 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-4
> 14/02/17 10:29:57 INFO file.Log: Removing old log
> /home/biadmin/.flume/file-channel2/data/log-4, result = true, minFileID 7
> 14/02/17 10:29:57 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-5
> 14/02/17 10:29:57 INFO file.Log: Removing old log
> /home/biadmin/.flume/file-channel2/data/log-5, result = true, minFileID 7
> 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 460
> 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650775504, queueSize: 520,
> queueHead: 10514
> 14/02/17 10:29:58 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 277565, logWriteOrderID = 1392650775504
> 14/02/17 10:29:58 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 277565
> logWriteOrderID: 1392650775504
> 14/02/17 10:29:58 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 540
> 14/02/17 10:29:59 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998182.tmp
> 14/02/17 10:29:59 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 423
> 14/02/17 10:30:00 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650775933, queueSize: 137,
> queueHead: 10917
> 14/02/17 10:30:00 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650775934, queueSize: 539,
> queueHead: 223681
> 14/02/17 10:30:01 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 304892, logWriteOrderID = 1392650775933
> 14/02/17 10:30:01 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 304892
> logWriteOrderID: 1392650775933
> 14/02/17 10:30:02 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 137
> 14/02/17 10:30:02 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 288266, logWriteOrderID = 1392650775934
> 14/02/17 10:30:02 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776074, queueSize: 0,
> queueHead: 11054
> 14/02/17 10:30:04 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 288266
> logWriteOrderID: 1392650775934
> 14/02/17 10:30:04 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel/data/log-6
> 14/02/17 10:30:04 INFO file.Log: Removing old log
> /home/biadmin/.flume/file-channel/data/log-6, result = true, minFileID 8
> 14/02/17 10:30:05 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 29
> 14/02/17 10:30:06 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 310581, logWriteOrderID = 1392650776074
> 14/02/17 10:30:13 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776105, queueSize: 550,
> queueHead: 223690
> 14/02/17 10:30:19 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 310581
> logWriteOrderID: 1392650776074
> 14/02/17 10:30:21 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:30:29 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776127, queueSize: 20,
> queueHead: 11052
> 14/02/17 10:30:29 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 299362, logWriteOrderID = 1392650776105
> 14/02/17 10:30:30 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 321308, logWriteOrderID = 1392650776127
> 14/02/17 10:30:30 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 299362
> logWriteOrderID: 1392650776105
> 14/02/17 10:30:30 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 321308
> logWriteOrderID: 1392650776127
> 14/02/17 10:30:31 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 21
> 14/02/17 10:30:32 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 38
> 14/02/17 10:30:34 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776192, queueSize: 569,
> queueHead: 223691
> 14/02/17 10:30:34 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776193, queueSize: 20,
> queueHead: 11070
> 14/02/17 10:30:34 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 310040, logWriteOrderID = 1392650776192
> 14/02/17 10:30:34 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 332801, logWriteOrderID = 1392650776193
> 14/02/17 10:30:34 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 310040
> logWriteOrderID: 1392650776192
> 14/02/17 10:30:35 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 332801
> logWriteOrderID: 1392650776193
> 14/02/17 10:30:37 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:30:39 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:30:39 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998182.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998182
> 14/02/17 10:30:40 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776236, queueSize: 0,
> queueHead: 11090
> 14/02/17 10:30:40 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998183.tmp
> 14/02/17 10:30:42 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776237, queueSize: 589,
> queueHead: 223691
> 14/02/17 10:30:58 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 333657, logWriteOrderID = 1392650776236
> 14/02/17 10:30:59 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 320738, logWriteOrderID = 1392650776237
> 14/02/17 10:31:01 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 333657
> logWriteOrderID: 1392650776236
> 14/02/17 10:31:03 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 320738
> logWriteOrderID: 1392650776237
> 14/02/17 10:31:04 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 125
> 14/02/17 10:31:05 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:31:07 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776384, queueSize: 464,
> queueHead: 223816
> 14/02/17 10:31:07 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776385, queueSize: 20,
> queueHead: 11088
> 14/02/17 10:31:19 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 344355, logWriteOrderID = 1392650776385
> 14/02/17 10:31:19 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 325863, logWriteOrderID = 1392650776384
> 14/02/17 10:31:20 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998183.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998183
> 14/02/17 10:31:22 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 325863
> logWriteOrderID: 1392650776384
> 14/02/17 10:31:22 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 344355
> logWriteOrderID: 1392650776385
> 14/02/17 10:31:23 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 20
> 14/02/17 10:31:23 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 1
> 14/02/17 10:31:23 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998184.tmp
> 14/02/17 10:31:24 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776427, queueSize: 0,
> queueHead: 11108
> 14/02/17 10:31:24 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776428, queueSize: 463,
> queueHead: 223817
> 14/02/17 10:31:25 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 335946, logWriteOrderID = 1392650776428
> 14/02/17 10:31:26 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 345211, logWriteOrderID = 1392650776427
> 14/02/17 10:31:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 345211
> logWriteOrderID: 1392650776427
> 14/02/17 10:31:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 335946
> logWriteOrderID: 1392650776428
> 14/02/17 10:31:27 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 40
> 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 70
> 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776540, queueSize: 473,
> queueHead: 223847
> 14/02/17 10:31:28 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650776541, queueSize: 40,
> queueHead: 11106
> 14/02/17 10:31:28 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 356818, logWriteOrderID = 1392650776540
> 14/02/17 10:31:28 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 356818
> logWriteOrderID: 1392650776540
> 14/02/17 10:31:28 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 366536, logWriteOrderID = 1392650776541
> 14/02/17 10:31:30 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 366536
> logWriteOrderID: 1392650776541
> 14/02/17 10:31:31 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 493
> 14/02/17 10:31:32 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 40
> 14/02/17 10:31:34 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650777082, queueSize: 0,
> queueHead: 11146
> 14/02/17 10:31:35 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650777083, queueSize: 0,
> queueHead: 224340
> 14/02/17 10:31:38 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 368733, logWriteOrderID = 1392650777082
> 14/02/17 10:31:38 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 379163, logWriteOrderID = 1392650777083
> 14/02/17 10:31:38 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 368733
> logWriteOrderID: 1392650777082
> 14/02/17 10:31:38 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 379163
> logWriteOrderID: 1392650777083
> 14/02/17 10:31:39 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 920
> 14/02/17 10:31:39 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 900
> 14/02/17 10:31:40 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650778995, queueSize: 900,
> queueHead: 224338
> 14/02/17 10:31:40 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650778996, queueSize: 920,
> queueHead: 11144
> 14/02/17 10:31:49 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 859009, logWriteOrderID = 1392650778996
> 14/02/17 10:31:49 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 859505, logWriteOrderID = 1392650778995
> 14/02/17 10:31:49 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 859009
> logWriteOrderID: 1392650778996
> 14/02/17 10:31:50 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 920
> 14/02/17 10:31:53 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 859505
> logWriteOrderID: 1392650778995
> 14/02/17 10:31:53 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650779929, queueSize: 0,
> queueHead: 12064
> 14/02/17 10:31:54 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998184.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998184
> 14/02/17 10:31:54 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998185.tmp
> 14/02/17 10:31:54 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 22
> 14/02/17 10:31:55 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650779951, queueSize: 918,
> queueHead: 224340
> 14/02/17 10:31:56 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 897089, logWriteOrderID = 1392650779929
> 14/02/17 10:31:56 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 870220, logWriteOrderID = 1392650779951
> 14/02/17 10:31:56 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 870220
> logWriteOrderID: 1392650779951
> 14/02/17 10:31:56 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 897089
> logWriteOrderID: 1392650779929
> 14/02/17 10:31:57 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 300
> 14/02/17 10:32:00 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650781760, queueSize: 300,
> queueHead: 12062
> 14/02/17 10:32:00 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 1198
> 14/02/17 10:32:01 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650781761, queueSize: 0,
> queueHead: 225538
> 14/02/17 10:32:02 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1057180, logWriteOrderID = 1392650781760
> 14/02/17 10:32:03 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1057180
> logWriteOrderID: 1392650781760
> 14/02/17 10:32:03 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1068832, logWriteOrderID = 1392650781761
> 14/02/17 10:32:03 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1068832
> logWriteOrderID: 1392650781761
> 14/02/17 10:32:04 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 798
> 14/02/17 10:32:07 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783137, queueSize: 500,
> queueHead: 12360
> 14/02/17 10:32:07 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 520
> 14/02/17 10:32:08 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783138, queueSize: 519,
> queueHead: 225537
> 14/02/17 10:32:12 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1336479, logWriteOrderID = 1392650783137
> 14/02/17 10:32:14 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1346456, logWriteOrderID = 1392650783138
> 14/02/17 10:32:14 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1336479
> logWriteOrderID: 1392650783137
> 14/02/17 10:32:15 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 100
> 14/02/17 10:32:16 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1346456
> logWriteOrderID: 1392650783138
> 14/02/17 10:32:17 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783761, queueSize: 400,
> queueHead: 12460
> 14/02/17 10:32:17 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 519
> 14/02/17 10:32:20 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650783762, queueSize: 0,
> queueHead: 226056
> 14/02/17 10:32:21 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1341143, logWriteOrderID = 1392650783761
> 14/02/17 10:32:23 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1367771, logWriteOrderID = 1392650783762
> 14/02/17 10:32:23 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1341143
> logWriteOrderID: 1392650783761
> 14/02/17 10:32:24 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1367771
> logWriteOrderID: 1392650783762
> 14/02/17 10:32:24 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 300
> 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 100
> 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650784174, queueSize: 300,
> queueHead: 12660
> 14/02/17 10:32:25 INFO hdfs.BucketWriter: Renaming
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998185.tmp to
> hdfs://bivm:9000/user/biadmin/bigdemo/telco_cdr_rec.1392650998185
> 14/02/17 10:32:25 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650784175, queueSize: 100,
> queueHead: 226054
> 14/02/17 10:32:25 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1402287, logWriteOrderID = 1392650784174
> 14/02/17 10:32:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1402287
> logWriteOrderID: 1392650784174
> 14/02/17 10:32:26 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1421128, logWriteOrderID = 1392650784175
> 14/02/17 10:32:26 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1421128
> logWriteOrderID: 1392650784175
> 14/02/17 10:32:27 INFO hdfs.BucketWriter: Creating
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> 14/02/17 10:32:27 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 480
> 14/02/17 10:32:28 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 278
> 14/02/17 10:32:28 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650785222, queueSize: 98,
> queueHead: 13042
> 14/02/17 10:32:32 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650785223, queueSize: 0,
> queueHead: 226332
> 14/02/17 10:32:33 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1514767, logWriteOrderID = 1392650785222
> 14/02/17 10:32:34 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1514767
> logWriteOrderID: 1392650785222
> 14/02/17 10:32:35 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 118
> 14/02/17 10:32:38 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1528845, logWriteOrderID = 1392650785223
> 14/02/17 10:32:38 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650785364, queueSize: 0,
> queueHead: 13160
> 14/02/17 10:32:40 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1528845
> logWriteOrderID: 1392650785223
> 14/02/17 10:32:41 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1529781, logWriteOrderID = 1392650785364
> 14/02/17 10:32:42 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1529781
> logWriteOrderID: 1392650785364
> 14/02/17 10:32:43 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel2/checkpoint/checkpoint, elements to
> sync = 500
> 14/02/17 10:32:44 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 500
> 14/02/17 10:32:45 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650786415, queueSize: 500,
> queueHead: 13158
> 14/02/17 10:32:47 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650786416, queueSize: 500,
> queueHead: 226330
> 14/02/17 10:32:53 INFO node.FlumeNode: Flume node stopping - agent
> 14/02/17 10:32:53 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle
> supervisor 9
> 14/02/17 10:32:53 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider stopping
> 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Node manager
> stopping
> 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Shutting
> down configuration: { sourceRunners:{exec-source=EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:START} }}
> sinkRunners:{hbase-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@4c004c counterGroup:{
> name:null counters:{runner.backoffs.consecutive=2, runner.backoffs=59} } },
> hdfs-sink=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01 counterGroup:{
> name:null counters:{runner.backoffs.consecutive=3, runner.backoffs=53} } }}
> channels:{ch1=FileChannel ch1 { dataDirs:
> [/home/biadmin/.flume/file-channel/data] }, ch2=FileChannel ch2 { dataDirs:
> [/home/biadmin/.flume/file-channel2/data] }} }
> 14/02/17 10:32:53 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Source exec-source
> 14/02/17 10:32:53 INFO lifecycle.LifecycleSupervisor: Stopping component:
> EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:START} }
> 14/02/17 10:32:53 INFO source.ExecSource: Stopping exec source with
> command:tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> 14/02/17 10:32:54 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1795949, logWriteOrderID = 1392650786416
> 14/02/17 10:32:54 INFO file.LogFileV3: Updating log-7.meta currentPosition
> = 1796885, logWriteOrderID = 1392650786415
> 14/02/17 10:32:57 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1795949
> logWriteOrderID: 1392650786416
> 14/02/17 10:32:57 ERROR source.ExecSource: Failed while running command:
> tail -F /home/biadmin/bigdemo/data/rec_telco.cdr
> java.io.IOException: Pipe closed
>         at java.io.PipedInputStream.read(PipedInputStream.java:302)
>         at java.lang.ProcessPipedInputStream.read(UNIXProcess.java:412)
>         at java.io.PipedInputStream.read(PipedInputStream.java:372)
>         at java.lang.ProcessInputStream.read(UNIXProcess.java:471)
>         at
> sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
>         at
> sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
>         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
>         at java.io.InputStreamReader.read(InputStreamReader.java:188)
>         at java.io.BufferedReader.fill(BufferedReader.java:147)
>         at java.io.BufferedReader.readLine(BufferedReader.java:310)
>         at java.io.BufferedReader.readLine(BufferedReader.java:373)
>         at
> org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:272)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:452)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO source.ExecSource: Command [tail -F
> /home/biadmin/bigdemo/data/rec_telco.cdr] exited with 130
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Sink hbase-sink
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4c004ccounterGroup:{ name:null counters:{runner.backoffs.consecutive=2,
> runner.backoffs=59} } }
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Component has
> already been stopped EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:exec-source,state:STOP} }
> 14/02/17 10:32:58 WARN file.Log: Interrupted while waiting for log shared
> lock
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1035)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1314)
>         at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:839)
>         at org.apache.flume.channel.file.Log.tryLockShared(Log.java:599)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:446)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:190)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.ChannelException: Failed to obtain lock for writing to
> the log. Try increasing the log write timeout value. [channel=ch2]
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at
> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at
> org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:190)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO
> client.HConnectionManager$HConnectionImplementation: Closed zookeeper
> sessionid=0x144401355b4001d
> 14/02/17 10:32:58 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel2/data/log-7 position: 1796885
> logWriteOrderID: 1392650786415
> 14/02/17 10:32:57 WARN hdfs.BucketWriter: Caught IOException writing to
> HDFSWriter (Filesystem closed). Closing file
> (hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp)
> and rethrowing exception.
> 14/02/17 10:32:58 WARN hdfs.BucketWriter: Caught IOException while closing
> file
> (hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp).
> Exception follows.
> java.io.IOException: Filesystem closed
>         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
>         at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:382)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO file.EventQueueBackingStoreFile: Start checkpoint
> for /home/biadmin/.flume/file-channel/checkpoint/checkpoint, elements to
> sync = 1
> 14/02/17 10:32:58 INFO hdfs.BucketWriter: HDFSWriter is already closed:
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> 14/02/17 10:32:58 ERROR hdfs.BucketWriter: Unexpected error
> java.io.IOException: Filesystem closed
>         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
>         at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:237)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:232)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:109)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:217)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO file.EventQueueBackingStoreFile: Updating
> checkpoint metadata: logWriteOrderID: 1392650786418, queueSize: 499,
> queueHead: 226331
> 14/02/17 10:32:58 INFO zookeeper.ZooKeeper: Session: 0x144401355b4001d
> closed
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Sink hdfs-sink
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@7b017b01counterGroup:{ name:null counters:{runner.backoffs.consecutive=3,
> runner.backoffs=53} } }
> 14/02/17 10:32:58 WARN file.Log: Interrupted while waiting for log shared
> lock
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1035)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1314)
>         at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:839)
>         at org.apache.flume.channel.file.Log.tryLockShared(Log.java:599)
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:536)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 ERROR flume.SinkRunner: Unable to deliver event.
> Exception follows.
> org.apache.flume.ChannelException: Failed to obtain lock for writing to
> the log. Try increasing the log write timeout value. [channel=ch1]
>         at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:539)
>         at
> org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO hdfs.HDFSEventSink: Closing
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec
> 14/02/17 10:32:58 INFO zookeeper.ClientCnxn: EventThread shut down
> 14/02/17 10:32:58 INFO file.LogFileV3: Updating log-8.meta currentPosition
> = 1795990, logWriteOrderID = 1392650786418
> 14/02/17 10:32:58 INFO hdfs.BucketWriter: HDFSWriter is already closed:
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec.1392650998186.tmp
> 14/02/17 10:32:58 WARN hdfs.HDFSEventSink: Exception while closing
> hdfs://bivm:9000/user/biadmin/bigdemo//telco_cdr_rec. Exception follows.
> java.io.IOException: Filesystem closed
>         at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:319)
>         at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1026)
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:524)
>         at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:426)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.doClose(BucketWriter.java:298)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.access$400(BucketWriter.java:53)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:260)
>         at
> org.apache.flume.sink.hdfs.BucketWriter$3.run(BucketWriter.java:258)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.runPrivileged(BucketWriter.java:143)
>         at
> org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:258)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$4.call(HDFSEventSink.java:757)
>         at
> org.apache.flume.sink.hdfs.HDFSEventSink$4.call(HDFSEventSink.java:755)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:149)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
>         at java.lang.Thread.run(Thread.java:738)
> 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component
> type: SINK, name: hdfs-sink stopped
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Channel ch1
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> FileChannel ch1 { dataDirs: [/home/biadmin/.flume/file-channel/data] }
> 14/02/17 10:32:58 INFO file.FileChannel: Stopping FileChannel ch1 {
> dataDirs: [/home/biadmin/.flume/file-channel/data] }...
> 14/02/17 10:32:58 INFO file.Log: Updated checkpoint for file:
> /home/biadmin/.flume/file-channel/data/log-8 position: 1795990
> logWriteOrderID: 1392650786418
> 14/02/17 10:32:58 INFO file.LogFile: Closing
> /home/biadmin/.flume/file-channel/data/log-8
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel/data/log-7
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel/data/log-8
> 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch1 stopped
> 14/02/17 10:32:58 INFO nodemanager.DefaultLogicalNodeManager: Stopping
> Channel ch2
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping component:
> FileChannel ch2 { dataDirs: [/home/biadmin/.flume/file-channel2/data] }
> 14/02/17 10:32:58 INFO file.FileChannel: Stopping FileChannel ch2 {
> dataDirs: [/home/biadmin/.flume/file-channel2/data] }...
> 14/02/17 10:32:58 INFO file.LogFile: Closing
> /home/biadmin/.flume/file-channel2/data/log-7
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-6
> 14/02/17 10:32:58 INFO file.LogFile: Closing RandomReader
> /home/biadmin/.flume/file-channel2/data/log-7
> 14/02/17 10:32:58 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: ch2 stopped
> 14/02/17 10:32:58 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle
> supervisor 9
>
>
>
> On 17 February 2014 16:38, Kris Ogirri <kanirip@gmail.com> wrote:
>
> Hello Jeff,
>
> Please find below requested logs.. Initiation part of the logs were
> unfortunately not included. I can run these again if necessary but the
> Zookeeper connection is included in the logs.
>
>
>
> On 17 February 2014 16:05, Jeff Lord <jlord@cloudera.com> wrote:
>
> Logs ?
>
> On Mon, Feb 17, 2014 at 5:51 AM, Kris Ogirri <kanirip@gmail.com> wrote:
> > Dear Mailing Group,
> >
> > I am currently having issues with the Hbase sink function. I have
> developed
> > an agent with a fanout channel setup ( single source, multiple channels,
> > multiple sinks) sinking to a HDFS cluster and Hbase deployment.
> >
> >  The issue is that although the HDFS is working well, the Hbase flow is
> > simply not working. There are no errors being reported by Flume for the
> > Hbase channel but there are never any records being written to the HBase
> > store. The Hbase table as stipulated in the config always remains empty.
> > Studying the Flume startup logs I observe that the session connection to
> > Zookeeper is always successfully established
> >
> > Are there any special configurations I am missing out?
> >
> > I am using the Async Event Serializer to persist the txns.
> >
> > Any assistance will be greatly appreciated.
> >
> >
> > Please see below for the flume configuration:
> >
> > [biadmin@bivm bin]$ cat flume-conf.properties.bigdemo
> > agent.sources=exec-source
> > agent.sinks=hdfs-sink hbase-sink
> > agent.channels=ch1 ch2
> >
> > agent.sources.exec-source.type=exec
> > agent.sources.exec-source.command=tail -F
> > /home/biadmin/bigdemo/data/rec_telco.cdr
> >
> > agent.sinks.hdfs-sink.type=hdfs
> > agent.sinks.hdfs-sink.hdfs.path=hdfs://XXXX:9000/user/biadmin/bigdemo/
> > agent.sinks.hdfs-sink.hdfs.filePrefix=telco_cdr_rec
> > # File size to trigger roll, in bytes (0: never roll based on file size)
> > agent.sinks.hdfs-sink.hdfs.rollSize = 134217728
> > agent.sinks.hdfs-sink.hdfs.rollCount = 0
> > # number of events written to file before it flushed to HDFS
> > agent.sinks.hdfs-sink.hdfs.batchSize = 10000
> > agent.sinks.hdfs-sink.hdfs.txnEventMax = 40000
> >
> >
> > agent.sinks.hbase-sink.type=org.apache.flume.sink.hbase.AsyncHBaseSink
> >
> agent.sinks.hbase-sink.serializer=org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer
> > agent.sinks.hbase-sink.table=telco_cdr_rec
> > agent.sinks.hbase-sink.columnFamily = colfam
> > agent.sinks.hbase-sink.channels = ch2
> > #agent.sinks.hbase-sink.hdfs.batchSize = 10000
> > #agent.sinks.hbase-sink.hdfs.txnEventMax = 40000
> >
> >
> > agent.channels.ch1.type=file
> > agent.channels.ch1.checkpointInterval=3000
> > agent.channels.ch1.transactionCapacity=10000
> >
> agent.channels.ch1.checkpointDir=/home/BDadmin/.flume/file-channel/checkpoint
> > agent.channels.ch1.dataDirs=/home/BDadmin/.flume/file-channel/data
> > agent.channels.ch1.write-timeout=30
> > agent.channels.ch1.keep-alive=30
> > #agent.channels.ch1.capacity=1000
> >
> > agent.channels.ch2.type=file
> > agent.channels.ch2.checkpointInterval=300
> > agent.channels.ch2.transactionCapacity=10000
> >
> agent.channels.ch2.checkpointDir=/home/BDadmin/.flume/file-channel2/checkpoint
> > agent.channels.ch2.dataDirs=/home/BDadmin/.flume/file-channel2/data
> > agent.channels.ch2.write-timeout=30
> > agent.channels.ch2.keep-alive=30
> > #agent.channels.ch2.capacity=1000
> >
> >
> > agent.sources.exec-source.channels=ch1 ch2
> > agent.sinks.hdfs-sink.channel=ch1
> > agent.sinks.hbase-sink.channel=ch2
> >
>
>
>
>
>
>

Mime
View raw message