flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Li <litao.bupt...@gmail.com>
Subject Re: [Transaction] About KafkaSource and HDFSEventSource Transaction GGuarantee
Date Tue, 14 Apr 2015 16:17:12 GMT
Hi all:

I have a question about "Transaction". For example, KafkaSource code like
this:

try {
    getChannelProcessor().processEventBatch(eventList);
    consumer.commitOffsets();
    return Status.READY
} catch(Exception e) {
    return Status.BACKOFF;
}

If processEventBatch() succeed, but commitOffsets() failed, will return
BACKOFF. But the eventList is already  write to channel.

----------------------------------

Also for HDFSEventSink code like this:

try {
    bucketWriter.append(event);
    bucketWriter.flush();
    transaction.commit();
    return Status.READY;
} catch(Exception e) {
    transaction.rollback();
    return Status.BACKOFF;
}

If bucketWriter.flush() succeed, but transaction.commit() failed, will
transaction.rollback() and return BACKOFF. But the event is already flush
to HDFS.


2015-04-15 0:09 GMT+08:00 Tao Li <litao.buptsse@gmail.com>:

> Hi all:
>
> I have a question about "Transaction". For example, KafkaSource code like
> this:
> try {
>     getChannelProcessor().processEventBatch(eventList);
>     consumer.commitOffsets();
>
> }
>

Mime
View raw message