Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
From: Robert B Hamilton <robert.hamilton@gm.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Flume rollback during restart possible?
Thread-Topic: Flume rollback during restart possible?
Thread-Index: AdCiPihBY5OSJHigQEKqQqAz6W8OwA==
Date: Mon, 8 Jun 2015 23:04:27 +0000
Message-ID: <6c940c200b8f475aa38e7d175bfa7995@DCWIPPEXCH006.nam.corp.gm.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: multipart/alternative;
	boundary="_000_6c940c200b8f475aa38e7d175bfa7995DCWIPPEXCH006namcorpgmc_"
MIME-Version: 1.0

--_000_6c940c200b8f475aa38e7d175bfa7995DCWIPPEXCH006namcorpgmc_
Content-Type: text/plain; charset="utf-7"
Content-Transfer-Encoding: quoted-printable

Hello all. I have an interesting case where we lose data in the event of a =
flume crash, which is easily reproducible when we kill -9  the flume agent.

 I believe that this may be because the Flume Sink is issuing a commit befo=
re it actually completes the fs sync.  If this is the case then the last fe=
w commits just before the crash would have removed events from the queue ev=
en though those events will needed to perform a recovery.  My question is, =
are those events still possibly in the WAL? If so, is it possible so someho=
w roll back the queue to a point in time before the commits were processed,=
 and restart from that state? How would I accomplish this?


Nothing in this message is intended to constitute an electronic signature u=
nless a specific statement to the contrary is included in this message.

Confidentiality Note: This message is intended only for the person or entit=
y to which it is addressed. It may contain confidential and/or privileged m=
aterial. Any review, transmission, dissemination or other use, or taking of=
 any action in reliance upon this message by persons or entities other than=
 the intended recipient is prohibited and may be unlawful. If you received =
this message in error, please contact the sender and delete it from your co=
mputer.

--_000_6c940c200b8f475aa38e7d175bfa7995DCWIPPEXCH006namcorpgmc_
Content-Type: text/html; charset="utf-7"
Content-Transfer-Encoding: quoted-printable

<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-7">
+ADw-html+AD4-
+ADw-head+AD4-
+ADw-meta name+AD0AIg-Generator+ACI- content+AD0AIg-Microsoft Word 15 (filt=
ered medium)+ACIAPg-
+ADw-style+AD4-
+ADwAIQ---
 /+ACo- Font Definitions +ACo-/
 +AEA-font-face
	+AHs-font-family:+ACI-MS Mincho+ACIAOw-
	panose-1:2 2 6 9 4 2 5 8 3 4+ADsAfQ-
+AEA-font-face
	+AHs-font-family:+ACI-Cambria Math+ACIAOw-
	panose-1:2 4 5 3 5 4 6 3 2 4+ADsAfQ-
+AEA-font-face
	+AHs-font-family:Calibri+ADs-
	panose-1:2 15 5 2 2 2 4 3 2 4+ADsAfQ-
+AEA-font-face
	+AHs-font-family:+ACIAXABA-MS Mincho+ACIAOw-
	panose-1:2 2 6 9 4 2 5 8 3 4+ADsAfQ-
 /+ACo- Style Definitions +ACo-/
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	+AHs-margin:0in+ADs-
	margin-bottom:.0001pt+ADs-
	font-size:11.0pt+ADs-
	font-family:+ACI-Calibri+ACI-,sans-serif+ADsAfQ-
a:link, span.MsoHyperlink
	+AHs-mso-style-priority:99+ADs-
	color:+ACM-0563C1+ADs-
	text-decoration:underline+ADsAfQ-
a:visited, span.MsoHyperlinkFollowed
	+AHs-mso-style-priority:99+ADs-
	color:+ACM-954F72+ADs-
	text-decoration:underline+ADsAfQ-
span.EmailStyle17
	+AHs-mso-style-type:personal-compose+ADs-
	font-family:+ACI-Calibri+ACI-,sans-serif+ADs-
	color:windowtext+ADsAfQ-
.MsoChpDefault
	+AHs-mso-style-type:export-only+ADs-
	font-family:+ACI-Calibri+ACI-,sans-serif+ADsAfQ-
+AEA-page WordSection1
	+AHs-size:8.5in 11.0in+ADs-
	margin:1.0in 1.0in 1.0in 1.0in+ADsAfQ-
div.WordSection1
	+AHs-page:WordSection1+ADsAfQ-
--+AD4-
+ADw-/style+AD4APAAh---+AFs-if gte mso 9+AF0APgA8-xml+AD4-
 +ADw-o:shapedefaults v:ext+AD0AIg-edit+ACI- spidmax+AD0AIg-1026+ACI- /+AD4=
-
+ADw-/xml+AD4APAAhAFs-endif+AF0---+AD4APAAh---+AFs-if gte mso 9+AF0APgA8-xm=
l+AD4-
 +ADw-o:shapelayout v:ext+AD0AIg-edit+ACIAPg-
  +ADw-o:idmap v:ext+AD0AIg-edit+ACI- data+AD0AIg-1+ACI- /+AD4-
 +ADw-/o:shapelayout+AD4APA-/xml+AD4APAAhAFs-endif+AF0---+AD4-
+ADw-/head+AD4-
+ADw-body lang+AD0AIg-EN-US+ACI- link+AD0AIgAj-0563C1+ACI- vlink+AD0AIgAj-9=
54F72+ACIAPg-
+ADw-div class+AD0AIg-WordSection1+ACIAPg-
+ADw-p class+AD0AIg-MsoNormal+ACIAPg-Hello all. I have an interesting case =
where we lose data in the event of a flume crash, which is easily reproduci=
ble when we kill -9 +ACY-nbsp+ADs-the flume agent.+ADw-o:p+AD4APA-/o:p+AD4A=
PA-/p+AD4-
+ADw-p class+AD0AIg-MsoNormal+ACIAPgA8-o:p+AD4AJg-nbsp+ADsAPA-/o:p+AD4APA-/=
p+AD4-
+ADw-p class+AD0AIg-MsoNormal+ACIAPgAm-nbsp+ADs-I believe that this may be =
because the Flume Sink is issuing a commit before it actually completes the=
 fs sync. +ACY-nbsp+ADs-If this is the case then the last few commits just =
before the crash would have removed events from the queue even though those
 events will needed to perform a recovery.+ACY-nbsp+ADs- My question is, ar=
e those events still possibly in the WAL? If so, is it possible so somehow =
roll back the queue to a point in time before the commits were processed, a=
nd restart from that state? How would I accomplish
 this?+ADw-o:p+AD4APA-/o:p+AD4APA-/p+AD4-
+ADw-p class+AD0AIg-MsoNormal+ACIAPgA8-o:p+AD4AJg-nbsp+ADsAPA-/o:p+AD4APA-/=
p+AD4-
+ADw-p class+AD0AIg-MsoNormal+ACIAPgA8-o:p+AD4AJg-nbsp+ADsAPA-/o:p+AD4APA-/=
p+AD4-
+ADw-p class+AD0AIg-MsoNormal+ACIAPgA8-o:p+AD4AJg-nbsp+ADsAPA-/o:p+AD4APA-/=
p+AD4-
+ADw-p class+AD0AIg-MsoNormal+ACIAPgA8-o:p+AD4AJg-nbsp+ADsAPA-/o:p+AD4APA-/=
p+AD4-
+ADw-/div+AD4-
+ADw-br+AD4-
+ADw-br+AD4-
Nothing in this message is intended to constitute an electronic signature u=
nless a specific statement to the contrary is included in this message.
+ADw-br+AD4-
+ADw-br+AD4-
Confidentiality Note: This message is intended only for the person or entit=
y to which it is addressed. It may contain confidential and/or privileged m=
aterial. Any review, transmission, dissemination or other use, or taking of=
 any action in reliance upon this
 message by persons or entities other than the intended recipient is prohib=
ited and may be unlawful. If you received this message in error, please con=
tact the sender and delete it from your computer.
+ADw-/body+AD4-
+ADw-/html+AD4-

--_000_6c940c200b8f475aa38e7d175bfa7995DCWIPPEXCH006namcorpgmc_--