Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAF9PaeP1gMofw_o_cBzvkyiUjAPQoftL+y3NxFnsqES-417ECQ@mail.gmail.com>
References: <CAF9PaeP1gMofw_o_cBzvkyiUjAPQoftL+y3NxFnsqES-417ECQ@mail.gmail.com>
From: Cody Koeninger <cody@koeninger.org>
Date: Wed, 7 Dec 2016 14:16:47 -0600
Message-ID: <CAKWX9VUi4Ehn4RYdGbjpxpHupBUQcWY-0CFgnZkwW-TPSw09Hw@mail.gmail.com>
Subject: Re: Reprocessing failed jobs in Streaming job
To: map reduced <k3t.git.1@gmail.com>
Cc: "user @spark" <user@spark.apache.org>
Content-Type: text/plain; charset=UTF-8
archived-at: Wed, 07 Dec 2016 20:16:54 -0000

Personally I think forcing the stream to fail (e.g. check offsets in
downstream store and throw exception if they aren't as expected) is
the safest thing to do.

If you proceed after a failure, you need a place to reliably record
the batches that failed for later processing.

On Wed, Dec 7, 2016 at 1:46 PM, map reduced <k3t.git.1@gmail.com> wrote:
> Hi,
>
> I am trying to solve this problem - in my streaming flow, every day few jobs
> fail due to some (say kafka cluster maintenance etc, mostly unavoidable)
> reasons for few batches and resumes back to success.
> I want to reprocess those failed jobs programmatically (assume I have a way
> of getting start-end offsets for kafka topics for failed jobs). I was
> thinking of these options:
> 1) Somehow pause streaming job when it detects failing jobs - this seems not
> possible.
> 2) From driver - run additional processing to check every few minutes using
> driver rest api (/api/v1/applications...) what jobs have failed and submit
> batch jobs for those failed jobs
>
> 1 - doesn't seem to be possible, and I don't want to kill streaming context
> just for few failing batches to stop the job for some time and resume after
> few minutes.
> 2 - seems like a viable option, but a little complicated, since even the
> batch job can fail due to whatever reasons and I am back to tracking that
> separately etc.
>
> Does anyone has faced this issue or have any suggestions?
>
> Thanks,
> KP

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org