Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOFpRGDMCenRwmP9XZizcoAUkj1DsSnQepo=Rzb+hfyOU5eJ7g@mail.gmail.com>
References: <SN2PR03MB2381F644DC2159C4BD627B2090480@SN2PR03MB2381.namprd03.prod.outlook.com>
 <BN3PR03MB2179BDCB4373993D1D20C1EED2480@BN3PR03MB2179.namprd03.prod.outlook.com>
 <CAO2Ppvv4G_sMVdEYc7g3G0RwC4Xn0=eAthx0nFs=Ltkg_mShSw@mail.gmail.com>
 <CA+4sW8iF5FVczenHs0qkOYtP2iK97BFr2DT3ivo9OoPwyNOCMQ@mail.gmail.com> <CAOFpRGDMCenRwmP9XZizcoAUkj1DsSnQepo=Rzb+hfyOU5eJ7g@mail.gmail.com>
From: Andrew Chung <afchung90@gmail.com>
Date: Wed, 18 May 2016 10:18:24 -0700
Message-ID: <CAHzzxZSxaVA-xyRg2w7US=vzzvY+Qo03psXmw72=xJphJ1gdkg@mail.gmail.com>
Subject: Re: Test failure in master
To: dev@reef.apache.org
Content-Type: multipart/alternative; boundary=089e01538ac8f4e29b0533210d3d
archived-at: Wed, 18 May 2016 17:19:17 -0000

--089e01538ac8f4e29b0533210d3d
Content-Type: text/plain; charset=UTF-8

Hi,

Yes, that is the main issue.

Right now when checking for idleness, we were only checking the
`ThreadPool` queue, but we never check whether the EventHandler is still
*active* or not. Thus, if we call `Thread.sleep` within the
`FailedEvaluatorHandler`, the `EvaluatorManager` will check the
`ThreadPool`, which will report that it is idle (as the `ThreadPool` queue
is now empty), and shut down. This might be fine in Java because we didn't
have to go through the InterOp layer and the EventHandler always finishes
before an idleness check, but this shows when going through the C# code.

There are trickier issues going on, namely when the following events occur:
1. We call close on an Evaluator, in *any* part of the code.
2. close triggers an idleness check, but with fix to REEF-1393, the
EventHandler is still active.
3. No more calls to check idleness for a while.
4. We don't exit before the Test times out.

I'm still currently running tests and checking different scenarios, but a
fix (albeit ugly and potentially resource consuming) to run a Thread at the
end of `close` that repeatedly checks that all Evaluator messages are
handled before triggering an idleness check *should* work. Please let me
know if a better fix exists.

Thanks,
Andrew

On Tue, May 17, 2016 at 5:56 PM, Markus Weimer <markus@weimo.de> wrote:

> Andrew, is REEF-1393 the root cause / fix for this?
>
> Markus
>

--089e01538ac8f4e29b0533210d3d--