Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAJZ2dcV+-7jiJ9_+ahVO-Q8qDNXCk4Tm02gHwncZT1=FyO-Ltw@mail.gmail.com>
References: 
 <CAJZ2dcW5JK3khtndSu4ixPfe5WgHOj-=iCJaUN3pdhK1DY_6ZQ@mail.gmail.com>
 <CAAdrtT3upgK-JXbaJ8MZ1_ZFbbWjM+VVPo0S0nCAZKwtrisUEg@mail.gmail.com>
 <CAJZ2dcV+-7jiJ9_+ahVO-Q8qDNXCk4Tm02gHwncZT1=FyO-Ltw@mail.gmail.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Wed, 16 Sep 2015 17:33:58 +0200
Message-ID: 
 <CAAdrtT1_L=1BdtMC4_X_eeSsX9o2JNozaN=i4iOU0Zdm_VjtqA@mail.gmail.com>
Subject: Re: CompactingHashTable question
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=089e0122797abb3f7d051fdf082c

--089e0122797abb3f7d051fdf082c
Content-Type: text/plain; charset=UTF-8

Yes, probing the HashTable with a key that does not exist will yield a join
function call with a null value (or empty iterator in case of CoGroup).

The semantics of the join are the same regardless of the hash table
implementation.
The fact that the error only occurs with the managed HT, indicates that
there is a bug somewhere :-(

2015-09-16 17:26 GMT+02:00 Vasiliki Kalavri <vasilikikalavri@gmail.com>:

> Hi,
>
> thanks a lot Fabian!
>
> I didn't know that join with the solution set is an outer join. That's a
> surprise :)
>
> So, if I understand correctly, I should have a null value when my other
> input to the join contains some key that doesn't exist in the solution set,
> right? That's not the case in my application; I'm not generating any new
> keys.
>
> Also, when setting the solutionSetUnManaged option, the exception doesn't
> occur anymore. Are the join semantics different when the solution set is in
> unmanaged memory?
>
> Cheers,
> Vasia.
>
>
> On 16 September 2015 at 16:50, Fabian Hueske <fhueske@gmail.com> wrote:
>
> > Hi Vasia,
> >
> > I looked into the code. A serializer should never return null when
> > deserializing. Either it does not detect that something went wrong with
> the
> > deserialization or it should throw an exception.
> >
> > Regarding the handling of null returns in the Drivers. If there is no
> entry
> > in the HT for a certain key, the HT will return null which is expected.
> > If a CoGroupWithSolutionSet*Driver receives a null value, it gives an
> empty
> > iterator to the user function. The JoinWithSolutionSet*Driver calls the
> > join function with a null value. Both behaviors are expected. A join
> with a
> > solution set is actually an outer join and a join function in such a join
> > needs to be able to handle null values on the solution set side.
> >
> > Cheers, Fabian
> >
> >
> > 2015-09-15 17:41 GMT+02:00 Vasiliki Kalavri <vasilikikalavri@gmail.com>:
> >
> > > Hello to my squirrels,
> > >
> > > I ran into an NPE for some iterations code and it looks like what's
> > > described in FLINK-2443 <
> > https://issues.apache.org/jira/browse/FLINK-2443
> > > >.
> > > I'm trying to understand the problem and I could really use your help
> :)
> > >
> > > So far, it seems that the exception is caused by a null value returned
> by
> > > CompactingHashTable.*getMatchFor*(PT probeSideRecord).
> > >
> > > This method returns null in the following cases:
> > > - when the hash table is "closed"
> > > - when the segment is done
> > > - if the serializer actually returns a null record
> > >
> > > It seems that on the join/cogroup driver side there is no check or
> > special
> > > handling when the build side record is null, i.e. the null record is
> > still
> > > passed to the join function.
> > > Is this correct and if not, what should the driver do in this case?
> > >
> > > Thank you!
> > >
> > > Cheers,
> > > Vasia.
> > >
> >
>

--089e0122797abb3f7d051fdf082c--