Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of jmozah@gmail.com designates
 209.85.213.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHH9OkDNowY4q4ay_+Tsb-j1PEu5hmNpYA5wnm4zyF7qhrkT1A@mail.gmail.com>
References: 
 <CAHH9OkBu6k1vmo5VxJDZ9o5Lehcuy=EpBfNJ8usEQuP+8Z5P3w@mail.gmail.com>
	<26B78B7F-D21B-4F41-BAB8-D5080E3536C4@gmail.com>
	<CAHH9OkD663j-_MV0XB4CT0dq-P2KF=KpcC_8L33nmO0JphwpJQ@mail.gmail.com>
	<C74BF59F-A5D4-4B06-8CD2-034D7CECA721@gmail.com>
	<CAHH9OkDX0S=2Pdd3xjCB_nLGos7LBeH0Stn226_ytq8HU2osDA@mail.gmail.com>
	<CAHH9OkDNowY4q4ay_+Tsb-j1PEu5hmNpYA5wnm4zyF7qhrkT1A@mail.gmail.com>
Date: Tue, 7 Aug 2012 15:27:26 +0530
Message-ID: 
 <CADrVvWUOG4Aj5Ax==64sdhf7aqJXtBTBo5U57KyGSuksi1b4uA@mail.gmail.com>
Subject: Re: [potential bug]Find rows which do not have any of the given
 columns
From: J Mohamed Zahoor <jmozah@gmail.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=14dae934062b92b39604c6aa05e9

--14dae934062b92b39604c6aa05e9
Content-Type: text/plain; charset=ISO-8859-1

Hi

Nice one. But i think this is valid behavior.
Time ranges are something which qualifies certain rows to be made available
to the client (something which is related to MVCC).
Once a certain rows are qualified... then the filters are applied on them.

The fact that both can be set simultaneously on a "Scan" object hints that
they orthogonal.

./zahoor

On Tue, Aug 7, 2012 at 2:10 AM, Shrijeet Paliwal <shrijeet@rocketfuel.com>wrote:

> - user
> +dev
>
> Hi Devs,
>
> Please follow the discussion to get full context. tl:dr "Did a scan with
> timerange and filters, scan o/p was incorrect. Repeated scan with filter
> only, scan o/p was correct."
>
> HBase version : 0.90.3
> Hadoop : CDH3u0
> Issues:
> The scan when set with both a time range and a filter can behave in
> an unintuitive way. Calling it unintuitive instead of wrong, since I do not
> know if this is a known limitation of scan. Picture a filter setup like
> mine - "Filter rows which have cells pertaining to certain columns". This
> filter is set on a scan which has a time range constraint as well.  AFAIK
> we skip Hfiles based on metadata when dealing with time ranges. If a region
> has two Hfiles. One of the Hfiles has cells for unwanted columns but the
> other one does not - we may get incorrect result based on what how time
> range is set (If the time range scan optimizer skips the Hfile containing
> unwanted cells).
>
> Does this sound like a valid issue? Also I can see this happening to more
> than one kind of SkipFilters.
>
> -Shrijeet
>
>
> On Mon, Aug 6, 2012 at 11:38 AM, Shrijeet Paliwal
> <shrijeet@rocketfuel.com>wrote:
>
> > It seems setting time range is a problem , I was doing  (*
> > scan.setTimeRange(Long.**valueOf(args[4]), Long.valueOf(args[5]));)*
> > *
> > *
> > I was working on assumption that filter logic works before scan logic, in
> > other words a KV dropped by filter will not make it to scan. In case of
> > time range this might not be true.
> >
> > -Shrijeet
> >
> >
> > On Mon, Aug 6, 2012 at 9:25 AM, jmozah <jmozah@gmail.com> wrote:
> >
> >> Hmmm.. Missed it. Otherwise i dont spot anything wrong in this.
> >> are you sure about the column names?
> >>
> >> ./zahoor
> >>
> >>
> >> On 06-Aug-2012, at 9:34 PM, Shrijeet Paliwal <shrijeet@rocketfuel.com>
> >> wrote:
> >>
> >> > I am using FilterList. Could you elaborate?
> >> >
> >> > On Mon, Aug 6, 2012 at 8:48 AM, jmozah <jmozah@gmail.com> wrote:
> >> >
> >> >>
> >> >>
> >> >> Use FilterList instead of List of Filters.
> >> >>
> >> >> ./Zahoor
> >> >>
> >> >> On 06-Aug-2012, at 12:12 PM, Shrijeet Paliwal <
> shrijeet@rocketfuel.com
> >> >
> >> >> wrote:
> >> >>
> >> >>> Hi All,
> >> >>>
> >> >>> I am writing a job which finds rows that do not have a cell
> >> corresponding
> >> >>> to any of the columns in the given set of columns.
> >> >>> This is how I have configured my scan (a combination of
> >> lQualifierFilters
> >> >>> and SkipFilter)
> >> >>>
> >> >>>   columnsSet = Splitter.on(',') .split(columns); //columns is a csv
> >> >>> containing column names
> >> >>>   List<Filter> qualifierFilters = new ArrayList<Filter>();
> >> >>>   for (String qual : columnsSet) {
> >> >>>     qualifierFilters.add(new QualifierFilter(CompareOp.NOT_EQUAL,
> >> >>>         new BinaryComparator(Bytes.toBytes(qual))));
> >> >>>   }
> >> >>>   Filter skipFilter = new SkipFilter(new
> >> >>> FilterList(Operator.MUST_PASS_ALL, qualifierFilters));
> >> >>>   Scan scan = new Scan();
> >> >>>   scan.addFamily(Bytes.toBytes(family));
> >> >>>   scan.setCacheBlocks(false);
> >> >>>   scan.setCaching(1000);
> >> >>>   scan.setFilter(skipFilter);
> >> >>>   scan.setTimeRange(Long.valueOf(args[4]), Long.valueOf(args[5]));
> >> >>>
> >> >>> In my test table the scan worked as expected. But in production
> run, I
> >> >> got
> >> >>> rows which had cells containing one of the given qualifiers (not
> >> >> expected)
> >> >>> Can some one help me spot the mistake?
> >> >>>
> >> >>> -Shrijeet
> >> >>
> >> >>
> >>
> >>
> >
>

--14dae934062b92b39604c6aa05e9--