incubator-accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Newton <eric.new...@gmail.com>
Subject Re: Filter use
Date Fri, 16 Dec 2011 13:52:01 GMT
Do you see a stack trace in the tablet server log?

On Fri, Dec 16, 2011 at 8:47 AM, Joey Daughtery
<jdaughtery@t-sciences.com>wrote:

> All
> I was finally able to get back to this.  When I implemented the following
> on Accumulo 1.4 and execute, I get the exception below.
>
> Also, where are the option keys defined for IteratorSetting.addOption?
>
> 1.  I subclassed the org.apache.accumulo.core.iterators.Filter object and
> implemented the accept(Key key, Value value) method.
>
> 2. Then I utilized it in the following way:
> IteratorSetting is = new IteratorSetting(15, "idFilter", fi.getClass());
>         is.addOption("valuesToSkip", userId);
>
>
>         Range r = new Range();
>         Scanner s = conn.createScanner("index_table", scanauths);
>         s.setRange(r);
>         s.addScanIterator(is);
>
>
>
> Caused by: org.apache.thrift.TApplicationException: Internal error
> processing startScan
>     at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
>     at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_startScan(TabletClientService.java:184)
>     at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.startScan(TabletClientService.java:157)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at cloudtrace.instrument.thrift.TraceWrap$2.invoke(TraceWrap.java:67)
>     at $Proxy1.startScan(Unknown Source)
>     at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:415)
>     at
> org.apache.accumulo.core.client.impl.ThriftScanner.scan(ThriftScanner.java:295)
>     ... 4 more
>
> Thanks
>
> Joe
>
>
> On Mon, Dec 12, 2011 at 10:23 AM, Billie J Rinaldi <
> billie.j.rinaldi@ugov.gov> wrote:
>
>> On Date: Dec 12, 2011 9:29 AM, "Aaron Cordova" <
>> aaron.cordova@interllective.com > wrote:
>> > 3. Configure your table to use your filtering iterator as described in
>> > this section of the user manual:
>> >
>> http://incubator.apache.org/accumulo/user_manual_1.3-incubating/Table_Configuration.html
>> > , where you specify the name of your Java class as the first option in
>> > the Shell, rather than 'ageoff' or 'regex'.
>>
>> Unfortunately the manual does not say how to specify filters
>> programmatically.  This is somewhat confusing in 1.3 because the Filter
>> class you will be writing is a plugin to the FilteringIterator, and the
>> Filter is not itself an Iterator.  Configuring the filter in the shell and
>> just setting its parameters programmatically is a possibility, but the
>> syntax Aaron gave for setting the parameter is incorrect.  Here is an
>> example of configuring a FilteringIterator:
>>
>> scanner.setScanIterators(priority,
>> "org.apache.accumulo.core.iterators.FilteringIterator", "myFilter");
>> scanner.setScanIteratorOption("myFilter", "0",
>> "org.apache.accumulo.core.iterators.filter.MyFilter");
>> scanner.setScanIteratorOption("myFilter", "0.valuesToSkip", "10");
>> scanner.setScanIteratorOption("myFilter", "1",
>> "org.apache.accumulo.core.iterators.filter.AgeOffFilter");
>> scanner.setScanIteratorOption("myFilter", "1.ttl", ttl);
>>
>> If you can start coding to 1.4 instead of 1.3, I would highly recommend
>> it.  All provided user iterators in 1.4 are located in the
>> org.apache.accumulo.core.iterators.user package.  Filters are Iterators and
>> are much easier to configure:
>>
>> IteratorSetting setting = new IteratorSetting(priority, MyFilter.class);
>> setting.addOption("valuesToSkip", "10");
>> scanner.addScanIterator(setting);
>>
>> ** it is recommended to configure the IteratorSetting with static methods
>> instead of addOption,
>>   e.g. TimestampFilter.setRange(setting, "20120101000000GMT",
>> "20120102000000GMT");
>>
>> One of the above options should work.  However, filtering based on the
>> Value, the unsorted portion of a sorted key/value store, is not going to be
>> an efficient operation.  Your query will have to scan over all the entries
>> in the table to find the Values you want.  This might be ok if the query
>> returns most of the data in the table, but if it just returns a small
>> fraction of the data it would be better to figure out how to get the
>> sorting to work to your advantage.
>>
>> Billie
>>
>
>

Mime
View raw message