Return-Path: Delivered-To: apmail-hbase-user-archive@www.apache.org Received: (qmail 59070 invoked from network); 5 Jun 2010 06:52:38 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 5 Jun 2010 06:52:38 -0000 Received: (qmail 26144 invoked by uid 500); 5 Jun 2010 06:52:37 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 25927 invoked by uid 500); 5 Jun 2010 06:52:35 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 25918 invoked by uid 99); 5 Jun 2010 06:52:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 06:52:34 +0000 X-ASF-Spam-Status: No, hits=4.7 required=10.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of m.vijayaraghava@gmail.com designates 209.85.214.169 as permitted sender) Received: from [209.85.214.169] (HELO mail-iw0-f169.google.com) (209.85.214.169) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 06:52:27 +0000 Received: by iwn39 with SMTP id 39so2123446iwn.14 for ; Fri, 04 Jun 2010 23:52:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=j4QyIi74L7Bmhn+rskBBWKv/48SCtH91Z9+mOZLzrqk=; b=uYs9lwRk5JOsWr2ZchIQmosL9WNhP4e6kgjBj895iJSkP2HqV2rPlS32SNzevFrbYT p1EvHMT9UoN6EtPgTU3BhxzfLuFlBaeCBrkkzKlBm3nqEzbrjUSYI0JwarLz6ITC8NW9 KzJ1WsGsBkj0I3VfB8YKDbHukA86ENu+oyHww= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=jv4jnp30E9Q6O3GFiawUJxVkBth+Dw1goa0aZHLpbUNaDvqIhkOlpe3u6q5MwjKc7q Xyekk5rLWaSv7dEcN3/qC5J7NSplhdSz+ek0xYvpJpgfkSwNhRp9wstCtSlHQGhxDk0U CMpvx2ghR/6/UjNUWBlEgd0ePtVRzaeHS/j7Y= Received: by 10.231.82.145 with SMTP id b17mr1891525ibl.187.1275720725579; Fri, 04 Jun 2010 23:52:05 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.173.7 with HTTP; Fri, 4 Jun 2010 23:51:45 -0700 (PDT) In-Reply-To: References: From: Raghava Mutharaju Date: Sat, 5 Jun 2010 02:51:45 -0400 Message-ID: Subject: Re: Scanner Exception To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=000e0cd7289cae4962048842e03e X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd7289cae4962048842e03e Content-Type: text/plain; charset=ISO-8859-1 I am using the stable release, 0.20.4. I think change of rows can be detected because reset() is called at the beginning of each row. For me, it would have been nice had I known if a column was the last one in the current family. I currently modified my schema a bit to make use of the currently existing filters. I swapped the column and value. Even earlier, the values were important and I didn't have any meaningful column names, so this change was ok. I will take a look at the filters in the trunk. Thank you for all the suggestions. Regards, Raghava. On Sat, Jun 5, 2010 at 2:42 AM, Stack wrote: > Are you on trunk? Filter interface is richer there. > > As to whether the below is possible, it sounds reasonable. I can't > say for certain though unless I dug in. You might be up against the > edge of whats possibly. Seems like you want to detect when we change > rows in your filter. Filters are inherently row-scoped. Perhaps that > is the issue you are running into? > > Look at trunk. I believe you get a last whack at influencing whats > returned just before its passed back to the client per row IIRC. > > St.Ack > > On Fri, Jun 4, 2010 at 12:40 PM, Raghava Mutharaju > wrote: > > Hi Stack, > > > > The custom filter is meant to do this --- given a columnFamily and > a > > value, it should check in all the columns whether that value is present > or > > not. If it is present then include the row or else skip it. > > > > I started of by writing this on the lines of > SingleColumnValueFilter(scvf). > > But the important difference is that in scvf, if the column is found and > the > > given value is not found against that column, then we can skip to next > row > > (using ReturnCode.NEXT_ROW). In my case, if there is a family match, I > have > > to check all the columns. Even if family does not match, I cannot do a > skip > > or next_row because there could be a match with other families of the > row. > > > > I put in a few log statements and traced the flow. I am wondering whether > > this type of filter is possible?? > > > > Regards, > > Raghava. > > > > On Fri, Jun 4, 2010 at 10:50 AM, Stack wrote: > > > >> Then something else is going on. Can you do a bit of digging? > >> St.Ack > >> > >> On Thu, Jun 3, 2010 at 11:38 PM, Raghava Mutharaju > >> wrote: > >> > Nope, it doesn't. > >> > > >> > After recompiling, I just changed the $HBASE_HOME to the newly created > >> build > >> > folder. I think that is the only change required? > >> > > >> > Regards, > >> > Raghava. > >> > > >> > On Fri, Jun 4, 2010 at 1:46 AM, Stack wrote: > >> > > >> >> Does your filter start working if you add it in here w/ a code of its > >> >> own and recompile and relaunch? > >> >> St.Ack > >> >> > >> >> On Thu, Jun 3, 2010 at 10:40 PM, Raghava Mutharaju > >> >> wrote: > >> >> > I found the codes in HBaseObjectWritable file and saw the jira too > :). > >> >> > This change only effects the performance isn't it (particularly for > >> small > >> >> > cell data), not the actual functionality. > >> >> > > >> >> > Regards, > >> >> > Raghava. > >> >> > > >> >> > On Fri, Jun 4, 2010 at 1:18 AM, Stack wrote: > >> >> > > >> >> >> Look in that java file and see how all extant filters are > mentioned > >> >> >> and assigned a code. You need to add yours too. > >> >> >> St.Ack > >> >> >> > >> >> >> On Thu, Jun 3, 2010 at 10:14 PM, Raghava Mutharaju > >> >> >> wrote: > >> >> >> > Thank you Angus and Guilherme :). I added the filter jar to > >> $HBase/lib > >> >> >> and > >> >> >> > the exceptions went away. Although, there now the filter doesn't > >> >> return > >> >> >> any > >> >> >> > rows - I have to check this out. > >> >> >> > > >> >> >> > Stack: > >> >> >> > About HBaseObjectWritable, I have used it in > serializing(write() > >> >> >> method) > >> >> >> > and deserializing(readFields() method) as was done in other > >> existing > >> >> >> > filters. Are you referring to this or something else? What does > >> adding > >> >> it > >> >> >> to > >> >> >> > HBaseObjectWritable mean? > >> >> >> > > >> >> >> > Thank you again. > >> >> >> > > >> >> >> > Regards, > >> >> >> > Raghava. > >> >> >> > > >> >> >> > On Fri, Jun 4, 2010 at 1:04 AM, Stack wrote: > >> >> >> > > >> >> >> >> You have to add it to HBaseObjectWritable too, IIRC. See how > >> other > >> >> >> >> filters are mentioned in there (The need to do this has to go > >> away. > >> >> I > >> >> >> >> filed HBASE-2666). > >> >> >> >> St.Ack > >> >> >> >> > >> >> >> >> On Thu, Jun 3, 2010 at 9:01 PM, Angus He > >> wrote: > >> >> >> >> > Even if HBase is running in standalone mode, the scan > operation > >> is > >> >> >> >> > still running in another jvm, to be specific, in HMaster > >> process. > >> >> >> >> > > >> >> >> >> > So you still have to either put the custom filter jar in > >> $HBASE/lib > >> >> or > >> >> >> >> > set up the $HBASE/conf/hbase-env.sh properly. > >> >> >> >> > > >> >> >> >> > On Fri, Jun 4, 2010 at 4:37 AM, Raghava Mutharaju > >> >> >> >> > wrote: > >> >> >> >> >> The custom filter doesn't need any additional jars. > >> >> >> >> >> Another point I forgot to mention is that, I am running this > on > >> a > >> >> >> single > >> >> >> >> >> node (laptop) to test my filter. > >> >> >> >> >> > >> >> >> >> >> Regards, > >> >> >> >> >> Raghava. > >> >> >> >> >> > >> >> >> >> >> On Thu, Jun 3, 2010 at 4:23 PM, Guilherme Germoglio < > >> >> >> >> germoglio@gmail.com>wrote: > >> >> >> >> >> > >> >> >> >> >>> please check if the jars needed for your custom filter > >> >> >> implementation > >> >> >> >> are > >> >> >> >> >>> in > >> >> >> >> >>> hbase's classpath > >> >> >> >> >>> > >> >> >> >> >>> On Thu, Jun 3, 2010 at 5:16 PM, Raghava Mutharaju < > >> >> >> >> >>> m.vijayaraghava@gmail.com > >> >> >> >> >>> > wrote: > >> >> >> >> >>> > >> >> >> >> >>> > Hi all, > >> >> >> >> >>> > > >> >> >> >> >>> > I wrote a custom filter and used it with scan. I am > >> >> getting > >> >> >> the > >> >> >> >> >>> > following exceptions. If I use any built-in filters, it > >> works > >> >> >> fine. I > >> >> >> >> >>> > searched around and one of the suggestions was to > increase > >> the > >> >> >> lease > >> >> >> >> >>> > timeout. Since it works fine for built-in filters, I am > >> >> assuming > >> >> >> that > >> >> >> >> >>> this > >> >> >> >> >>> > is not the case. Is it that there is something wrong with > my > >> >> >> filter > >> >> >> >> >>> > implementation? > >> >> >> >> >>> > > >> >> >> >> >>> > Regards, > >> >> >> >> >>> > Raghava. > >> >> >> >> >>> > > >> >> >> >> >>> > Master log: > >> >> >> >> >>> > > >> >> >> >> >>> > org.apache.hadoop.hbase.UnknownScannerException: Name: -1 > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1889) > >> >> >> >> >>> > at > >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> >> >> Method) > >> >> >> >> >>> > > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> >> >> >> >>> > at > java.lang.reflect.Method.invoke(Method.java:597) > >> >> >> >> >>> > > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > >> >> >> >> >>> > 2010-06-03 16:02:10,775 INFO > >> org.apache.hadoop.ipc.HBaseServer: > >> >> >> IPC > >> >> >> >> >>> > Server handler 9 on 60770, call next(-1, 1) from > >> >> >> >> 130.108.56.225:61091: > >> >> >> >> >>> > error: org.apache.hadoop.hbase.UnknownScannerException: > >> Name: > >> >> -1 > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> > Console: > >> >> >> >> >>> > > >> >> >> >> >>> > Exception in thread "main" > >> >> >> >> >>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: > >> >> Trying > >> >> >> to > >> >> >> >> >>> contact > >> >> >> >> >>> > region server 130.108.56.225:60770 for region > >> >> >> table1,,1274648045785, > >> >> >> >> row > >> >> >> >> >>> > '', > >> >> >> >> >>> > but failed after 10 attempts. > >> >> >> >> >>> > Exceptions: > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > java.io.IOException: Call to /130.108.56.225:60770failed > >> on > >> >> >> local > >> >> >> >> >>> > exception: java.io.EOFException > >> >> >> >> >>> > > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1055) > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:2003) > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable.java:1923) > >> >> >> >> >>> > at > >> >> >> >> > org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:403) > >> >> >> >> >>> > at > >> >> >> >> >>> > > >> >> >> >> > >> >> > org.knoesis.reasoning.MR.FilterReadClient.main(FilterReadClient.java:58) > >> >> >> >> >>> > > >> >> >> >> >>> > Line 58 above has a call to > getScanner(CustomFilterInstance) > >> of > >> >> >> >> HTable. > >> >> >> >> >>> > > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> -- > >> >> >> >> >>> Guilherme > >> >> >> >> >>> > >> >> >> >> >>> msn: guigermoglio@hotmail.com > >> >> >> >> >>> homepage: http://sites.google.com/site/germoglio/ > >> >> >> >> >>> > >> >> >> >> >> > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > > >> >> >> >> > -- > >> >> >> >> > Regards > >> >> >> >> > Angus > >> >> >> >> > > >> >> >> >> > >> >> >> > > >> >> >> > >> >> > > >> >> > >> > > >> > > > --000e0cd7289cae4962048842e03e--