Return-Path: X-Original-To: apmail-nutch-dev-archive@www.apache.org Delivered-To: apmail-nutch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52E5993E2 for ; Wed, 2 Nov 2011 19:20:33 +0000 (UTC) Received: (qmail 1600 invoked by uid 500); 2 Nov 2011 19:20:32 -0000 Delivered-To: apmail-nutch-dev-archive@nutch.apache.org Received: (qmail 1568 invoked by uid 500); 2 Nov 2011 19:20:32 -0000 Mailing-List: contact dev-help@nutch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nutch.apache.org Delivered-To: mailing list dev@nutch.apache.org Received: (qmail 1558 invoked by uid 99); 2 Nov 2011 19:20:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2011 19:20:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lewis.mcgibbney@gmail.com designates 74.125.82.46 as permitted sender) Received: from [74.125.82.46] (HELO mail-ww0-f46.google.com) (74.125.82.46) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Nov 2011 19:20:28 +0000 Received: by wwe5 with SMTP id 5so768714wwe.15 for ; Wed, 02 Nov 2011 12:20:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=9C7tI4RyxmPQz0oLbd9Qcj3472Ap5iVpg4k8ISQqCiI=; b=WwPax9yDZ01IntOgUxp7yqASvoKuP7xokatyqp72Z8URLaytsvBKW4+Z7auE4r2iMz 1nf3RbAailwqOQBsqChLscumRHG9Zd0UcERIvMSo4U7TNlpZ9WuiX+DZvzWJwQPjvyKP TGQxu3l3N8VJWEfAAP0AXJ0Uxren7+7pQWIHI= MIME-Version: 1.0 Received: by 10.216.137.86 with SMTP id x64mr1840002wei.2.1320261605269; Wed, 02 Nov 2011 12:20:05 -0700 (PDT) Received: by 10.216.49.71 with HTTP; Wed, 2 Nov 2011 12:20:05 -0700 (PDT) In-Reply-To: References: Date: Wed, 2 Nov 2011 19:20:05 +0000 Message-ID: Subject: Re: Setting properties in gora.properties From: Lewis John Mcgibbney To: dev@nutch.apache.org, gora-dev@incubator.apache.org Content-Type: multipart/alternative; boundary=0016e6d7843bfe204b04b0c55bbc --0016e6d7843bfe204b04b0c55bbc Content-Type: text/plain; charset=ISO-8859-1 Hi, (I'm keeping gora-dev's in on this one as well) A quick update to this one, having submitted a work in progress patch for NUTCH-1189, so that properties are easier for the gora-cassandra module to find I'm trying to get the configuration to work as follows # Cassandra store properties # gora.CassandraMapping.keyspace= # gora.CassandraMapping.name= # gora.CassandraMapping.class= # gora.CassandraMapping.qualifier= # gora.CassandraMapping.family= # gora.CassandraMapping.type= # gora.CassandraMapping.cluster= gora.CassandraMapping.host=localhost://127.0.0.1:9160 However I'm still getting Caused by: java.io.IOException: Property with base name "servers" could not be found, make sure to include this property in gora.properties file This is caused by a null value being returned for baseKey parameter in DataStoreFactory. I think it's important to note, that in order to get Nutchgora to build with the gora-cassandra properties uncommented in NUTCHGORA/ivy/ivy.xml it was necessary to include the following exclusions Firstly, does anyone have a solution regarding the removing or substituting the exclusions? Enis has mentioned that they should be included within gora-cassandra/ivy/ivy.xml, however as this has not been loaded to maven repos it is not currently a possibility therefore they need to be included in the Nutchgora side of things. Secondly, does anyone know how the illusive baseKey parameter should be included within the gora.properties? Presumably something similar to # gora.CassandraMapping.servers=somethingSomething.com This is nearly sorted (I hope) but there are a couple of areas that need attention, therefore anyone with some insight would be great. Thanks, and sorry for ridiculous thread Lewis On Tue, Nov 1, 2011 at 2:01 PM, Lewis John Mcgibbney < lewis.mcgibbney@gmail.com> wrote: > Hi, > > I'm currently trying to complete NUTCH-902 and GORA-39 and kill two birds > with the one stone, however I've uprooted some more nasties which I'm now > trying to address. When configuring Nutchgora with Cassandra I'm getting > the following > > lewis@lewis-01:~/ASF/nutchgora/runtime/local$ bin/nutch inject urls > crawldb > InjectorJob: starting > InjectorJob: urlDir: urls > InjectorJob: org.apache.gora.util.GoraException: java.io.IOException: > java.io.IOException: Property with base name "servers" could not be found, > make sure to include this property in gora.properties file > at > org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:110) > at > org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:93) > at > org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:59) > at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243) > at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268) > at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:282) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:292) > Caused by: java.io.IOException: java.io.IOException: Property with base > name "servers" could not be found, make sure to include this property in > gora.properties file > at > org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:462) > at > org.apache.gora.cassandra.store.CassandraStore.initialize(CassandraStore.java:91) > at > org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:81) > at > org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:104) > ... 7 more > Caused by: java.io.IOException: Property with base name "servers" could > not be found, make sure to include this property in gora.properties file > at > org.apache.gora.store.DataStoreFactory.findPropertyOrDie(DataStoreFactory.java:254) > at > org.apache.gora.cassandra.store.CassandraStore.createClient(CassandraStore.java:394) > at > org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:425) > ... 10 more > > Can someone please explain a bit about what kind of properties we > can/should add to gora.properties for cassandra setup. I've tried editing > gora.properties as follows with no luck > > #gora.sqlstore.jdbc.driver=org.hsqldb.jdbcDriver > #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest > servers=localhost/127.0.0.1:9160 > > If there are any resources people are aware of on the net then I'll begin > getting my head around them. > > Thanks in advance > > Lewis > > > -- > *Lewis* > > -- *Lewis* --0016e6d7843bfe204b04b0c55bbc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,

(I'm keeping gora-dev's in on this one as well)

A= quick update to this one, having submitted a work in progress patch for NU= TCH-1189, so that properties are easier for the gora-cassandra module to fi= nd I'm trying to get the configuration to work as follows

# Cassandra store properties
# gora.CassandraMapping.keyspace=3D
= # gora.CassandraMapping.name<= /a>=3D
# gora.CassandraMapping.class=3D
# gora.CassandraMapping.quali= fier=3D
# gora.CassandraMapping.family=3D
# gora.CassandraMapping.type=3D
# g= ora.CassandraMapping.cluster=3D
gora.CassandraMapping.host=3Dlocalhost:/= /127.0.0.1:9160

However I'= m still getting

Caused by: java.io.IOException: Property with base name "servers&q= uot; could=20 not be found, make sure to include this property in gora.properties file
This is caused by a null value being returned for baseKey parameter in= DataStoreFactory.

I think it's important to note, that in orde= r to get Nutchgora to build with the gora-cassandra properties uncommented = in NUTCHGORA/ivy/ivy.xml it was necessary to include the following exclusio= ns

=A0=A0=A0 =A0=A0=A0 <dependency org=3D"org.apache.gora" na= me=3D"gora-cassandra" rev=3D"0.1.1-incubating" conf=3D&= quot;*->default">
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 <exclude o= rg=3D"org.apache.thrift" />
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 <exclude org=3D"org.apache.cassandra&= quot; />
=A0=A0=A0 =A0=A0=A0 </dependency>

Firstly, does= anyone have a solution regarding the removing or substituting the exclusio= ns? Enis has mentioned that they should be included within gora-cassandra/i= vy/ivy.xml, however as this has not been loaded to maven repos it is not cu= rrently a possibility therefore they need to be included in the Nutchgora s= ide of things.
Secondly, does anyone know how the illusive baseKey parameter should be inc= luded within the gora.properties?
Presumably something similar to
# gora.CassandraMapping.servers=3DsomethingSomething.com

This is n= early sorted (I hope) but there are a couple of areas that need attention, = therefore anyone with some insight would be great.

Thanks, and sorry for ridiculous thread

Lewis

On Tue, Nov 1, 2011 at 2:01 PM, Lewis John Mcgibbney <lewis.mcgibbn= ey@gmail.com> wrote:
Hi,

I'm currently trying to comp= lete NUTCH-902 and GORA-39 and kill two birds with the one stone, however I= 've uprooted some more nasties which I'm now trying to address. Whe= n configuring Nutchgora with Cassandra I'm getting the following

lewis@lewis-01:~/ASF/nutchgora/runtime/local$ bin/nutch inject urls cra= wldb
InjectorJob: starting
InjectorJob: urlDir: urls
InjectorJob: org.apac= he.gora.util.GoraException: java.io.IOException: java.io.IOException: Prope= rty with base name "servers" could not be found, make sure to inc= lude this property in gora.properties file
=A0=A0=A0 at org.apache.gora.store.DataStoreFactory.createDataStore(DataSto= reFactory.java:110)
=A0=A0=A0 at org.apache.gora.store.DataStoreFactory.= createDataStore(DataStoreFactory.java:93)
=A0=A0=A0 at org.apache.nutch.= storage.StorageUtils.createWebStore(StorageUtils.java:59)
=A0=A0=A0 at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)=A0=A0=A0 at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:2= 68)
=A0=A0=A0 at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java= :282)
=A0=A0=A0 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java= :65)
=A0=A0=A0 at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:292)<= br>Caused by: java.io.IOException: java.io.IOException: Property with base = name "servers" could not be found, make sure to include this prop= erty in gora.properties file
=A0=A0=A0 at org.apache.gora.cassandra.store.CassandraStore.readMapping(Cas= sandraStore.java:462)
=A0=A0=A0 at org.apache.gora.cassandra.store.Cassa= ndraStore.initialize(CassandraStore.java:91)
=A0=A0=A0 at org.apache.gor= a.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:81)
=A0=A0=A0 at org.apache.gora.store.DataStoreFactory.createDataStore(DataSto= reFactory.java:104)
=A0=A0=A0 ... 7 more
Caused by: java.io.IOExcepti= on: Property with base name "servers" could not be found, make su= re to include this property in gora.properties file
=A0=A0=A0 at org.apache.gora.store.DataStoreFactory.findPropertyOrDie(DataS= toreFactory.java:254)
=A0=A0=A0 at org.apache.gora.cassandra.store.Cassa= ndraStore.createClient(CassandraStore.java:394)
=A0=A0=A0 at org.apache.= gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:425) =A0=A0=A0 ... 10 more

Can someone please explain a bit about what ki= nd of properties we can/should add to gora.properties for cassandra setup. = I've tried editing gora.properties as follows with no luck

#gora= .sqlstore.jdbc.driver=3Dorg.hsqldb.jdbcDriver
#gora.sqlstore.jdbc.url=3Djdbc:hsqldb:hsql://localhost/nutchtest
servers= =3Dlocalhost/127.0.0.1:= 9160

If there are any resources people are aware of on the net t= hen I'll begin getting my head around them.

Thanks in advance
Lewis


--
Lewis




--
Lewis

--0016e6d7843bfe204b04b0c55bbc--