nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Setting properties in gora.properties
Date Wed, 02 Nov 2011 19:20:05 GMT
Hi,

(I'm keeping gora-dev's in on this one as well)

A quick update to this one, having submitted a work in progress patch for
NUTCH-1189, so that properties are easier for the gora-cassandra module to
find I'm trying to get the configuration to work as follows

# Cassandra store properties
# gora.CassandraMapping.keyspace=
# gora.CassandraMapping.name=
# gora.CassandraMapping.class=
# gora.CassandraMapping.qualifier=
# gora.CassandraMapping.family=
# gora.CassandraMapping.type=
# gora.CassandraMapping.cluster=
gora.CassandraMapping.host=localhost://127.0.0.1:9160

However I'm still getting

Caused by: java.io.IOException: Property with base name "servers" could not
be found, make sure to include this property in gora.properties file

This is caused by a null value being returned for baseKey parameter in
DataStoreFactory.

I think it's important to note, that in order to get Nutchgora to build
with the gora-cassandra properties uncommented in NUTCHGORA/ivy/ivy.xml it
was necessary to include the following exclusions

        <dependency org="org.apache.gora" name="gora-cassandra"
rev="0.1.1-incubating" conf="*->default">
            <exclude org="org.apache.thrift" />
            <exclude org="org.apache.cassandra" />
        </dependency>

Firstly, does anyone have a solution regarding the removing or substituting
the exclusions? Enis has mentioned that they should be included within
gora-cassandra/ivy/ivy.xml, however as this has not been loaded to maven
repos it is not currently a possibility therefore they need to be included
in the Nutchgora side of things.
Secondly, does anyone know how the illusive baseKey parameter should be
included within the gora.properties?
Presumably something similar to

# gora.CassandraMapping.servers=somethingSomething.com

This is nearly sorted (I hope) but there are a couple of areas that need
attention, therefore anyone with some insight would be great.

Thanks, and sorry for ridiculous thread

Lewis

On Tue, Nov 1, 2011 at 2:01 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:

> Hi,
>
> I'm currently trying to complete NUTCH-902 and GORA-39 and kill two birds
> with the one stone, however I've uprooted some more nasties which I'm now
> trying to address. When configuring Nutchgora with Cassandra I'm getting
> the following
>
> lewis@lewis-01:~/ASF/nutchgora/runtime/local$ bin/nutch inject urls
> crawldb
> InjectorJob: starting
> InjectorJob: urlDir: urls
> InjectorJob: org.apache.gora.util.GoraException: java.io.IOException:
> java.io.IOException: Property with base name "servers" could not be found,
> make sure to include this property in gora.properties file
>     at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:110)
>     at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:93)
>     at
> org.apache.nutch.storage.StorageUtils.createWebStore(StorageUtils.java:59)
>     at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:243)
>     at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:268)
>     at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:282)
>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>     at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:292)
> Caused by: java.io.IOException: java.io.IOException: Property with base
> name "servers" could not be found, make sure to include this property in
> gora.properties file
>     at
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:462)
>     at
> org.apache.gora.cassandra.store.CassandraStore.initialize(CassandraStore.java:91)
>     at
> org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:81)
>     at
> org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:104)
>     ... 7 more
> Caused by: java.io.IOException: Property with base name "servers" could
> not be found, make sure to include this property in gora.properties file
>     at
> org.apache.gora.store.DataStoreFactory.findPropertyOrDie(DataStoreFactory.java:254)
>     at
> org.apache.gora.cassandra.store.CassandraStore.createClient(CassandraStore.java:394)
>     at
> org.apache.gora.cassandra.store.CassandraStore.readMapping(CassandraStore.java:425)
>     ... 10 more
>
> Can someone please explain a bit about what kind of properties we
> can/should add to gora.properties for cassandra setup. I've tried editing
> gora.properties as follows with no luck
>
> #gora.sqlstore.jdbc.driver=org.hsqldb.jdbcDriver
> #gora.sqlstore.jdbc.url=jdbc:hsqldb:hsql://localhost/nutchtest
> servers=localhost/127.0.0.1:9160
>
> If there are any resources people are aware of on the net then I'll begin
> getting my head around them.
>
> Thanks in advance
>
> Lewis
>
>
> --
> *Lewis*
>
>


-- 
*Lewis*

Mime
View raw message