Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of jtaylor@salesforce.com
 designates 209.85.128.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANNBgPK+zibNihUkVSDXJVVBM+SoQA7rSd8OWA_zhZc0SAv_4Q@mail.gmail.com>
References: 
 <CAFAd71W0Of-89kTLPZNSksGp2qSpoieFU75KQemJ-QqK65DLLw@mail.gmail.com>
	<CANNBgPK+zibNihUkVSDXJVVBM+SoQA7rSd8OWA_zhZc0SAv_4Q@mail.gmail.com>
Date: Thu, 2 Jan 2014 22:23:43 -0800
Message-ID: 
 <CAG_TOPBZ7oH=KOBvMFJ1Jx3=Kqi42E_OmpoWi8_VBGer=tf0Tw@mail.gmail.com>
Subject: Re: use hbase as distributed crawl's scheduler
From: James Taylor <jtaylor@salesforce.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=047d7bea31a6adec9a04ef0af3e0

--047d7bea31a6adec9a04ef0af3e0
Content-Type: text/plain; charset=ISO-8859-1

Otis,
I didn't realize Nutch uses HBase underneath. Might be interesting if you
serialized data in a Phoenix-compliant manner, as you could run SQL queries
directly on top of it.

Thanks,
James


On Thu, Jan 2, 2014 at 10:17 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> Have a look at http://nutch.apache.org .  Version 2.x uses HBase under the
> hood.
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Jan 3, 2014 at 1:12 AM, Li Li <fancyerii@gmail.com> wrote:
>
> > hi all,
> >      I want to use hbase to store all urls(crawled or not crawled).
> > And each url will has a column named priority which represent the
> > priority of the url. I want to get the top N urls order by priority(if
> > priority is the same then url whose timestamp is ealier is prefered).
> >      in using something like mysql, my client application may like:
> >      while true:
> >          select  url from url_db order by priority,addedTime limit
> > 1000 where status='not_crawled';
> >          do something with this urls;
> >          extract more urls and insert them into url_db;
> >      How should I design hbase schema for this application? Is hbase
> > suitable for me?
> >      I found in this article
> >
> http://blog.semantics3.com/how-we-built-our-almost-distributed-web-crawler/
> > ,
> > they use redis to store urls. I think hbase is originated from
> > bigtable and google use bigtable to store webpage, so for huge number
> > of urls, I prefer distributed system like hbase.
> >
>

--047d7bea31a6adec9a04ef0af3e0--