Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: 
 <CAG_TOPD-skF0DpD-PUPaqdsmrNR-EkkbfYbtvCYaKJy5V1EEFw@mail.gmail.com>
References: 
 <CAFAd71W0Of-89kTLPZNSksGp2qSpoieFU75KQemJ-QqK65DLLw@mail.gmail.com>
 <CANNBgPK+zibNihUkVSDXJVVBM+SoQA7rSd8OWA_zhZc0SAv_4Q@mail.gmail.com>
 <CAFAd71Uzeq_jUWuRUvU_gN4CXAbhBwo6GHtwr=+BvmphGqw7gA@mail.gmail.com>
 <CAG_TOPDpp+TWm0wZ3es6dPFwtn5x1grzrrXA-YsSHdXzRnVf+Q@mail.gmail.com>
 <CAFAd71UF8aVnwTOAK67uhnoJ=0dNfhpDU2_EEU9YQep91=+V+w@mail.gmail.com>
 <CAG_TOPD-skF0DpD-PUPaqdsmrNR-EkkbfYbtvCYaKJy5V1EEFw@mail.gmail.com>
From: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
Date: Fri, 3 Jan 2014 07:19:02 -0500
Message-ID: 
 <CAPQV63VmvxL8kj0FMpxqUJT-Xvy+Q0GmeunVU0qv_H5Bm6VDRw@mail.gmail.com>
Subject: Re: use hbase as distributed crawl's scheduler
To: user <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=089e013a27088f18b804ef0feb49

--089e013a27088f18b804ef0feb49
Content-Type: text/plain; charset=UTF-8

Interesting. This is exactly what I'm doing ;)

I'm using 3 tables to achieve this.

One table with the URL already crawled (80 millions), one URL with the URL
to crawle (2 billions) and one URL with the URLs been processed. I'm not
running any SQL requests against my dataset but I have MR jobs doing many
different things. I have many other tables to help with the work on the
URLs.

I'm "salting" the keys using the URL hash so I can find them back very
quickly. There can be some collisions so I store also the URL itself on the
key. So very small scans returning 1 or something 2 rows allow me to
quickly find a row knowing the URL.

I also have secondary index tables to store the CRCs of the pages to
identify duplicate pages based on this value.

And so on ;) Working on that for 2 years now. I might have been able to use
Nuthc and others, but my goal was to learn and do that with a distributed
client on a single dataset...

Enjoy.

JM


2014/1/3 James Taylor <jtaylor@salesforce.com>

> Sure, no problem. One addition: depending on the cardinality of your
> priority column, you may want to salt your table to prevent hotspotting,
> since you'll have a monotonically increasing date in the key. To do that,
> just add " SALT_BUCKETS=<n>" on to your query, where <n> is the number of
> machines in your cluster. You can read more about salting here:
> http://phoenix.incubator.apache.org/salted.html
>
>
> On Thu, Jan 2, 2014 at 11:36 PM, Li Li <fancyerii@gmail.com> wrote:
>
> > thank you. it's great.
> >
> > On Fri, Jan 3, 2014 at 3:15 PM, James Taylor <jtaylor@salesforce.com>
> > wrote:
> > > Hi LiLi,
> > > Have a look at Phoenix (http://phoenix.incubator.apache.org/). It's a
> > SQL
> > > skin on top of HBase. You can model your schema and issue your queries
> > just
> > > like you would with MySQL. Something like this:
> > >
> > > // Create table that optimizes for your most common query
> > > // (i.e. the PRIMARY KEY constraint should be ordered as you'd want
> your
> > > rows ordered)
> > > CREATE TABLE url_db (
> > >     status TINYINT,
> > >     priority INTEGER NOT NULL,
> > >     added_time DATE,
> > >     url VARCHAR NOT NULL
> > >     CONSTRAINT pk PRIMARY KEY (status, priority, added_time, url));
> > >
> > > int lastStatus = 0;
> > > int lastPriority = 0;
> > > Date lastAddedTime = new Date(0);
> > > String lastUrl = "";
> > >
> > > while (true) {
> > >     // Use row value constructor to page through results in batches of
> > 1000
> > >     String query = "
> > >         SELECT * FROM url_db
> > >         WHERE status=0 AND (status, priority, added_time, url) > (?, ?,
> > ?,
> > > ?)
> > >         ORDER BY status, priority, added_time, url
> > >         LIMIT 1000"
> > >     PreparedStatement stmt = connection.prepareStatement(query);
> > >
> > >     // Bind parameters
> > >     stmt.setInt(1, lastStatus);
> > >     stmt.setInt(2, lastPriority);
> > >     stmt.setDate(3, lastAddedTime);
> > >     stmt.setString(4, lastUrl);
> > >     ResultSet resultSet = stmt.executeQuery();
> > >
> > >     while (resultSet.next()) {
> > >         // Remember last row processed so that you can start after that
> > for
> > > next batch
> > >         lastStatus = resultSet.getInt(1);
> > >         lastPriority = resultSet.getInt(2);
> > >         lastAddedTime = resultSet.getDate(3);
> > >         lastUrl = resultSet.getString(4);
> > >
> > >         doSomethingWithUrls();
> > >
> > >         UPSERT INTO url_db(status, priority, added_time, url)
> > >         VALUES (1, ?, CURRENT_DATE(), ?);
> > >
> > >     }
> > > }
> > >
> > > If you need to efficiently query on url, add a secondary index like
> this:
> > >
> > > CREATE INDEX url_index ON url_db (url);
> > >
> > > Please let me know if you have questions.
> > >
> > > Thanks,
> > > James
> > >
> > >
> > >
> > >
> > > On Thu, Jan 2, 2014 at 10:22 PM, Li Li <fancyerii@gmail.com> wrote:
> > >
> > >> thank you. But I can't use nutch. could you tell me how hbase is used
> > >> in nutch? or hbase is only used to store webpage.
> > >>
> > >> On Fri, Jan 3, 2014 at 2:17 PM, Otis Gospodnetic
> > >> <otis.gospodnetic@gmail.com> wrote:
> > >> > Hi,
> > >> >
> > >> > Have a look at http://nutch.apache.org .  Version 2.x uses HBase
> > under
> > >> the
> > >> > hood.
> > >> >
> > >> > Otis
> > >> > --
> > >> > Performance Monitoring * Log Analytics * Search Analytics
> > >> > Solr & Elasticsearch Support * http://sematext.com/
> > >> >
> > >> >
> > >> > On Fri, Jan 3, 2014 at 1:12 AM, Li Li <fancyerii@gmail.com> wrote:
> > >> >
> > >> >> hi all,
> > >> >>      I want to use hbase to store all urls(crawled or not crawled).
> > >> >> And each url will has a column named priority which represent the
> > >> >> priority of the url. I want to get the top N urls order by
> > priority(if
> > >> >> priority is the same then url whose timestamp is ealier is
> prefered).
> > >> >>      in using something like mysql, my client application may like:
> > >> >>      while true:
> > >> >>          select  url from url_db order by priority,addedTime limit
> > >> >> 1000 where status='not_crawled';
> > >> >>          do something with this urls;
> > >> >>          extract more urls and insert them into url_db;
> > >> >>      How should I design hbase schema for this application? Is
> hbase
> > >> >> suitable for me?
> > >> >>      I found in this article
> > >> >>
> > >>
> >
> http://blog.semantics3.com/how-we-built-our-almost-distributed-web-crawler/
> > >> >> ,
> > >> >> they use redis to store urls. I think hbase is originated from
> > >> >> bigtable and google use bigtable to store webpage, so for huge
> number
> > >> >> of urls, I prefer distributed system like hbase.
> > >> >>
> > >>
> >
>

--089e013a27088f18b804ef0feb49--