Return-Path: X-Original-To: apmail-couchdb-user-archive@www.apache.org Delivered-To: apmail-couchdb-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 28D27101B2 for ; Wed, 9 Oct 2013 18:26:03 +0000 (UTC) Received: (qmail 35571 invoked by uid 500); 9 Oct 2013 18:25:57 -0000 Delivered-To: apmail-couchdb-user-archive@couchdb.apache.org Received: (qmail 35404 invoked by uid 500); 9 Oct 2013 18:25:56 -0000 Mailing-List: contact user-help@couchdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@couchdb.apache.org Delivered-To: mailing list user@couchdb.apache.org Received: (qmail 35384 invoked by uid 99); 9 Oct 2013 18:25:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Oct 2013 18:25:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of mark.deibert@gmail.com designates 209.85.220.176 as permitted sender) Received: from [209.85.220.176] (HELO mail-vc0-f176.google.com) (209.85.220.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Oct 2013 18:25:51 +0000 Received: by mail-vc0-f176.google.com with SMTP id lf11so815767vcb.21 for ; Wed, 09 Oct 2013 11:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Yh0ixSQNWKj7gg/fVgIar5s0AHYX+gUkjFnPeF3I4xw=; b=zqsxpsSwTePnAU1riRMS1kI1IJna3PDo9D8lOmY0bRd3olLyw2+wX9LWtxtCodhsv3 1D6nBMqTEHUKClrPGDSb7cT/CwZlEoGgKsYRtwCWSu2BLxImUuvwzUs3TzzLEMzDbMJy wQtXy1Nki7Q+Rnn6ubbZsYGDOKRmL8kpSsfA9FyHJquWXZHgXrsTte4Xc6fwV57OZbgq J/cPI56zezd2g7E+J4yp6x4OTrfidSTEXtLewuqQWMNIISm+1uINtec61cQc9J6RE79Z p0t4xQyrpCQ32gceCQNQZMgsaVpSAvE1EzrXyDs4dB+dr/qkz/l1c0Q+wKqpFkzReoz9 KEgw== MIME-Version: 1.0 X-Received: by 10.52.190.201 with SMTP id gs9mr18209vdc.97.1381343129830; Wed, 09 Oct 2013 11:25:29 -0700 (PDT) Received: by 10.220.198.199 with HTTP; Wed, 9 Oct 2013 11:25:29 -0700 (PDT) In-Reply-To: References: Date: Wed, 9 Oct 2013 14:25:29 -0400 Message-ID: Subject: Re: crawl webpages into couchdb From: Mark Deibert To: user@couchdb.apache.org Content-Type: multipart/alternative; boundary=089e013a1634910c0304e85302e2 X-Virus-Checked: Checked by ClamAV on apache.org --089e013a1634910c0304e85302e2 Content-Type: text/plain; charset=ISO-8859-1 Are we in a CouchDB group or a "web crawler" apps group? :-/ On Wed, Oct 9, 2013 at 2:09 PM, Chad Cross wrote: > Affi, > > CouchDB doesn't natively solve the web crawling issue. I'm currently > experimenting with Scrapy (http://scrapy.org) for web crawling, but I > haven't advanced enough to start pushing my crawling data into CouchDB. > Maybe some users out there have experience with Scrapy and CouchDB? > > -Chad > > > On Wed, Oct 9, 2013 at 1:45 PM, Brad Rhoads wrote: > > > Or better yet, casperjs. > > On Oct 7, 2013 3:37 PM, "Mark Hahn" wrote: > > > > > Use node, phantomjs, and the nano couchdb driver. > > > > > > > > > On Mon, Oct 7, 2013 at 2:24 PM, affi wrote: > > > > > > > hi , > > > > i am a beginner at couchdb and am learning it for a uni project. i > have > > > > watched many tutorials on JSON and understand how to add documents. > > but i > > > > dont > > > > understand how to crawl webpages and store them in the couchdb > > database. > > > > would definitely appreciate some help with this. thanks > > > > > > > > > > > > > > --089e013a1634910c0304e85302e2--