Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 45097 invoked from network); 23 Apr 2010 04:59:09 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 23 Apr 2010 04:59:09 -0000 Received: (qmail 51949 invoked by uid 500); 23 Apr 2010 04:59:08 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 51688 invoked by uid 500); 23 Apr 2010 04:59:08 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 51675 invoked by uid 99); 23 Apr 2010 04:59:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:59:07 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.160.176 as permitted sender) Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:59:02 +0000 Received: by gyf1 with SMTP id 1so5119310gyf.35 for ; Thu, 22 Apr 2010 21:58:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type :content-transfer-encoding; bh=oWsmk7/ZLQqgfFkFK/ccDKv44/A2WdDiYvw/4GeWrko=; b=dYYYuKUR/BF2q/w4yf+qXzAzPH8U11D22VuK/d/ByUZrVUxqrWGf8Imw2n9bW5Yd/K 1Txqb+VgDXGtrHgdrgy6SZR2WxY8enUIzAHfiz3/gDU2bZWPsaPf3c9GAlWLxZ0L2Ll2 sO9RKJDB1YgLOcjw1ix+RXrRN6aPALoOujf0g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=pm7K86ivwVy8/AofXMFD+NNBKaoTA0xpiRQx9euUy0N3Uak9JBZpAspbMpzGCHSJYt j7zJVLg/7Zyj1CUOsO24Zk+xuLhkpm+UfulPfHrMwyzsB0ShFJqY0tOIrH6/DtMU2Owx dllrN8ALNdaouWOxukMjOvUQglfaHqWgKJmTI= MIME-Version: 1.0 Received: by 10.150.230.19 with HTTP; Thu, 22 Apr 2010 21:58:41 -0700 (PDT) In-Reply-To: References: Date: Fri, 23 Apr 2010 00:58:41 -0400 Received: by 10.150.13.7 with SMTP id 7mr2648867ybm.238.1271998721584; Thu, 22 Apr 2010 21:58:41 -0700 (PDT) Message-ID: Subject: Re: DBOutputFormat over SSH? From: Gautam Singaraju To: general@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Sonal, I will check it out. --- Gautam On Fri, Apr 23, 2010 at 12:27 AM, Sonal Goyal wrote= : > Hi Gautam, > > DBOutputFormat inserts records one by one, which is inherently slow. You = can > use open source Apache licensed hiho framework which provides MySQL's "lo= ad > data infile " functionality. =A0It may be more suited to your needs. > > HIHO is available at http://code.google.com/p/hiho/ > > I havent tested it over ssh, please let me know if you need any help sett= ing > it up. > > Thanks and Regards, > Sonal > www.meghsoft.com > > > On Fri, Apr 23, 2010 at 9:48 AM, Gautam Singaraju < > gautam.singaraju@gmail.com> wrote: > >> All, >> >> I have a use-case where I need to crunch a large amount of data and >> push to the results (comparatively a smaller set) to a mysql db at a >> remote location. As per security concerns, only SSH ports are open. I >> tried using Java Secure Channel [1] in combination with some custom >> JDBC code from the reducers. >> >> Can anyone comment on the performance of DBOutputFormat? Have there >> been any efforts to tunnel this through SSH? This is going to be an >> expensive operation; any suggestions would be welcome. >> >> [1] http://www.jcraft.com/jsch/ >> --- >> Gautam Singaraju >> >