Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B5464E8B6 for ; Thu, 17 Jan 2013 11:18:28 +0000 (UTC) Received: (qmail 33063 invoked by uid 500); 17 Jan 2013 11:18:23 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 32680 invoked by uid 500); 17 Jan 2013 11:18:20 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 32666 invoked by uid 99); 17 Jan 2013 11:18:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jan 2013 11:18:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of balijamahesh.mca@gmail.com designates 209.85.217.171 as permitted sender) Received: from [209.85.217.171] (HELO mail-lb0-f171.google.com) (209.85.217.171) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jan 2013 11:18:13 +0000 Received: by mail-lb0-f171.google.com with SMTP id gg13so283185lbb.16 for ; Thu, 17 Jan 2013 03:17:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=16G0/2NrKGlL/MEUbtMJpWQ9dACi8QxQoCHmRxtr5K0=; b=Acib/SwOVGmR07ChcWJ+uflizM+2JRlXNuLpaWf9IMrs1rs5QoZM/9tn9stc0RLSkK xJpcFzqgK6TAsoz3KXwqH5N6nfcRDpBkusLBYXUI7S7/Eb1lFJ1OeuSUT/ZulqhRINGE WieD9Z1OOo5rn3vGK1l/XJNB7BDdjGvBddq8pJkz2UvyfL/hRaQGB+AhDmlT4sPmZ+Bv VRJCViQ8XpcvqF+FdnSKmjHzWAPI4JpXEhmYcOuENXn01AecqK289PSbumNRapOBqcxW Dfq3cnJ6zr9YjZGZrJH2XjQfXTkYee4h6X13IA6kPX5WYG0WttTdqsBZxy8r+wUdrCUD 1bkA== MIME-Version: 1.0 X-Received: by 10.152.144.38 with SMTP id sj6mr4368683lab.48.1358421472705; Thu, 17 Jan 2013 03:17:52 -0800 (PST) Received: by 10.112.74.198 with HTTP; Thu, 17 Jan 2013 03:17:52 -0800 (PST) In-Reply-To: References: Date: Thu, 17 Jan 2013 16:47:52 +0530 Message-ID: Subject: Re: How to copy log files from remote windows machine to Hadoop cluster From: Mahesh Balija To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e89a8f23471555fab104d37a2548 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f23471555fab104d37a2548 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable That link talks about just installing Flume on Windows machine (NOT even have configs to push logs to the Hadoop cluster), but what if I have to collect logs from various clients, then I will endup installing in all clients. I have installed Flume successfully on Linux but I have to configure it such a way that it should gather the log files from the remote windows box? Harsh can you throw some light on this? On Thu, Jan 17, 2013 at 4:21 PM, Mohammad Tariq wrote: > Yes. It is possible. I haven't tries windows+flume+hadoop combo > personally, but it should work. You may find this linkuseful. Alex > has explained beautifully how to run Flume on a windows box.If I > get time i'll try to simulate your use case and let you know. > > BTW, could you please share with us whatever you have tried?? > > Warm Regards, > Tariq > https://mtariq.jux.com/ > cloudfront.blogspot.com > > > On Thu, Jan 17, 2013 at 4:09 PM, Mahesh Balija > wrote: > >> I have studied Flume but I didn't find any thing useful in my case. >> My requirement is there is a directory in Windows machine, in which the >> files will be generated and keep updated with new logs. I want to have a >> tail kind of mechanism (using exec source) through which I can push the >> latest updates into the cluster. >> Or I have to simply push once in a day to the cluster using spooling >> directory mechanism. >> >> Can somebody assist whether it is possible using Flume if so the >> configurations needed for this specific to remote windows machine. >> >> But >> >> On Thu, Jan 17, 2013 at 3:48 PM, Mirko K=E4mpf w= rote: >> >>> Give Flume (http://flume.apache.org/) a chance to collect your data. >>> >>> Mirko >>> >>> >>> >>> 2013/1/17 sirenfei >>> >>>> ftp auto upload? >>>> >>>> >>>> 2013/1/17 Mahesh Balija : >>>> > the Hadoop cluster (HDFS) either in synchronous or asynchronou >>>> >>> >>> >> > --e89a8f23471555fab104d37a2548 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable That link talks about just installing Flume on Windows machine (NOT even ha= ve configs to push logs to the Hadoop cluster), but what if I have to colle= ct logs from various clients, then I will endup installing in all clients. =

I have installed Flume successfully on Linux but I have to configure it= such a way that it should gather the log files from the remote windows box= ?

Harsh can you throw some light on this?

On Thu, Jan 17, 2013 at 4:21 PM, Mohammad Tariq <dontariq@gmail.com&g= t; wrote:
Yes. It is possible. I haven't tries windows+flume+had= oop combo
personally, but it should work. You may find this link useful. Alex
has explained beautifully how to run Flume on a windows box.If=A0I=A0<= /div>
get time i'll try to simulate your use case and let you know.=

BTW, could you please share with us whatever you = have tried??



On Thu, Jan 17, 2013 at 4:09 PM, Mahesh = Balija <balijamahesh.mca@gmail.com> wrote:
I have studied Flume but I didn't find any thing useful in my case.
= My requirement is there is a directory in Windows machine, in which the fil= es will be generated and keep updated with new logs. I want to have a tail = kind of mechanism (using exec source) through which I can push the latest u= pdates into the cluster.
Or I have to simply push once in a day to the cluster using spooling direct= ory mechanism.

Can somebody assist whether it is possible using Flum= e if so the configurations needed for this specific to remote windows machi= ne.

But

On Thu, Jan 17, 2013 a= t 3:48 PM, Mirko K=E4mpf <mirko.kaempf@gmail.com> wrote= :
Give Flume (http://f= lume.apache.org/) a chance to collect your data.

Mirko



2013/1/17 sirenfei <sirenxue@gmail= .com>
ftp auto upload?


2013/1/17 Mahesh Balija <balijamahesh.mca@gmail.com>:
> the Hadoop cluster (HDFS) either in synchronous or asynchron= ou




--e89a8f23471555fab104d37a2548--