Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of matthieu@actionx.com
 designates 209.85.212.182 as permitted sender)
MIME-Version: 1.0
Date: Wed, 26 Sep 2012 21:04:07 -0400
Message-ID: 
 <CAPxQDUimh62-RY9BZSsn1Lwi3U4xGizgs74KWf10YhBEf3BBxw@mail.gmail.com>
Subject: Advice on Migrating to hadoop + hive
From: Matthieu Labour <matthieu@actionx.com>
To: user <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016e6da98c42e143c04caa48400

--0016e6da98c42e143c04caa48400
Content-Type: text/plain; charset=ISO-8859-1

Hi
I have posted in this user group before and received great help. Thank you!
I am hoping to also get some advice for the following hive/hadoop question:
The way we currently process our log files is the following: we collect log
files. We run a program via cron job that processes/consolidates them and
inserts rows in Postgresql database. Analysts connect to the database,
performs sql queries, generate excel reports. Our logs are growing. The
process of getting the data into the database is getting too slow.
We are thinking leveraging hadoop and my questions are the following.
Should we use hadoop to insert to Postgresql or can we get rid of
Postgresql and rely on Hive only ?
If we use Hive, can we persist the Hive table so we only load the data (run
the hadoop job) one time ?
Can we insert into existing Hive table and add a day of data without the
need to reprocess all previous days files ?
Are there Hive visual tools (Similar to Postgres Maestro) that would make
it easier for the analyst to build/run queries? (Ideally they would need to
work with Amazon EWS)
Thank you for your help
Cheers
Matthieu

--0016e6da98c42e143c04caa48400
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi<div>I have posted in this user group before and received great help. Tha=
nk you! I am hoping to also get some advice for the following hive/hadoop q=
uestion:</div><div>The way we currently process our log files is the follow=
ing: we collect log files. We run a program via cron job that processes/con=
solidates them and inserts rows in Postgresql database. Analysts connect to=
 the database, performs sql queries, generate excel reports. Our logs are g=
rowing. The process of getting the data into the database is getting too sl=
ow.</div>
<div>We are thinking leveraging hadoop and my questions are the following.=
=A0</div><div>Should we use hadoop to insert to Postgresql or can we get ri=
d of Postgresql and rely on Hive only ?</div><div>If we use Hive, can we pe=
rsist the Hive table so we only load the data (run the hadoop job) one time=
 ?</div>
<div>Can we insert into existing Hive table and add a day of data without t=
he need to reprocess all previous days files ?</div><div>Are there Hive vis=
ual tools (Similar to Postgres Maestro) that would make it easier for the a=
nalyst to build/run queries? (Ideally they would need to work with Amazon E=
WS)</div>
<div><div>Thank you for your help</div><div>Cheers</div><div>Matthieu</div>=
<div><br></div>
</div>

--0016e6da98c42e143c04caa48400--