hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vasilis Liaskovitis (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-396) Hive performance benchmarks
Date Wed, 27 Jan 2010 17:10:36 GMT

    [ https://issues.apache.org/jira/browse/HIVE-396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805574#action_12805574
] 

Vasilis Liaskovitis commented on HIVE-396:
------------------------------------------

I am trying to run the hive benchmarks on a small 8-node hadoop cluster, starting from the
2009-07-21 tarball. Iam using hadoop-0.20.1 and hive-0.4.1. I have a few questions, any help
is welcome

- datasize: the report states 60GB of data used for the uservisitis table. The default datagen/htmlgen/config.txt
uses ~15GB of Uservisits tables per node. I believe the way the generatedata.py script runs,
we get one rankings/ uservisits table per node i.e. on an 10 data nodes we 'd get 10tablesx15GB=150GB
of data. Do the benchmarks in the report use all of the generated data i.e. UserVistis_0,
UserVists_1,..., UserVistis_10 across the 10 data nodes? Or a subset of these? Same question
for the rankings table.
  
- I assume that the hadoop-site.xml and hive-site.xml files under confs/ (from the 2009-07-21
tarball) accurately reflect the hadoop/hive configuration used to produce the results in the
report. Let me know if that's not the case, I ve been using these configs to run the benchmarks.

- Regarding query4 (join) specifically ,  the reduce phase of the first job (main job with
) shows a strange behaviour. Looking at the jobtracker GUI, out of 60 reducers, 59 reducers
finish relatively fast (~1 min), but one reducer  task takes much longer (~19min). Does that
point to a specific hadoop/ hive misconfiguration? No other hadoop job that I 've run exhibits
this. Is this intuitive behaviour given a join operation?


> Hive performance benchmarks
> ---------------------------
>
>                 Key: HIVE-396
>                 URL: https://issues.apache.org/jira/browse/HIVE-396
>             Project: Hadoop Hive
>          Issue Type: New Feature
>            Reporter: Zheng Shao
>            Assignee: Yuntao Jia
>         Attachments: AlansMRcode.tgz, hive_benchmark_2009-06-18.pdf, hive_benchmark_2009-06-18.tar.gz,
hive_benchmark_2009-07-12.pdf, hive_benchmark_2009-07-21.tar.gz
>
>
> We need some performance benchmark to measure and track the performance improvements
of Hive.
> Some references:
> PIG performance benchmarks PIG-200
> PigMix: http://wiki.apache.org/pig/PigMix

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message