bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Núñez <>
Subject Re: Hue Using MR1?
Date Fri, 06 Dec 2013 12:28:26 GMT
I guess the last thing I would add is a bit of QA around the init scripts
and other installed items. They seem to be frayed around the edges.

On 2013-12-6 18:42 , "Bruno Mahé" <> wrote:

>See inline.
>On 12/06/2013 01:02 AM, Steven Núñez wrote:
>> Gents,
>> I¹m starting an install from BigTop 0.70: HDFS, YARN and Hue, with the
>> goal of building up from there to a minimal stack. In theory, this
>> should be as simple as Œyum install hadoop\* hue\*¹; in practice this
>> turns out to be surprisingly broken. For example, after the yum install
>> hue is reporting a configuration error:
>>     hadoop.mapred_clusters.default.hadoop_mapred_home Current value:
>>     /usr/lib/hadoop-0.20-mapreduce
>>     Path does not exist on the filesystem.
>> Nowhere is this being set from the files that BigTop installed. Why is
>> Hue looking for MR1 stuff?
>> Attempting to fix another misconfiguration reported by hue:
>>     hadoop.hdfs_clusters.default.webhdfs_url Current value: None
>>     Failed to access filesystem root
>Packages do not configure services for you. Not because it is broken,
>but because it is not their responsibility.
>In order to configure services correctly, packages would have to know
>about the cluster topology and probably a few more things.
>The closest thing you could find to that are the "pseudo conf"
>configurations which assume everything is run locally.
>In your example, you are just looking at the default Hue's configuration.
>If you want to deploy Apache Bigtop on multiple nodes, I would recommend
>you to take a look at the puppet recipes, which do configure services.
>Even if you do not use puppet, you can still look at the configuration
>templates to help you derive one for your cluster.
>Would that be helpful to you if more packages were to provide "pseudo
>conf" configurations where everything is assumed to run locally?
>> Is simple, however when restarting the daemons with: "for i in
>> hadoop-hdfs-namenode hadoop-hdfs-datanode ; do service $i restart ;
>> done", both daemons fail to restart (say, what happened to
>> Extracts from the logs show the reasons:
>>   * Datanode Problem binding to
>>     [] Address already in use
>>   * Namenode Cannot lock storage
>>     /var/lib/hadoop-hdfs/cache/hdfs/dfs/name. The directory is already
>>     locked
>> So what I¹ve done is a fresh installed, modified two config files, as
>> described in the hue configuration section
>> <>, and tried
>> to restart the HDFS daemons and they¹re failing. This probably isn¹t
>> what I¹d call Œworking¹.
> is not meant to be used with packages. It would not work.
>Can you describe *exactly* what you did from the beginning along with
>the output?
>The easier it is to reproduce, the easier it will be to fix/help you :)
>> I know that BigTop is at 0.70, but then again version numbers don¹t mean
>> much these days. Is this kind of error to be expected after a fresh
>> install? When will it be safe to assume that a simple stack (HDFS, YARN,
>> Hue, Oozie, Zookeeper) installation would be working Œout of the box¹
>> without loads of manual configuration?
>> Perhaps I¹m missing the point (I¹m definitely missing the
>> documentation), but at this stage it seems BigTop¹s primary advantage is
>> to ensure that the collection of packages is version compatible. Would
>> that be fair to say? I¹m not unappreciative of the value there, but just
>> want to set expectations with people on what they¹re getting with BigTop
>> in its current state. I recall some discussion of configuration before,
>> and there were somewhat different opinions. One that I recall was (my
>> words, summarizing my understanding):
>>     BigTop doesn¹t do any configuration, that¹s all left to the
>>     individual packages; BigTop just places them on the filesystem in
>>     (somewhat, there¹s still .cfg, .ini, .conf files) a consistent
>> Would that be a fair statement?
>> Cheers,
>> - Steve
>The packages do not set the configuration. But the Apache Bigtop puppet
>recipes will set the configuration.
>Packages will, as you describe, put files on the filesystem in a
>consistent manner but also create users, and other tasks to provide a
>rich integration with the system (init scripts, ulimit, etc.).
>Most devs on this list probably either use the puppet recipes or have a
>set of config files ready to be reused for their different installations.
>So we do not always encounter the same issues as you do from a fresh
>A hackathon is coming up this week end and I would like to tackle some
>tasks to make Apache Bigtop more user friendly. So besides some more
>consistent documentation and pseudo conf packages, would there be
>anything else you would like to see happening?

View raw message