tajo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Schwabe" <Christian.Schw...@gmx.com>
Subject Re: Cluster setup :: HDFS :: Fully Distributed Mode
Date Fri, 26 Sep 2014 23:01:13 GMT

Hello guys,

It was quite possible two errors in this case. I did not create the Tajo folder and the
parameters tajo.rootdir was not set correctly in tajo-site.xml.
1) Create folder with the following statement: sh hadoop fs -mkdir hdfs://192.168.178.101:54310/tajo
2) Add the correct parameter as following:


I have found the error. The correct parameters would be as follows:
<property>
  <name>tajo.rootdir</name>
  <value>hdfs://192.168.178.101:54310/tajo</value>
</property>


resolves the problem.


While I write this E-Mail 11GB *csv-files were transferred into HDFS. After that I will be
ready to create Tables from HDFS.


I would not like celebrate too soon, but it seems as though at present it would work as it
should. Maaaaaaaaany many thanks to all your friendly help !!! 


Would it make sense, or it is desired that I write an article with my experience in detail
to setup a Hadoop cluster and Tajo runs on it? In this case if you agree can you explain to
me how to get the sourcecode for the whole current documentation?





Warm and best regards,
a tired Chris



P.S.: What an exciting day! Great Feeling. :)


Am 26.09.2014 21:59:49, schrieb Christian Schwabe:

> Hello Hyunsik,
> as always, you are right.
> > I copied the tajo-site.xml also to TajoMaster.
> > All services started correctly.
> > But the tajo-site.xml also contains this param:
> 
> <property>
> >   <name>tajo.rootdir</name>
> >   <value>hdfs://192.168.178.101:50070/tajo</value>
> > </property>
> 
> 
> There seems to be something wrong, related to the following log in TajoMaster-Log.
> > Is that not the correct way to address HDFS-Filesystem?
> 
> I hope this is the last problem to start a new test while Tajo is fully distributed :)
> 


> Thanks for all Hyunsik,
> > Chris
> 
> 
> Am 26.09.2014 21:32:00, schrieb Hyunsik Choi:
> > Hi Chris,
> > 
> > I got the main cause :)
> > 
> > In my view, the result of netstat is very abnormal because your
> > binding address ip is 127.0.0.1. If so, other machines cannot access
> > your TajoMaster.
> > 
> > Your tajo-site looks valid. So, please check two things:
> >  - All machines use tajo-site.xml that you attached to this mailing list.
> >  - Please ensure restarting your cluster after you set the tajo-site.xml.
> > 
> > Probably, you can fix the problem easily if you check above things.
> > 
> > Have a nice weekend.
> > 
> > - Hyunsik
> > 
> > On Fri, Sep 26, 2014 at 12:12 PM, Christian Schwabe
> > <> > Christian.Schwabe@gmx.com> > > wrote:

> > > Hello Hyunsik,
> > > 
> > > TajoMaster on "Master" MacBook seems to run normally. I can't see any
> > > unnormal state.
> > > Attached the log from TajoMaster.
> > > 
> > > 
> > > christians-mbp:bin hduser$ netstat -tn | grep 26003
> > > 
> > > tcp4       0      0  127.0.0.1.26003        127.0.0.1.50262
> > > ESTABLISHED
> > > 
> > > tcp4       0      0  127.0.0.1.50262        127.0.0.1.26003
> > > ESTABLISHED
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Am 26.09.2014 20:22:08, schrieb Hyunsik Choi:
> > > 
> > > I can figure out your environment. :)
> > > 
> > > Could you check TajoMaster log? You need to check if TajoMaster is
> > > running normally.
> > > 
> > > Also, you can ensure that TajoMaster is running by netstat as follows:
> > > 
> > > netstat -tn | grep 26003
> > > 
> > > Thanks,
> > > Hyunsik
> > > 
> > > 
> > > 
> > > On Fri, Sep 26, 2014 at 10:26 AM, Christian Schwabe
> > > <> > > Christian.Schwabe@gmx.comwr> > > ote:
> > > 
> > > 
> > > Hello Hyunsik,
> > > 
> > > no problem. Anyway, thanks for your response.
> > > I have two physical MacBooks here.
> > > Everyone Macbook Pro is configured with the user hduser and the same
> > > password. All Path are the same.
> > > No virtualization for master and slave in use.
> > > 
> > > IP-Address:
> > > -Master :: 192.168.178.101
> > > -Slave :: 192.168.178.39
> > > 
> > > They are currently connected via WLAN but Ethernet is also prepared for
> > > final configuration.
> > > 
> > > Is my configuration in tajo-site.xml need or are these only optional
> > > parameters?
> > > The submitted tajo-site.xml is the config for slave.
> > > 
> > > I don't know exactly what for configuration environments you mean. If any
> > > question is still unanswered feel free to ask.
> > > 
> > > I hope it is still not too late for you and you're reading this e-mail
> > > today.
> > > 
> > > Warm regards,
> > > Chris
> > > 
> > > 
> > > Am 26.09.2014 19:09:19, schrieb Hyunsik Choi:
> > > 
> > > Hi Chris,
> > > 
> > > I'm sorry for late response. It seems to be a network problem because
> > > it's very simple cluster configuration.
> > > 
> > > Q. Could you share your network environment?
> > > 
> > > Q. Is master or slave running on virtual machine?
> > > 
> > > Best regards,
> > > Hyunsik
> > > 
> > > 
> > > On Fri, Sep 26, 2014 at 9:26 AM, Christian Schwabe
> > > <> > > Christian.Schwabe@gmx.comwr> > > ote:
> > > 
> > > 
> > > Hello guys,
> > > 
> > > my current problem is that the Tajo Worker on Slave couldn't connect to
> > > Master.
> > > i need further/more information about
> > > http://tajo.apache.org/docs/current/configuration/cluster_setup.html#settings
> > > What does these parameters mean?
> > > 
> > > I've attached the worker log from slave where only one worker run.
> > > I've attached the tajo-site.xml from slave.
> > > Is there any setting incorrect?
> > > 
> > > IP-Address:
> > > -Master :: 192.168.178.101
> > > -Slave :: 192.168.178.39
> > > 
> > > Hopefully u can help me.
> > > 
> > > Best regards,
> > > Chris
> > > 
> > > 
> > > Am 26.09.2014 12:56:55, schrieb Christian Schwabe:
> > > 
> > > 
> > > 
> > > Hello guys,
> > > 
> > > sorry for spamming.
> > > I found the solution.
> > > The path of home directory for Tajo was not the same on both machines.
> > > When I now start Tajo on master in HA mode the worker on the slave machine
> > > starts, too.
> > > BUT in the webUI for the master i didn't see the second worker alive.
> > > 
> > > Warm regards,
> > > Chris
> > > 
> > > 
> > > 
> > > Am 26.09.2014 12:31:58, schrieb Christian Schwabe:
> > > 
> > > 
> > > Hello guys,
> > > 
> > > next try to get an answer..
> > > Hadoop is still running successfully on both machines master and slave.
> > > Start Tajo in HA mode on master. In webUI i see one Query Master, one Worker
> > > and one Masters. Is that correct so far?
> > > Now i start on slave machine the Tajo worker with 'sh tajo-daemon.sh start
> > > worker' and worker starts successfully. Also the conf/masters and
> > > conf/slaves are created on master machine and slave machine
> > > 
> > > However, I see in the WebUI on master no second worker. What is still wrong?
> > > Thanks for any advice.
> > > 
> > > 
> > > Best regards,
> > > Chris
> > > 
> > > 
> > > 
> > > Am 23.09.2014 22:58:25, schrieb Christian Schwabe:
> > > 
> > > Hello guys,
> > > 
> > > some days later I already made some progress.
> > > HDFS run already successfully in background. I follow these instructions:
> > > http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
> > > 
> > > TajoMasters starts already on HA mode, but how to configure Tajo in detail
> > > to use hdfs? I read this documentation
> > > http://tajo.apache.org/docs/current/configuration/cluster_setup.html,> >
>  but i
> > > don’t know how to configure tajo to store files to hdfs.
> > > Also i don’t know how to setup the second machine to start only a worker
and
> > > connect to master.
> > > I already applied masters and workers file.
> > > 
> > > MacBook1
> > > 
> > > workers content:
> > > 
> > > localhost
> > > 192.168.178.39 // second worker
> > > 
> > > masters content:
> > > 
> > > localhost
> > > 
> > > MacBook2
> > > 
> > > workers content:
> > > 
> > > localhost
> > > 192.168.178.101
> > > 
> > > masters content:
> > > 
> > > 192.168.178.101
> > > 
> > > 
> > > 
> > > Actually I miss a complete guide in the documentation for setup a second
> > > worker to the master. When I realize that with your help, I will post these
> > > instructions and place it to documentation available. I promise!
> > > 
> > > Hopefully you can help me.
> > > 
> > > 
> > > Beat regards,
> > > Chris
> > > 
> > > 
> > > Am 22.09.2014 um 08:46 schrieb Christian Schwabe
> > > <> > > Christian.Schwabe@gmx.com> > > >:
> > > 
> > > 
> > > Hello guys,
> > > 
> > > can someone help to setup a cluster with a second worker?
> > > 
> > > 
> > > Best regards,
> > > Chris
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 





Mime
View raw message