tajo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Schwabe <Christian.Schw...@gmx.com>
Subject Re: starting difficulties
Date Sun, 17 Aug 2014 14:02:40 GMT
Hello Hyunsik,

grateful for any information I have implemented your tips. In plain text, I use hadoop now
in version 2.4.0 
And I've above all on to use  the source files and to compile it using Maven. Previously I
ensure to set variable "Platform" had a value of "x64". 
That you thought I use Java 8 is only due to the fact that java -version shows this version.
In fact, however, I have linked in the tajo-env.sh the JDK version 1.7.0_67.
Unfortunately still nothing works. The same error with the ArrayIndexOutOfException. I'm sorry
for the conditions that I prepare.
Unfortunately I do not really know what I'm doing wrong yet.

Best regards,
Chris

Am 17.08.2014 um 09:24 schrieb Hyunsik Choi <hyunsik@apache.org>:

> Hi Chris,
> 
> I'm not sure if your problem is caused by Hadoop 3.0 version. But, interestingly, I haven't
experienced such a problem so far when I deployed Tajo in various systems with Hadoop and
JVM versions required by our installation guide.
> 
> BTW, Hadoop 3.0 is too far away from the stable Hadoop. Hadoop 3.0 is neither alpha or
beta, and it is just development version. So, we do not guarantee that Tajo will work in Hadoop
3.0. In addition, as far as I know, you use Java 1.8. We do not guarantee Tajo works in Java
1.8 without any problem. In addition, currently, all hadoop distributions supports 1.6 and
1.7, and some distributions only support 1.7. 
> 
> I hope my answer is helpful for your work.
> 
> Best regards,
> Hyunsik
> 
> 
> On Sat, Aug 16, 2014 at 12:00 AM, Christian Schwabe <Christian.Schwabe@gmx.com>
wrote:
> Hello together,
> 
> yesterday i pulled the new status from tajo github.
> Recompiled the sources with the known build-statement: mvn clean package -DskipTests
-Pdist -Dtar
> First, however, I still ensures that the variable Platform has the value of x64
> Since I made this step i have the same error for the first day i tried Apache Tajo.
> I have only pulled Tajo, recompiled.
> 
> I don’t believe that this message () is a new issue 
> 
> I don't think this is a new error (http://mail-archives.apache.org/mod_mbox/tajo-user/201408.mbox/%3cB99E4FEE-997C-4B4C-BEDD-C4618244BB90@gmx.com%3e),
but rather resulted from me at that time not yet noticed the COUNT (*) did not work. Otherwise
I would have surely noted earlier that something else is wrong here.
> 
> I mean so far I've been able to gain some experience. Therefore, I can not understand
what could have gone wrong now with these few steps currently. It is a reason for never change
a running system ... :(
> 
> I have attached the logs from worker and master. 
> 
> P.S.: I use Hadoop in Version 3.0.0-Snapshot. Previously, there were no problems with
this version of Hadoop. 
> 
> default> SELECT * FROM table1;
> id,  name,  score,  type
> -------------------------------
> 1,  abc,  1.1,  a
> 2,  def,  2.3,  b
> 3,  ghi,  3.4,  c
> 4,  jkl,  4.5,  d
> 5,  mno,  5.6,  e
> (5 rows, 0.229 sec, 59 B selected)
> 
> default> SELECT COUNT(*) FROM table1;
> Progress: 0%, response time: 1.224 sec
> Progress: 50%, response time: 2.225 sec
> Progress: 50%, response time: 3.227 sec
> Progress: 50%, response time: 4.229 sec
> ERROR: 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> 	at org.apache.tajo.util.BitArray.fromByteBuffer(BitArray.java:84)
> 	at org.apache.tajo.storage.RawFile$RawFileScanner.next(RawFile.java:243)
> 	at org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:254)
> 	at org.apache.tajo.engine.planner.physical.HashAggregateExec.compute(HashAggregateExec.java:51)
> 	at org.apache.tajo.engine.planner.physical.HashAggregateExec.next(HashAggregateExec.java:87)
> 	at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
> 	at org.apache.tajo.worker.Task.run(Task.java:425)
> 	at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:425)
> 	at java.lang.Thread.run(Thread.java:744)
> 
> 
> 
> Best regards,
> a sad Chris
> 
> Am 28.07.2014 um 04:55 schrieb Jose Alvarez Muguerza <jose.muguerza@globant.com>:
> 
>> Hi Christian, 
>> An alternative, please try recompiling your ASF hadoop distribution with "+ export
Platform=x64"  parameter since your platform is 64-Bit 
>> 
>> Good luck 
>> 
>> 
>> 
>> On Sunday, July 27, 2014, Christian Schwabe <Christian.Schwabe@gmx.com> wrote:
>> Hello Hyunsik,
>> 
>> Please excuse my incomplete answer. I forgot to mention that also the recompile and
leads to the already mentioned java.lang.ArrayIndexOutOfBoundsException.
>> 
>> Kind regards,
>> Christian
>> 
>> Am 27.07.2014 um 21:46 schrieb Christian Schwabe <Christian.Schwabe@gmx.com>:
>> 
>>> Hello Hyunsik, today I have started the trial to recompile with following 'mvn
clean package -DskipTests -Pdist -Dtar -Dhadoop.version=2.4.0' statement. I regret to inform
you that the recompile not brought the desired success.
>>> 
>>> I use Hadoop in version 2.4.0. 
>>> 
>>> This is my current JVM version:
>>> java version "1.8.0_05"
>>> Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
>>> Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
>>> 
>>> My Tajo environment (tajo-env.sh) has following content:
>>> 
>>> # Licensed to the Apache Software Foundation (ASF) under one
>>> # or more contributor license agreements.  See the NOTICE file
>>> # distributed with this work for additional information
>>> # regarding copyright ownership.  The ASF licenses this file
>>> # to you under the Apache License, Version 2.0 (the
>>> # "License"); you may not use this file except in compliance
>>> # with the License.  You may obtain a copy of the License at
>>> #
>>> #     http://www.apache.org/licenses/LICENSE-2.0
>>> #
>>> # Unless required by applicable law or agreed to in writing, software
>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>> # See the License for the specific language governing permissions and
>>> # limitations under the License.
>>> 
>>> # Set Tajo-specific environment variables here.
>>> 
>>> # The only required environment variable is JAVA_HOME.  All others are
>>> # optional.  When running a distributed configuration it is best to
>>> # set JAVA_HOME in this file, so that it is correctly defined on
>>> # remote nodes.
>>> 
>>> # —————————————————————————————
>>> # Hadoop home. Required
>>> #export HADOOP_HOME=/home/christian/6_Semester/Thesis/02_Hadoop/hadoop-2.4.0
>>> 
>>> #Mac OS X Konfiguration
>>> export HADOOP_HOME=/Users/chris/6_Semester/Thesis/02_Hadoop/hadoop-2.4.0
>>> # —————————————————————————————
>>> 
>>> # The java implementation to use.  Required.
>>> # export JAVA_HOME=/usr/java/default
>>> #export JAVA_HOME=/usr
>>> #Mac OS X Konfiguration
>>> export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
>>> 
>>> # Extra Java CLASSPATH elements.  Optional.
>>> # export TAJO_CLASSPATH=/xxx/extlib/*:/xxx/xxx.jar
>>> 
>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>> # export TAJO_MASTER_HEAPSIZE=1000
>>> 
>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>> # export TAJO_WORKER_HEAPSIZE=1000
>>> 
>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>> # export TAJO_QUERYMASTER_HEAPSIZE=1000
>>> 
>>> # Extra Java runtime options.  Empty by default.
>>> # export TAJO_OPTS=-server
>>> 
>>> # Extra TajoMaster's java runtime options for TajoMaster. Empty by default
>>> # export TAJO_MASTER_OPTS=
>>> 
>>> # Extra TajoWorker's java runtime options for TajoMaster. Empty by default
>>> # export TAJO_WORKER_OPTS=
>>> 
>>> # Extra  QueryMaster mode TajoWorker's java runtime options for TajoMaster. Empty
by default
>>> # export TAJO_QUERYMASTER_OPTS=
>>> 
>>> # Where log files are stored.  $TAJO_HOME/logs by default.
>>> # export TAJO_LOG_DIR=${TAJO_HOME}/logs
>>> 
>>> # The directory where pid files are stored. /tmp by default.
>>> # export TAJO_PID_DIR=/var/tajo/pids
>>> 
>>> # A string representing this instance of tajo. $USER by default.
>>> # export TAJO_IDENT_STRING=$USER
>>> 
>>> # The scheduling priority for daemon processes.  See 'man nice'.
>>> # export TAJO_NICENESS=10
>>> 
>>> # Tajo cluster mode. the default mode is standby mode.
>>> export TAJO_WORKER_STANDBY_MODE=true
>>> 
>>> # It must be required to use HCatalogStore
>>> # export HIVE_HOME=
>>> # export HIVE_JDBC_DRIVER_DIR=
>>> 
>>> Findest du da einen Fehler in der Konfiguration oder hast du noch einen Tip was
genau falsch sein könnte? Ich kann in der nächsten Woche leider erst einmal nicht antworten,
werde mich sobald ich wieder da bin wieder melden.
>>> 
>>> Am 27.07.2014 um 06:06 schrieb Hyunsik Choi <hyunsik@apache.org>:
>>> 
>>>> Hi Christian,
>>>> 
>>>> In order to clarify your problem, could you share your environments,
>>>> such as Hadoop version and JVM version? For some hadoop version, you
>>>> need to give '-Dhadoop.version=2.4.0' to maven build command.
>>>> Otherwise, incorrect class sources may be built for your binary dist.
>>>> 
>>>> Best regards,
>>>> Hyunsik
>>>> 
>>>> On Sat, Jul 26, 2014 at 6:11 AM, Christian Schwabe
>>>> <Christian.Schwabe@gmx.com> wrote:
>>>>> Hello Hyunsik,
>>>>> 
>>>>> thanks for your quick response.
>>>>> I clone the actual github repository and compile the Version by my own.
>>>>> So actual I use the version 0.9.0-Snapshot.
>>>>> After your answer it seems als the bugs exist in the current version?
Can
>>>>> you confirm this?
>>>>> 
>>>>> Kind regards,
>>>>> Chris
>>>>> 
>>>>> On 25. Juli 2014 18:39:37 MESZ, Hyunsik Choi <hyunsik@apache.org>
wrote:
>>>>>> 
>>>>>> Hi Christian,
>>>>>> 
>>>>>> Above all, thank you for your interest in Tajo.
>>>>>> 
>>>>>> The problem seems to be an old bug.
>>>>>> Are you using Tajo 0.9.0? Did you get Tajo via git?
>>>>>> 
>>>>>> Best regards,
>>>>>> Hyunsik
>>>>>> 
>>>>>> On Fri, Jul 25, 2014 at 9:23 PM, Christian Schwabe
>>>>>> <Christian.Schwabe@gmx.com> wrote:
>>>>>>> 
>>>>>>> Hello guys,
>>>>>>> 
>>>>>>> this is the first time for me to write to an email list. Sorry
for any
>>>>>>> wrong
>>>>>>> formatting.
>>>>>>> I am a student of computer science and use the framework Apache
Tajo as
>>>>>>> part
>>>>>>> of my thesis. However, I have initial teething problems and
>>>>>>> unfortunately do
>>>>>>> not know how to remedy them.
>>>>>>> 
>>>>>>> First, I want to describe what steps do I do to start Tajo:
>>>>>>> 1) Start Tajo with tajo-start.sh
>>>>>>> 2) Create a new Table
>>>>>>> 3) Start bash tsql
>>>>>>> 
>>>>>>> Here details to my table contents which is composed from a csv
file:
>>>>>>> 
>>>>>>> default> SELECT * from table1;
>>>>>>> id,  name,  score,  type
>>>>>>> ________________________________
>>>>>>> 
>>>>>>> 1,  abc,  1.1,  a
>>>>>>> 2,  def,  2.3,  b
>>>>>>> 3,  ghi,  3.4,  c
>>>>>>> 4,  jkl,  4.5,  d
>>>>>>> 5,  mno,  5.6,  e
>>>>>>> (5 rows, 0.275 sec, 59 B selected)
>>>>>>> 
>>>>>>> Everything seems to work up here. Also substr () or concat ()
function.
>>>>>>> Do I
>>>>>>> use now but the count (*) or sum () I get the following exception:
>>>>>>> 
>>>>>>> default> SELECT count(*) FROM table1;
>>>>>>> ERROR: 1
>>>>>>> java.lang.ArrayIndexOutOfBoundsException: 1
>>>>>>> at org.apache.tajo.util.BitArray.fromByteBuffer(BitArray.java:84)
>>>>>>> at org.apache.tajo.storage.RawFile$RawFileScanner.next(RawFile.java:243)
>>>>>>> at
>>>>>>> 
>>>>>>> org.apache.tajo.engine.planner.physical.SeqScanExec.next(SeqScanExec.java:254)
>>>>>>> at
>>>>>>> 
>>>>>>> org.apache.tajo.engine.planner.physical.HashAggregateExec.compute(HashAggregateExec.java:51)
>>>>>>> at
>>>>>>> 
>>>>>>> org.apache.tajo.engine.planner.physical.HashAggregateExec.next(HashAggregateExec.java:87)
>>>>>>> at
>>>>>>> 
>>>>>>> org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
>>>>>>> at org.apache.tajo.worker.Task.run(Task.java:425)
>>>>>>> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:406)
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>> 
>>>>>>> 
>>>>>>> Can you tell me what I'm doing wrong at this point that I have
neither
>>>>>>> the
>>>>>>> count (*) still can use sum ()?
>>>>>>> Is this a known problem? Is it fixed? When will it be fixed?
How can I
>>>>>>> fix
>>>>>>> it myself? I suppose that I have a wrong configuration, although
I have
>>>>>>> not
>>>>>>> had much konfikuriert. Because I can hardly imagine that such
trivial
>>>>>>> functionalities not in my used version 0.9.0 are available.
>>>>>>> 
>>>>>>> Kind regards,
>>>>>>> Christian Schwabe
>>> 
>> 
>> 
>> 
>> -- 
>> 
>> Jose Alvarez Muguerza | Architect 
>> GLOBANT | AR: +54 11 4109 1700 ext. 15895 | US: +1 877 215 5230 ext. 15895 |
>> 					
>>  
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>  Please consider the environment before printing this email
>> 
> 
> 
> 


Mime
View raw message