tajo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Schwabe <Christian.Schw...@gmx.com>
Subject Re: Big big query
Date Thu, 18 Sep 2014 05:00:39 GMT
Hello Hyunsik,

thanks anyway. One question is still unanswered. How To configure Tajo for parallel processing?
I already describe in the post before this one. 

Best regards,
Chris

On 18. September 2014 06:42:51 MESZ, Hyunsik Choi <hyunsik@apache.org> wrote:
>iimissed one thing. I'd like to reserve your dinner when I visit German
>someday :p
>
>Anyway, thank you for your interest in Tajo. Please keep going your
>trials
>with Tajo.
>
>Thanks!
>
>2014년 9월 18일 목요일, Christian Schwabe<Christian.Schwabe@gmx.com>님이
작성한
>메시지:
>
>>
>> Hello together,
>>
>> I am speechless. The query has been successfully completed. Running
>time: 30
>> minutes.
>>
>> I've done the following:
>> git pull
>> git clean -d -f
>> git reset --hard
>> patch -p0 < TAJO-1049_3.patch
>>
>> You are great. I'm very thankful.
>> If you would live in Germany, I would invite you to dinner, all of
>your
>> team :-D
>>
>> It is now important for me to understand yet what is there now
>happens in
>> the background. In https://issues.apache.org/jira/browse/TAJO-1048
>Fixed
>> the bug that the session variables have some parameters are not
>> overwritten at run time, right?
>> Difficult with the understanding it is at
>> https://issues.apache.org/jira/browse/TAJO-1049th What is the
>limiting
>> factor you have fixed it, which now apparently had this effect?
>>
>> In the just traversed test I noticed that only one task is processed.
>I
>> started just another run with the following configuration in
>tajo-site.xml
>> to use multiple cores:
>>
>> <configuration>
>>     <property>
>>       <name>tajo.resource-tracker.heartbeat.timeout-secs</name>
>>       <value>240000</value>
>>     </property>
>>     <property>
>>      
><name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
>>       <value>4</value>
>>     </property>
>> </configuration>
>>
>> But the tasks are apparently still only processed serially by one
>thread.
>> Is this still an incorrect configuration due? Before the error was
>not
>> fixed, I had made ​​the same settings and determines the tasks were
>> distributed on three cores with four selected cores.
>>
>> I would like to highlight particularly once: I am very grateful to
>you. I
>> very much hope that other interested persons in Tajo from our
>discussions
>> and the experience gained in this and also can learn from it.
>>
>> Warm regards and a thankful,
>> Chris
>>
>> Am 17.09.2014 20:04:21, schrieb Hyunsik Choi:
>>
>> You don't need to apply TAJO-983.140916.patch. As I mentioned, this
>> patch is still ongoing. You only need TAJO-1049_3.patch.
>>
>> Thanks,
>> Hyunsik
>>
>>
>> On Thu, Sep 18, 2014 at 2:45 AM, Christian Schwabe
>> <Christian.Schwabe@gmx.com <#14885373167cb354_>> wrote:
>>
>> Hello Hyunsik,
>>
>> Done. Same errors.
>> No compilation possible.
>>
>> Warm regards,
>> Chris
>>
>>
>> Am 17.09.2014 um 19:05 schrieb Hyunsik Choi <hyunsik@apache.org
>> <#14885373167cb354_>>:
>>
>> Did you update and clean your git repository? In order to apply the
>> submitted patches without conflict, you should use the latest and
>> clean revision.
>>
>> In order to clean and update your revision, please type the following
>> commands:
>>
>> git clean -d -f
>> git reset --hard
>> patch -p0 < filename.patch
>>
>>
>> Best regards,
>> Hyunsik
>>
>> On Thu, Sep 18, 2014 at 1:57 AM, Christian Schwabe
>> <Christian.Schwabe@gmx.com <#14885373167cb354_>> wrote:
>>
>> Hello Hyunsik,
>>
>> i read the bugs to this patch under the review.
>> What i’ve done:
>> -pull to the latest review
>> -I applied the following patches:
>>
>> patch -p0 <TAJO-1049_3.patch
>> patch -p0 <TAJO-983.140916.patch
>>
>> get the following output:
>>
>>
>> christians-mbp:tajo chris$ patch -p0 <TAJO-983.140916.patch
>> patching file
>tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
>> Hunk #2 succeeded at 52 (offset -1 lines).
>> Hunk #3 FAILED at 68.
>> 1 out of 6 hunks FAILED -- saving rejects to file
>> tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java.rej
>> patching file
>tajo-core/src/main/java/org/apache/tajo/worker/Task.java
>> Hunk #1 FAILED at 48.
>> Hunk #2 succeeded at 97 (offset 1 line).
>> Hunk #3 succeeded at 197 (offset 1 line).
>> Hunk #4 succeeded at 594 (offset 1 line).
>> Hunk #5 succeeded at 639 (offset 1 line).
>> Hunk #6 FAILED at 699.
>> Hunk #7 succeeded at 754 (offset 2 lines).
>> 2 out of 7 hunks FAILED -- saving rejects to file
>> tajo-core/src/main/java/org/apache/tajo/worker/Task.java.rej
>> patching file
>>
>>
>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
>> patching file
>> tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
>> Hunk #2 FAILED at 94.
>> Hunk #3 FAILED at 140.
>> Hunk #4 FAILED at 170.
>> Hunk #5 FAILED at 204.
>> Hunk #6 FAILED at 220.
>> 5 out of 6 hunks FAILED -- saving rejects to file
>> tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java.rej
>> patching file
>>
>>
>tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
>> patching file
>>
>>
>tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
>> christians-mbp:tajo chris$ patch -p0 <TAJO-1049_3.patch
>> patching file
>>
>tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
>>
>> Patchs seems to be applied correctly, despite the error.
>>
>> recompiled code with mvn clean package -DskipTests -Pdist -Dtar
>> -Dhadoop.version=2.4.0 and get the following output — see attached
>log from
>> maven. Build process crashed.
>>
>> There seems to be something wrong. I hope i’ve done all right.
>>
>>
>>
>>
>>
>> Warm regards,
>> Chris
>>
>>
>>
>>
>> Am 17.09.2014 um 16:39 schrieb Hyunsik Choi <hyunsik@apache.org
>> <#14885373167cb354_>>:
>>
>> Hi Chris,
>>
>> Do you mean that your query is finished well? Good to hear. But, you
>> need to know that the patch still has some bugs. Please see the
>review
>> https://reviews.apache.org/r/25242/.
>>
>> If your problem is solved by this patch, your problem was related to
>> Fetcher. I'm also expecting your problem is related to this issue
>> (https://github.com/apache/tajo/pull/141/files), which already
>> committed to master branch. You can use this fix from master branch
>> too.
>>
>> Please refer to this document in order to learn how to apply patch.
>We
>> mostly use '--no-prefix' option in order to generate patch files. So,
>> you should use the patch command with -p0 instead of -p1 as follows:
>>
>> patch -p0 < ${PATCH_FILE}.patch
>>
>> By default, patch command uses -p1.
>>
>> Thanks,
>> Hyunsik
>>
>> On Wed, Sep 17, 2014 at 8:07 AM, Christian Schwabe
>> <Christian.Schwabe@gmx.com <#14885373167cb354_>> wrote:
>>
>> Hello guys,
>>
>> after days of efforts to apply the patch from May Hai, I finally be
>able to
>> announce a success today. I want to a brief description where the
>problem
>> was. The few guides out there to apply a patch, using the Git clients
>often
>> use git apply. However, I had no success. See discussion below.
>>
>> christians-mbp:tajo chris$ git apply --check
>> /Users/chris/Desktop/TAJO-983.140916.patch
>> error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such
>file or
>> directory
>> error: src/main/java/org/apache/tajo/worker/Task.java: No such file
>or
>> directory
>> error:
>>
>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java:
>> No
>> such file or directory
>> error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such
>file
>> or directory
>> error:
>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java:
>> No such file or directory
>> error:
>src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java:
>> No
>> such file or directory
>>
>> ###########################################
>>
>> christians-mbp:tajo chris$ git apply --check
>> /Users/chris/Desktop/TAJO-983.140916.patch
>> error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such
>file or
>> directory
>> error: src/main/java/org/apache/tajo/worker/Task.java: No such file
>or
>> directory
>> error:
>>
>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java:
>> No
>> such file or directory
>> error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such
>file
>> or directory
>> error:
>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java:
>> No such file or directory
>> error:
>src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java:
>> No
>> such file or directory
>>
>> ###########################################
>>
>> christians-mbp:tajo chris$ git apply -v --check
>> /Users/chris/Desktop/TAJO-983.140916.patch
>> Checking patch src/main/java/org/apache/tajo/worker/Fetcher.java...
>> error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such
>file or
>> directory
>> Checking patch src/main/java/org/apache/tajo/worker/Task.java...
>> error: src/main/java/org/apache/tajo/worker/Task.java: No such file
>or
>> directory
>> Checking patch
>>
>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java...
>> error:
>>
>src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java:
>> No
>> such file or directory
>> Checking patch
>src/test/java/org/apache/tajo/worker/TestFetcher.java...
>> error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such
>file
>> or directory
>> Checking patch
>>
>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java...
>> error:
>src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java:
>> No such file or directory
>> Checking patch
>> src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java...
>> error:
>src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java:
>> No
>> such file or directory
>>
>> ###########################################
>>
>> christians-mbp:tajo-core chris$ git apply
>> /Users/chris/Desktop/TAJO-983.140916.patch
>> /Users/chris/Desktop/TAJO-983.140916.patch:58: trailing whitespace.
>>
>>
>>
>> /Users/chris/Desktop/TAJO-983.140916.patch:80: trailing whitespace.
>>
>>
>>
>> /Users/chris/Desktop/TAJO-983.140916.patch:487: trailing whitespace.
>> return " (start=" + startOffset() + ", length=" + length + ",
>> fromRemote=" + fromRemote
>> warning: 3 lines add whitespace errors.
>>
>> ###########################################
>>
>> Removal of whitespaces also brought no success.
>> My hope depended least on this patch. As far Tajo not run in my
>environment
>> and my standard configuration, I will unfortunately have to complete
>my
>> thesis with the result that it is currently not suitable for
>practical use.
>> This really is not a reproach to formulate that clear again. I really
>do
>> appreciate your hard work very much. However, I also think it's
>slightly
>> doubtful that the advice you have given can't read anywhere and only
>> inquired. I think that's still a pity. Surely you can not really
>expect, or
>> rather the can but not really your goal to be always be the first
>point of
>> contact. I think at this point there is still a lot of catching up to
>do.
>> This is still a large deficit. Surely you can not really expect, or
>rather
>> it is hopefully not really your goal to be always be the first point
>of
>> contact. I think at this point there is still a lot of catching up to
>do.
>>
>> I am not come within the scope of my thesis within the time to the
>desired
>> goal, but of course I'm still interested in a close contact with you
>and
>> and
>> a further effort to solve the outstanding problem.
>>
>> Until I have encountered the following with the patch was
>successfully
>> carried out. Unfortunately, this patch did not bring the hoped
>solution. No
>> change in the behavior described herein. Are there any other ideas
>that I
>> could try to fix the problem?
>>
>> I really thank you very much for your support thus far. I really
>thank you
>> very much for your support thus far. I really thank you very much for
>your
>> support thus far. I really thank you very much for your support thus
>far. I
>> really thank you very much for your support thus far. I really thank
>you
>> very much for your support thus far. I really thank you very much for
>your
>> support thus far. I really thank you very much for your support thus
>far.
>> ;-)
>>
>>
>> christians-mbp:tajo chris$ cd tajo-yarn-pullserver/
>> christians-mbp:tajo-yarn-pullserver chris$ patch -p1 <
>> ../TAJO-983.140916.patch
>> can't find file to patch at input line 5
>> Perhaps you used the wrong -p or --strip option?
>> The text leading up to this was:
>> --------------------------
>> |diff --git
>tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
>> tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
>> |index 64475fe..7310736 100644
>> |--- tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
>> |+++ tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
>> --------------------------
>> File to patch:
>> Skip this patch? [y] y
>> Skipping patch.
>> 6 out of 6 hunks ignored
>> can't find file to patch at input line 98
>> Perhaps you used the wrong -p or --strip option?
>> The text leading up to this was:
>> --------------------------
>> |diff --git tajo-core/src/main/java/org/apache/tajo/worker/Task.java
>> tajo-core/src/main/java/org/apache/tajo/worker/Task.java
>> |index 7b4cbe1..ecc4078 100644
>> |--- tajo-core/src/main/java/org/apache/tajo/worker/Task.java
>> |+++ tajo-core/src/main/java/org/apache/tajo/worker/Task.java
>> --------------------------
>> File to patch: y
>> y: No such file or directory
>> Skip this patch? [y] y
>> Skipping patch.
>> 7 out of 7 hunks ignored
>> can't find file to patch at input line 339
>> Perhaps you used the wrong -p or --strip option?
>> The text leading up to this was:
>> --------------------------
>> |diff --git
>>
>>
>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
>>
>>
>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
>> |index 4f11168..911c0c2 100644
>> |---
>>
>>
>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
>> |+++
>>
>>
>tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
>> --------------------------
>> File to patch: y
>> y: No such file or directory
>> Skip this patch? [y] y
>> Skipping patch.
>> 1 out of 1 hunk ignored
>> can't find file to patch at input line 362
>> Perhaps you used the wrong -p or --strip option?
>> The text leading up to this was:
>> --------------------------
>> |diff --git
>tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
>> tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
>> |index b755e02..cb19d54 100644
>> |--- tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
>> |+++ tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
>> --------------------------
>> File to patch: y
>> y: No such file or directory
>> Skip this patch? [y] y
>> Skipping patch.
>> 6 out of 6 hunks ignored
>> patching file
>> src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
>> patching file
>> src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
>> christians-mbp:tajo-yarn-pullserver chris$ pwd
>> /Users/chris/6_Semester/Thesis/07_git/tajo/tajo-yarn-pullserver
>>
>>
>> Warm regards,
>> Chris
>>
>>
>>
>>
>>
>>

Mime
View raw message