tajo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Schwabe" <Christian.Schw...@gmx.com>
Subject Re: Big big query
Date Wed, 17 Sep 2014 20:03:52 GMT

Hello together,

I am speechless. The query has been successfully completed. Running time: 30 minutes.

I've done the following:
git pull
git clean -d -f
git reset --hard
patch -p0 < TAJO-1049_3.patch

You are great. I'm very thankful. 
If you would live in Germany, I would invite you to dinner, all of your team :-D

It is now important for me to understand yet what is there now happens in the background.
In https://issues.apache.org/jira/browse/TAJO-1048 Fixed the bug that the session variables
have some parameters are not overwritten at run time, right? 
Difficult with the understanding it is at https://issues.apache.org/jira/browse/TAJO-1049th
What is the limiting factor you have fixed it, which now apparently had this effect?

In the just traversed test I noticed that only one task is processed. I started just another
run with the following configuration in tajo-site.xml to use multiple cores:

<configuration>
    <property>
      <name>tajo.resource-tracker.heartbeat.timeout-secs</name>
      <value>240000</value>
    </property>
    <property>
      <name>tajo.worker.resource.tajo.worker.resource.cpu-cores</name>
      <value>4</value>
    </property>
</configuration>

But the tasks are apparently still only processed serially by one thread. 




   Is this still an incorrect configuration due? Before the error was not fixed, I had made
​​the same settings and determines the tasks were distributed on three cores with four
selected cores.


I would like to highlight particularly once: I am very grateful to you. I very much hope that
other interested persons in Tajo from our discussions and the experience gained in this and
also can learn from it.

Warm regards and a thankful,
Chris

Am 17.09.2014 20:04:21, schrieb Hyunsik Choi:
> You don't need to apply TAJO-983.140916.patch. As I mentioned, this
> patch is still ongoing. You only need TAJO-1049_3.patch.
> 
> Thanks,
> Hyunsik
> 
> 
> On Thu, Sep 18, 2014 at 2:45 AM, Christian Schwabe
> <> Christian.Schwabe@gmx.com> > wrote:
> > Hello Hyunsik,
> > 
> > Done. Same errors.
> > No compilation possible.
> > 
> > Warm regards,
> > Chris
> > 
> > 
> > Am 17.09.2014 um 19:05 schrieb Hyunsik Choi <> > hyunsik@apache.org>
> >:
> > 
> > > Did you update and clean your git repository? In order to apply the
> > > submitted patches without conflict, you should use the latest and
> > > clean revision.
> > > 
> > > In order to clean and update your revision, please type the following commands:
> > > 
> > > > git clean -d -f
> > > > git reset --hard
> > > > patch -p0 < filename.patch

> > > Best regards,
> > > Hyunsik
> > > 
> > > On Thu, Sep 18, 2014 at 1:57 AM, Christian Schwabe
> > > <> > > Christian.Schwabe@gmx.com> > > > wrote:
> > > > Hello Hyunsik,
> > > > 
> > > > i read the bugs to this patch under the review.
> > > > What i’ve done:
> > > > -pull to the latest review
> > > > -I applied the following patches:
> > > > 
> > > > patch -p0 <TAJO-1049_3.patch
> > > > patch -p0 <TAJO-983.140916.patch
> > > > 
> > > > > > get the following output:

> > > > christians-mbp:tajo chris$ patch -p0 <TAJO-983.140916.patch
> > > > patching file tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
> > > > Hunk #2 succeeded at 52 (offset -1 lines).
> > > > Hunk #3 FAILED at 68.
> > > > 1 out of 6 hunks FAILED -- saving rejects to file
> > > > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java.rej
> > > > patching file tajo-core/src/main/java/org/apache/tajo/worker/Task.java
> > > > Hunk #1 FAILED at 48.
> > > > Hunk #2 succeeded at 97 (offset 1 line).
> > > > Hunk #3 succeeded at 197 (offset 1 line).
> > > > Hunk #4 succeeded at 594 (offset 1 line).
> > > > Hunk #5 succeeded at 639 (offset 1 line).
> > > > Hunk #6 FAILED at 699.
> > > > Hunk #7 succeeded at 754 (offset 2 lines).
> > > > 2 out of 7 hunks FAILED -- saving rejects to file
> > > > tajo-core/src/main/java/org/apache/tajo/worker/Task.java.rej
> > > > patching file
> > > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
> > > > patching file
> > > > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
> > > > Hunk #2 FAILED at 94.
> > > > Hunk #3 FAILED at 140.
> > > > Hunk #4 FAILED at 170.
> > > > Hunk #5 FAILED at 204.
> > > > Hunk #6 FAILED at 220.
> > > > 5 out of 6 hunks FAILED -- saving rejects to file
> > > > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java.rej
> > > > patching file
> > > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
> > > > patching file
> > > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
> > > > christians-mbp:tajo chris$ patch -p0 <TAJO-1049_3.patch
> > > > patching file
> > > > tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
> > > > 
> > > > Patchs seems to be applied correctly, despite the error.
> > > > 
> > > > recompiled code with mvn clean package -DskipTests -Pdist -Dtar
> > > > -Dhadoop.version=2.4.0 and get the following output — see attached log
from
> > > > maven. Build process crashed.
> > > > 
> > > > There seems to be something wrong. I hope i’ve done all right.
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Warm regards,
> > > > Chris
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Am 17.09.2014 um 16:39 schrieb Hyunsik Choi <> > > > hyunsik@apache.org>
> > > >:
> > > > 
> > > > Hi Chris,
> > > > 
> > > > Do you mean that your query is finished well? Good to hear. But, you
> > > > need to know that the patch still has some bugs. Please see the review
> > > > https://reviews.apache.org/r/25242/.
> > > > 
> > > > If your problem is solved by this patch, your problem was related to
> > > > Fetcher. I'm also expecting your problem is related to this issue
> > > > (> > > > https://github.com/apache/tajo/pull/141/files>
> > > ), which already
> > > > committed to master branch. You can use this fix from master branch
> > > > too.
> > > > 
> > > > Please refer to this document in order to learn how to apply patch. We
> > > > mostly use '--no-prefix' option in order to generate patch files. So,
> > > > you should use the patch command with -p0 instead of -p1 as follows:
> > > > 
> > > > patch -p0 < ${PATCH_FILE}.patch
> > > > 
> > > > By default, patch command uses -p1.
> > > > 
> > > > Thanks,
> > > > Hyunsik
> > > > 
> > > > On Wed, Sep 17, 2014 at 8:07 AM, Christian Schwabe
> > > > <> > > > Christian.Schwabe@gmx.com> > > > >
wrote:
> > > > 
> > > > Hello guys,
> > > > 
> > > > after days of efforts to apply the patch from May Hai, I finally be able
to
> > > > announce a success today. I want to a brief description where the problem
> > > > was. The few guides out there to apply a patch, using the Git clients
often
> > > > use git apply. However, I had no success. See discussion below.
> > > > 
> > > > christians-mbp:tajo chris$ git apply --check
> > > > /Users/chris/Desktop/TAJO-983.140916.patch
> > > > error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such file
or
> > > > directory
> > > > error: src/main/java/org/apache/tajo/worker/Task.java: No such file or
> > > > directory
> > > > error:
> > > > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java:
No
> > > > such file or directory
> > > > error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such
file
> > > > or directory
> > > > error: src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java:
> > > > No such file or directory
> > > > error: src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java:
No
> > > > such file or directory
> > > > 
> > > > ###########################################
> > > > 
> > > > christians-mbp:tajo chris$ git apply --check
> > > > /Users/chris/Desktop/TAJO-983.140916.patch
> > > > error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such file
or
> > > > directory
> > > > error: src/main/java/org/apache/tajo/worker/Task.java: No such file or
> > > > directory
> > > > error:
> > > > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java:
No
> > > > such file or directory
> > > > error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such
file
> > > > or directory
> > > > error: src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java:
> > > > No such file or directory
> > > > error: src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java:
No
> > > > such file or directory
> > > > 
> > > > ###########################################
> > > > 
> > > > christians-mbp:tajo chris$ git apply -v --check
> > > > /Users/chris/Desktop/TAJO-983.140916.patch
> > > > Checking patch src/main/java/org/apache/tajo/worker/Fetcher.java...
> > > > error: src/main/java/org/apache/tajo/worker/Fetcher.java: No such file
or
> > > > directory
> > > > Checking patch src/main/java/org/apache/tajo/worker/Task.java...
> > > > error: src/main/java/org/apache/tajo/worker/Task.java: No such file or
> > > > directory
> > > > Checking patch
> > > > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java...
> > > > error:
> > > > src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java:
No
> > > > such file or directory
> > > > Checking patch src/test/java/org/apache/tajo/worker/TestFetcher.java...
> > > > error: src/test/java/org/apache/tajo/worker/TestFetcher.java: No such
file
> > > > or directory
> > > > Checking patch
> > > > src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java...
> > > > error: src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java:
> > > > No such file or directory
> > > > Checking patch
> > > > src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java...
> > > > error: src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java:
No
> > > > such file or directory
> > > > 
> > > > ###########################################
> > > > 
> > > > christians-mbp:tajo-core chris$ git apply
> > > > /Users/chris/Desktop/TAJO-983.140916.patch
> > > > /Users/chris/Desktop/TAJO-983.140916.patch:58: trailing whitespace.
> > > > 
> > > > 
> > > > 
> > > > /Users/chris/Desktop/TAJO-983.140916.patch:80: trailing whitespace.
> > > > 
> > > > 
> > > > 
> > > > /Users/chris/Desktop/TAJO-983.140916.patch:487: trailing whitespace.
> > > > return " (start=" + startOffset() + ", length=" + length + ",
> > > > fromRemote=" + fromRemote
> > > > warning: 3 lines add whitespace errors.
> > > > 
> > > > ###########################################
> > > > 
> > > > Removal of whitespaces also brought no success.
> > > > My hope depended least on this patch. As far Tajo not run in my environment
> > > > and my standard configuration, I will unfortunately have to complete my
> > > > thesis with the result that it is currently not suitable for practical
use.
> > > > This really is not a reproach to formulate that clear again. I really
do
> > > > appreciate your hard work very much. However, I also think it's slightly
> > > > doubtful that the advice you have given can't read anywhere and only
> > > > inquired. I think that's still a pity. Surely you can not really expect,
or
> > > > rather the can but not really your goal to be always be the first point
of
> > > > contact. I think at this point there is still a lot of catching up to
do.
> > > > This is still a large deficit. Surely you can not really expect, or rather
> > > > it is hopefully not really your goal to be always be the first point of
> > > > contact. I think at this point there is still a lot of catching up to
do.
> > > > 
> > > > I am not come within the scope of my thesis within the time to the desired
> > > > goal, but of course I'm still interested in a close contact with you and
and
> > > > a further effort to solve the outstanding problem.
> > > > 
> > > > Until I have encountered the following with the patch was successfully
> > > > carried out. Unfortunately, this patch did not bring the hoped solution.
No
> > > > change in the behavior described herein. Are there any other ideas that
I
> > > > could try to fix the problem?
> > > > 
> > > > I really thank you very much for your support thus far. I really thank
you
> > > > very much for your support thus far. I really thank you very much for
your
> > > > support thus far. I really thank you very much for your support thus far.
I
> > > > really thank you very much for your support thus far. I really thank you
> > > > very much for your support thus far. I really thank you very much for
your
> > > > support thus far. I really thank you very much for your support thus far.
> > > > ;-)
> > > > 
> > > > 
> > > > christians-mbp:tajo chris$ cd tajo-yarn-pullserver/
> > > > christians-mbp:tajo-yarn-pullserver chris$ patch -p1 <
> > > > ../TAJO-983.140916.patch
> > > > can't find file to patch at input line 5
> > > > Perhaps you used the wrong -p or --strip option?
> > > > The text leading up to this was:
> > > > --------------------------
> > > > |diff --git tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
> > > > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
> > > > |index 64475fe..7310736 100644
> > > > |--- tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
> > > > |+++ tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
> > > > --------------------------
> > > > File to patch:
> > > > Skip this patch? [y] y
> > > > Skipping patch.
> > > > 6 out of 6 hunks ignored
> > > > can't find file to patch at input line 98
> > > > Perhaps you used the wrong -p or --strip option?
> > > > The text leading up to this was:
> > > > --------------------------
> > > > |diff --git tajo-core/src/main/java/org/apache/tajo/worker/Task.java
> > > > tajo-core/src/main/java/org/apache/tajo/worker/Task.java
> > > > |index 7b4cbe1..ecc4078 100644
> > > > |--- tajo-core/src/main/java/org/apache/tajo/worker/Task.java
> > > > |+++ tajo-core/src/main/java/org/apache/tajo/worker/Task.java
> > > > --------------------------
> > > > File to patch: y
> > > > y: No such file or directory
> > > > Skip this patch? [y] y
> > > > Skipping patch.
> > > > 7 out of 7 hunks ignored
> > > > can't find file to patch at input line 339
> > > > Perhaps you used the wrong -p or --strip option?
> > > > The text leading up to this was:
> > > > --------------------------
> > > > |diff --git
> > > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
> > > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
> > > > |index 4f11168..911c0c2 100644
> > > > |---
> > > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
> > > > |+++
> > > > tajo-core/src/main/java/org/apache/tajo/worker/dataserver/retriever/FileChunk.java
> > > > --------------------------
> > > > File to patch: y
> > > > y: No such file or directory
> > > > Skip this patch? [y] y
> > > > Skipping patch.
> > > > 1 out of 1 hunk ignored
> > > > can't find file to patch at input line 362
> > > > Perhaps you used the wrong -p or --strip option?
> > > > The text leading up to this was:
> > > > --------------------------
> > > > |diff --git tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
> > > > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
> > > > |index b755e02..cb19d54 100644
> > > > |--- tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
> > > > |+++ tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
> > > > --------------------------
> > > > File to patch: y
> > > > y: No such file or directory
> > > > Skip this patch? [y] y
> > > > Skipping patch.
> > > > 6 out of 6 hunks ignored
> > > > patching file
> > > > src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
> > > > patching file
> > > > src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java
> > > > christians-mbp:tajo-yarn-pullserver chris$ pwd
> > > > /Users/chris/6_Semester/Thesis/07_git/tajo/tajo-yarn-pullserver
> > > > 
> > > > 
> > > > Warm regards,
> > > > Chris
> > > > 
> > > > 
> > > > 





Mime
View raw message