mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Chauhan <an...@malloc64.com>
Subject Re: Why rely on url scheme for fetching?
Date Mon, 03 Nov 2014 18:50:54 GMT
Hi,

Okay, thanks Ian. What's the expected ETA on getting 0.21.0 out? jira didn't have a release
date set.

-- Ankur

> On 3 Nov 2014, at 10:37, Ian Downes <idownes@twitter.com.INVALID> wrote:
> 
> Unfortunately, this will not get in 0.21.0 as we're tagging that today.
> 
> Please tag the ticket(s) as Target Version = 0.22.0.
> 
> Ian
> 
> On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan <ankur@malloc64.com <mailto:ankur@malloc64.com>>
wrote:
> 
>> Hi Tim/others,
>> 
>> Is this to be included in the 0.21.0 release? If so, I don't know how to
>> tag it etc. I would really (shamelessly) love it to be included as it would
>> really simplify my intended usecase of using snackfs (cassandra backed
>> filesystem).
>> 
>> -- Ankur
>> 
>>> On 3 Nov 2014, at 09:28, Ankur Chauhan <ankur@malloc64.com> wrote:
>>> 
>>> Yea, I saw those today morning. I'll hold off a little mesos-336 changes
>> a lot of stuff.
>>> 
>>> Sent from my iPhone
>>> 
>>> On Nov 3, 2014, at 9:18 AM, Adam Bordelon <adam@mesosphere.io <mailto:adam@mesosphere.io>
<mailto:
>> adam@mesosphere.io <mailto:adam@mesosphere.io>>> wrote:
>>> 
>>>> + Bernd, who has done some fetcher work, including additional testing,
>> for MESOS-1316, MESOS-1945, and MESOS-336
>>>> 
>>>> On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon <dhamon@twopensource.com
<mailto:dhamon@twopensource.com>
>> <mailto:dhamon@twopensource.com <mailto:dhamon@twopensource.com>>>
wrote:
>>>> Hi Ankur
>>>> 
>>>> I think this is a great approach. It makes the code much simpler,
>> extensible, and more testable. Anyone that's heard me rant knows I am a big
>> fan of unit tests over integration tests, so this shouldn't surprise anyone
>> :)
>>>> 
>>>> If you haven't already, please read the documentation on contributing
>> to Mesos and the style guide to ensure all the naming is as expected, then
>> you can push the patch to reviewboard to get it reviewed and committed.
>>>> 
>>>> On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan <ankur@malloc64.com <mailto:ankur@malloc64.com>
>> <mailto:ankur@malloc64.com <mailto:ankur@malloc64.com>>> wrote:
>>>> Hi,
>>>> 
>>>> I did some learning today! This is pretty much a very rough draft of
>> the tests/refactor of mesos-fetcher that I have come up with. Again, If
>> there are some obvious mistakes, please let me know. (this is my first pass
>> after all).
>>>> https://github.com/ankurcha/mesos/compare/prefer_2 <https://github.com/ankurcha/mesos/compare/prefer_2>
<
>> https://github.com/ankurcha/mesos/compare/prefer_2 <https://github.com/ankurcha/mesos/compare/prefer_2>>
>>>> 
>>>> My main intention is to break the logic of the fetcher info some very
>> discrete components that i can write tests against. I am still re-learning
>> cpp/mesos code styles etc so I may be a little slow to catch up but I would
>> really appreciate any comments and/or suggestions.
>>>> 
>>>> -- Ankur
>>>> @ankurcha
>>>> 
>>>>> On 2 Nov 2014, at 18:17, Ankur Chauhan <ankur@malloc64.com <mailto:ankur@malloc64.com>
<mailto:
>> ankur@malloc64.com <mailto:ankur@malloc64.com>>> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> I noticed that the current set of tests in
>> `src/tests/fetcher_tests.cpp` is pretty coarse grained and are more on the
>> lines of a functional test. I was going to add some tests but it seems like
>> if I am to do that I would need to add a test dependency on hadoop.
>>>>> 
>>>>> As an alternative, I propose adding a good set of unit tests around
>> the methods used by `src/launcher/fetcher.cpp` and `src/hdfs/hdfs.cpp`.
>> This should be able to catch a good portion of cases at the same time
>> keeping the dependencies and runtime of tests low. What do you guys thing
>> about this?
>>>>> 
>>>>> PS: I am pretty green in terms of gtest and the overall c++ testing
>> methodology. Can someone give me pointers to good examples of tests in the
>> codebase.
>>>>> 
>>>>> -- Ankur
>>>>> 
>>>>>> On 1 Nov 2014, at 22:54, Adam Bordelon <adam@mesosphere.io <mailto:adam@mesosphere.io>
<mailto:
>> adam@mesosphere.io <mailto:adam@mesosphere.io>>> wrote:
>>>>>> 
>>>>>> Thank you Ankur. At first glance, it looks great. We'll do a more
>> thorough review of it very soon.
>>>>>> I know Tim St. Clair had some ideas for fixing MESOS-1711 <
>> https://issues.apache.org/jira/browse/MESOS-1711 <https://issues.apache.org/jira/browse/MESOS-1711>>;
he may want to review
>> too.
>>>>>> 
>>>>>> On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan <ankur@malloc64.com
<mailto:ankur@malloc64.com>
>> <mailto:ankur@malloc64.com <mailto:ankur@malloc64.com>>> wrote:
>>>>>> Hi Tim,
>>>>>> 
>>>>>> I just created a review https://reviews.apache.org/r/27483/ <https://reviews.apache.org/r/27483/>
<
>> https://reviews.apache.org/r/27483/ <https://reviews.apache.org/r/27483/>>
It's my first stab at it and I will
>> try to add more tests as I figure out how to do the hadoop mocking and
>> stuff. Have a look and let me know what you think about it so far.
>>>>>> 
>>>>>> -- Ankur
>>>>>> 
>>>>>>> On 1 Nov 2014, at 20:05, Ankur Chauhan <ankur@malloc64.com
<mailto:ankur@malloc64.com> <mailto:
>> ankur@malloc64.com <mailto:ankur@malloc64.com>>> wrote:
>>>>>>> 
>>>>>>> Yea, i saw that the minute i pressed send. I'll start the review
>> board so that people can have a look at the change.
>>>>>>> 
>>>>>>> -- Ankur
>>>>>>> 
>>>>>>>> On 1 Nov 2014, at 20:01, Tim Chen <tim@mesosphere.io <mailto:tim@mesosphere.io>
<mailto:
>> tim@mesosphere.io <mailto:tim@mesosphere.io>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Ankur,
>>>>>>>> 
>>>>>>>> There is a fetcher_tests.cpp in src/tests.
>>>>>>>> 
>>>>>>>> Tim
>>>>>>>> 
>>>>>>>> On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan <ankur@malloc64.com
<mailto:ankur@malloc64.com>
>> <mailto:ankur@malloc64.com <mailto:ankur@malloc64.com>>> wrote:
>>>>>>>> Hi Tim,
>>>>>>>> 
>>>>>>>> I am trying to find/write some test cases. I couldn't find
a
>> fetcher_tests.{cpp|hpp} so once I have something, I'll post on review
>> board. I am new to gmock/gtest so bear with me while i get up to speed.
>>>>>>>> 
>>>>>>>> -- Ankur
>>>>>>>> 
>>>>>>>>> On 1 Nov 2014, at 19:23, Timothy Chen <tim@mesosphere.io
<mailto:tim@mesosphere.io> <mailto:
>> tim@mesosphere.io <mailto:tim@mesosphere.io>>> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Ankur,
>>>>>>>>> 
>>>>>>>>> Can you post on reviewboard? We can discuss more about
the code
>> there.
>>>>>>>>> 
>>>>>>>>> Tim
>>>>>>>>> 
>>>>>>>>> Sent from my iPhone
>>>>>>>>> 
>>>>>>>>> On Nov 1, 2014, at 6:29 PM, Ankur Chauhan <ankur@malloc64.com
<mailto:ankur@malloc64.com>
>> <mailto:ankur@malloc64.com <mailto:ankur@malloc64.com>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Tim,
>>>>>>>>>> 
>>>>>>>>>> I don't think there is an issue which is directly
in line with
>> what i wanted but the closest one that I could find in JIRA is
>> https://issues.apache.org/jira/browse/MESOS-1711 <https://issues.apache.org/jira/browse/MESOS-1711>
<
>> https://issues.apache.org/jira/browse/MESOS-1711 <https://issues.apache.org/jira/browse/MESOS-1711>>
>>>>>>>>>> 
>>>>>>>>>> I have a branch (
>> https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher <https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher>
<
>> https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher <https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher>>
) that
>> has a change that would enable users to specify whatever hdfs compatible
>> uris to the mesos-fetcher but maybe you can weight in on it. Do you think
>> this is the right track? if so, i would like to pick this issue and submit
>> a patch for review.
>>>>>>>>>> 
>>>>>>>>>> -- Ankur
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On 1 Nov 2014, at 04:32, Tom Arnfeld <tom@duedil.com
<mailto:tom@duedil.com> <mailto:
>> tom@duedil.com <mailto:tom@duedil.com>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Completely +1 to this. There are now quite a
lot of hadoop
>> compatible filesystem wrappers out in the wild and this would certainly be
>> very useful.
>>>>>>>>>>> 
>>>>>>>>>>> I'm happy to contribute a patch. Here's a few
related issues
>> that might be of interest;
>>>>>>>>>>> 
>>>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1887
<https://issues.apache.org/jira/browse/MESOS-1887> <
>> https://issues.apache.org/jira/browse/MESOS-1887 <https://issues.apache.org/jira/browse/MESOS-1887>>
>>>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1316
<https://issues.apache.org/jira/browse/MESOS-1316> <
>> https://issues.apache.org/jira/browse/MESOS-1316 <https://issues.apache.org/jira/browse/MESOS-1316>>
>>>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-336
<https://issues.apache.org/jira/browse/MESOS-336> <
>> https://issues.apache.org/jira/browse/MESOS-336 <https://issues.apache.org/jira/browse/MESOS-336>>
>>>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1248
<https://issues.apache.org/jira/browse/MESOS-1248> <
>> https://issues.apache.org/jira/browse/MESOS-1248 <https://issues.apache.org/jira/browse/MESOS-1248>>
>>>>>>>>>>> 
>>>>>>>>>>> On 31 October 2014 22:39, Tim Chen <tim@mesosphere.io
<mailto:tim@mesosphere.io> <mailto:
>> tim@mesosphere.io <mailto:tim@mesosphere.io>>> wrote:
>>>>>>>>>>> I believe there is already a JIRA ticket for
this, if you search
>> for fetcher in Mesos JIRA I think you can find it.
>>>>>>>>>>> 
>>>>>>>>>>> Tim
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan
<
>> ankur@malloc64.com <mailto:ankur@malloc64.com> <mailto:ankur@malloc64.com
<mailto:ankur@malloc64.com>>> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> I have been looking at some of the stuff around
the fetcher and
>> saw something interesting. The code for fetcher::fetch method is dependent
>> on a hard coded list of url schemes. No doubt that this works but is very
>> restrictive.
>>>>>>>>>>> Hadoop/HDFS in general is pretty flexible when
it comes to being
>> able to fetch stuff from urls and has the ability to fetch a large number
>> of types of urls and can be extended by adding configuration into the
>> conf/hdfs-site.xml and core-site.xml
>>>>>>>>>>> 
>>>>>>>>>>> What I am proposing is that we refactor the fetcher.cpp
to
>> prefer to use the hdfs (using hdfs/hdfs.hpp) to do all the fetching if
>> HADOOP_HOME is set and $HADOOP_HOME/bin/hadoop is available. This logic
>> already exists and we can just use it. The fallback logic for using
>> net::download or local file copy is may be left in place for installations
>> that do not have hadoop configured. This means that if hadoop is present we
>> can directly fetch urls such as tachyon://... snackfs:// ... cfs:// ....
>> ftp://... s3://... http:// ... file:// with no extra effort. This makes
>> up for a much better experience when it comes to debugging and
>> extensibility.
>>>>>>>>>>> 
>>>>>>>>>>> What do others think about this?
>>>>>>>>>>> 
>>>>>>>>>>> - Ankur
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Dominic Hamon | @mrdo | Twitter
>>>> There are no bad ideas; only good ideas that go horribly wrong.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message