ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Onischuk <aonis...@hortonworks.com>
Subject Re: Review Request 43675: Preupload.py should pre-create hdfs directories
Date Thu, 18 Feb 2016 10:40:54 GMT


> On Feb. 17, 2016, 9:39 p.m., Sumit Mohanty wrote:
> > ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py,
line 460
> > <https://reviews.apache.org/r/43675/diff/1/?file=1253095#file1253095line460>
> >
> >     Overall changes look good. Can we enable it unconditionally? What will happen
when a path got created say "/usr/abc" then user changed the property to be "/usr/xyz" and
then manually deleted "/usr/abc" in HDFS. Now, if user changes the path back to "/usr/abc"
will Ambari re-create the path?
> 
> Andrew Onischuk wrote:
>     Unconditionally? I don't get it, if we remove this condition it's gonna ignore every
single hdfsResource.
>     
>     Also the usecase you mentioned will indeed cause a problem. 
>     But the usecase is a bit weird, because if user deleted the directory he did it for
purpose obviously.
> 
> Sumit Mohanty wrote:
>     What I meant is if we need a config to enable/disable the ability for HdfsResourceProvider
to track work using that file. I agree that my example is odd. A more real life scenario will
be to Add HBase as a service, delete it and delete its data folders from HDFS. Then add it
back and the data folders will not be created again.

there is no config to enable/disable that in the patch, I think it is what you mean by "unconditionally".

Also good catch on the case you mentioned, but I don't see how it can be fixed.


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43675/#review119530
-----------------------------------------------------------


On Feb. 17, 2016, 9:07 p.m., Andrew Onischuk wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/43675/
> -----------------------------------------------------------
> 
> (Updated Feb. 17, 2016, 9:07 p.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-15079
>     https://issues.apache.org/jira/browse/AMBARI-15079
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This should eliminate any 'hadoop --config ... jar' fast-hdfs-resource.jar calls during
deploy.
> By executing them pre-deploy in preupload.py, and adding to ignore file.
> This should benefit ~6 minutes of time deploy, taking the data from 24min cluster. And
probably less for faster deployments.
> 
> 
> Diffs
> -----
> 
>   ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py
ebcf1a4 
>   ambari-common/src/main/python/resource_management/libraries/resources/hdfs_resource.py
7c12409 
>   ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/params.py
2bd2626 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
f3a97fc 
>   ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
b150464 
>   ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
6837bf1 
>   ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
29c4784 
>   ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
dc17dba 
>   ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/params.py
47af240 
>   ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
7ec85b5 
>   ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/params.py
69c03ea 
>   ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py
7a2f6f6 
>   ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/params_linux.py
f923723 
>   ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/params_linux.py
09b7876 
>   ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py
68c4f37 
>   ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py
25da2a1 
>   ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/params_linux.py
25f867e 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
f0b6927 
>   ambari-server/src/main/resources/scripts/Ambaripreupload.py 61db286 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
2a9d7c5 
>   ambari-server/src/test/python/stacks/2.0.6/AMBARI_METRICS/test_metrics_collector.py
96e2286 
>   ambari-server/src/test/python/stacks/2.0.6/HBASE/test_hbase_master.py 13b2e33 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 39244ff 
>   ambari-server/src/test/python/stacks/2.0.6/HDFS/test_service_check.py 0f5afa8 
>   ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 494d16c 
>   ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_service_check.py ea17c27

>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py ba1b84a 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_service_check.py a6d0145 
>   ambari-server/src/test/python/stacks/2.0.6/PIG/test_pig_service_check.py c5de4c3 
>   ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py 185db68 
>   ambari-server/src/test/python/stacks/2.0.6/YARN/test_mapreduce2_service_check.py 54ecf33

>   ambari-server/src/test/python/stacks/2.1/FALCON/test_falcon_server.py b14a527 
>   ambari-server/src/test/python/stacks/2.1/TEZ/test_service_check.py 728a1a8 
>   ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 4fccd91 
>   ambari-server/src/test/python/stacks/2.2/PIG/test_pig_service_check.py bf42f9b 
>   ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py f7cd3b5 
>   ambari-server/src/test/python/stacks/2.3/MAHOUT/test_mahout_service_check.py c9f376b

>   ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 8e7d367

>   ambari-server/src/test/python/stacks/2.3/YARN/test_ats_1_5.py 0b3d434 
> 
> Diff: https://reviews.apache.org/r/43675/diff/
> 
> 
> Testing
> -------
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message