Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <D2904BFC.34E15%cnauroth@hortonworks.com>
References: 
 <CAH2nEUgsSeoJTJkD4T8z=D18ECnRgQ3Qvz861gU5+NPSsNgT=A@mail.gmail.com>
 <CACmJb3zHG8ghn+bSgcZ+=jV3tqvxkbPfSPz3guy_c_8gqe1GVQ@mail.gmail.com>
 <CAH2nEUjHTxY5mZQF7nizp_fv1=ju+jEu9QnKr3zgPQiyoz4WSQ@mail.gmail.com>
 <CACmJb3yJSFXfZ3L9084ucpLs-bccCD6XaF2o=TT6_synUz_FUA@mail.gmail.com>
 <CAH2nEUg=jVrjVa50GpYebshBKTvABMVHxfhkNmQA_EzW==damA@mail.gmail.com>
 <CACmJb3zhKc2UEO23dj_PeLNXOYQM+N9gGvEbns_uZ01w=K0p_A@mail.gmail.com>
 <D2904BFC.34E15%cnauroth@hortonworks.com>
From: Samuel <samuelrivas@gmail.com>
Date: Fri, 11 Dec 2015 23:27:09 +0100
Message-ID: 
 <CAH2nEUi31v+LiPAntMpRTsNkxjeEtygJ42SWeLD=qz7SWuhMcA@mail.gmail.com>
Subject: Re: Dependency on absolute path /bin/ls in Shell.java
To: Chris Nauroth <cnauroth@hortonworks.com>
Cc: Namikaze Minato <lloydsensei@gmail.com>,
	"user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8

Thanks for the information. If you are already aware of the problem,
that is enough for me :)

Best

On 11 December 2015 at 18:49, Chris Nauroth <cnauroth@hortonworks.com> wrote:
> Hello Samuel,
>
> Issue HADOOP-11935 tracks an improvement to re-implement this code path
> using either JNI to OS syscalls or the JDK 7 java.nio.file APIs (probably
> the latter).
>
> https://issues.apache.org/jira/browse/HADOOP-11935
>
>
> For right now, I don't see a viable workaround besides ensuring that the
> command is accessible at /bin/ls on your system.
>
> --Chris Nauroth
>
>
>
>
> On 12/11/15, 8:05 AM, "Namikaze Minato" <lloydsensei@gmail.com> wrote:
>
>>My question was, which spark command are you using, and since you
>>already did the analysis, which function of Shell.java is this spark
>>code using?
>>
>>Regards,
>>LLoyd
>>
>>On 11 December 2015 at 15:43, Samuel <samuelrivas@gmail.com> wrote:
>>> I am not using hadoop-util directly, it is Spark code what uses it
>>> (i,e. not directly under my control).
>>>
>>> Regarding ls, for my particular use case it is fine if you use "ls"
>>> instead of "/bin/ls".
>>>
>>> However, I do agree that using ls to fetch file permissions is
>>> incorrect, so a better solution (in terms of code quality) would be
>>> not to use ls at all.
>>>
>>>
>>> On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com>
>>>wrote:
>>>> So what you ultimately need is a piece of java code listing the rwx
>>>> permissions for user, group and others that is not using ls
>>>> internally, is that correct?
>>>> If "RawLocalFileSystem" is not HDFS, do you really need to use
>>>> hadoop-util for that?
>>>> Can you tell us more about your use case?
>>>>
>>>> Regards,
>>>> LLoyd
>>>>
>>>>
>>>>
>>>>
>>>> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>>>>>> Using ls to figure out permissions is a bad design anyway, so I would
>>>>>> not be surprised if this hardcode was reported as a bug.
>>>>>
>>>>> Of course, I have no idea why it was implemented like this. I assume
>>>>> it was written at some point in time where Java didn't provide the
>>>>> needed APIS (?)
>>>>>
>>>>> Implementing the permission check without relying in ls at all is also
>>>>> a solution for the problem I have :)
>>>>>
>>>>>> LLoyd
>>>>>>
>>>>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
>>>>>> > Hi,
>>>>>> >
>>>>>> > I am experiencing some crashes when using spark over local files
>>>>>>(mainly for
>>>>>> > testing). Some operations fail with
>>>>>> >
>>>>>> > java.lang.RuntimeException: Error while running command to get file
>>>>>> > permissions : java.io.IOException: Cannot run program "/bin/ls":
>>>>>>error=2, No
>>>>>> > such file or directory
>>>>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>>>>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>>>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:3
>>>>>>81)
>>>>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>>>>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem
>>>>>>.java:593)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.
>>>>>>java:51)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermiss
>>>>>>ionInfo(RawLocalFileSystem.java:514)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermissi
>>>>>>on(RawLocalFileSystem.java:489)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$an
>>>>>>on$1$$anonfun$12.apply(newParquet.scala:292)
>>>>>> >
>>>>>> > etcetera...
>>>>>> >
>>>>>> > Which seems to be related to Shell.java in org.apache.hadoop-util,
>>>>>>that uses
>>>>>> > ls -ld to figure out file permissions (that is in
>>>>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that
>>>>>>instead of just
>>>>>> > calling ls, Shell .java calls /bin/ls, which is usually available,
>>>>>>but in
>>>>>> > certain circumstances might not. Regardless of the reasons not to
>>>>>>have ls in
>>>>>> > /bin, hardcoding the directory bans users from using the standard
>>>>>>mechanisms
>>>>>> > to decide which binaries to run in their systems (in this case,
>>>>>>$PATH), so I
>>>>>> > wonder if there is a particular reason why that path has been
>>>>>>hardcoded to
>>>>>> > an absolute path instead to something resolvable using$PATH.
>>>>>> >
>>>>>> > Or in other words, is this a bug or a feature?
>>>>>> >
>>>>>> > Best
>>>>>> >
>>>>>> > --
>>>>>> > Samuel
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Samuel
>>>
>>>
>>>
>>> --
>>> Samuel
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>


-- 
Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org