hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: Dependency on absolute path /bin/ls in Shell.java
Date Fri, 11 Dec 2015 17:49:05 GMT
Hello Samuel,

Issue HADOOP-11935 tracks an improvement to re-implement this code path
using either JNI to OS syscalls or the JDK 7 java.nio.file APIs (probably
the latter).

https://issues.apache.org/jira/browse/HADOOP-11935


For right now, I don't see a viable workaround besides ensuring that the
command is accessible at /bin/ls on your system.

--Chris Nauroth




On 12/11/15, 8:05 AM, "Namikaze Minato" <lloydsensei@gmail.com> wrote:

>My question was, which spark command are you using, and since you
>already did the analysis, which function of Shell.java is this spark
>code using?
>
>Regards,
>LLoyd
>
>On 11 December 2015 at 15:43, Samuel <samuelrivas@gmail.com> wrote:
>> I am not using hadoop-util directly, it is Spark code what uses it
>> (i,e. not directly under my control).
>>
>> Regarding ls, for my particular use case it is fine if you use "ls"
>> instead of "/bin/ls".
>>
>> However, I do agree that using ls to fetch file permissions is
>> incorrect, so a better solution (in terms of code quality) would be
>> not to use ls at all.
>>
>>
>> On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com>
>>wrote:
>>> So what you ultimately need is a piece of java code listing the rwx
>>> permissions for user, group and others that is not using ls
>>> internally, is that correct?
>>> If "RawLocalFileSystem" is not HDFS, do you really need to use
>>> hadoop-util for that?
>>> Can you tell us more about your use case?
>>>
>>> Regards,
>>> LLoyd
>>>
>>>
>>>
>>>
>>> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>>>>> Using ls to figure out permissions is a bad design anyway, so I would
>>>>> not be surprised if this hardcode was reported as a bug.
>>>>
>>>> Of course, I have no idea why it was implemented like this. I assume
>>>> it was written at some point in time where Java didn't provide the
>>>> needed APIS (?)
>>>>
>>>> Implementing the permission check without relying in ls at all is also
>>>> a solution for the problem I have :)
>>>>
>>>>> LLoyd
>>>>>
>>>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
>>>>> > Hi,
>>>>> >
>>>>> > I am experiencing some crashes when using spark over local files
>>>>>(mainly for
>>>>> > testing). Some operations fail with
>>>>> >
>>>>> > java.lang.RuntimeException: Error while running command to get file
>>>>> > permissions : java.io.IOException: Cannot run program "/bin/ls":
>>>>>error=2, No
>>>>> > such file or directory
>>>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>>>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>>>> >         at
>>>>> > 
>>>>>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:3
>>>>>81)
>>>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>>>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>>>>> >         at
>>>>> > 
>>>>>org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem
>>>>>.java:593)
>>>>> >         at
>>>>> > 
>>>>>org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.
>>>>>java:51)
>>>>> >         at
>>>>> > 
>>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermiss
>>>>>ionInfo(RawLocalFileSystem.java:514)
>>>>> >         at
>>>>> > 
>>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermissi
>>>>>on(RawLocalFileSystem.java:489)
>>>>> >         at
>>>>> > 
>>>>>org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$an
>>>>>on$1$$anonfun$12.apply(newParquet.scala:292)
>>>>> >
>>>>> > etcetera...
>>>>> >
>>>>> > Which seems to be related to Shell.java in org.apache.hadoop-util,
>>>>>that uses
>>>>> > ls -ld to figure out file permissions (that is in
>>>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that
>>>>>instead of just
>>>>> > calling ls, Shell .java calls /bin/ls, which is usually available,
>>>>>but in
>>>>> > certain circumstances might not. Regardless of the reasons not to
>>>>>have ls in
>>>>> > /bin, hardcoding the directory bans users from using the standard
>>>>>mechanisms
>>>>> > to decide which binaries to run in their systems (in this case,
>>>>>$PATH), so I
>>>>> > wonder if there is a particular reason why that path has been
>>>>>hardcoded to
>>>>> > an absolute path instead to something resolvable using$PATH.
>>>>> >
>>>>> > Or in other words, is this a bug or a feature?
>>>>> >
>>>>> > Best
>>>>> >
>>>>> > --
>>>>> > Samuel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Samuel
>>
>>
>>
>> --
>> Samuel
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: user-help@hadoop.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message