hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samuel <samuelri...@gmail.com>
Subject Re: Dependency on absolute path /bin/ls in Shell.java
Date Fri, 11 Dec 2015 14:43:23 GMT
I am not using hadoop-util directly, it is Spark code what uses it
(i,e. not directly under my control).

Regarding ls, for my particular use case it is fine if you use "ls"
instead of "/bin/ls".

However, I do agree that using ls to fetch file permissions is
incorrect, so a better solution (in terms of code quality) would be
not to use ls at all.


On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com> wrote:
> So what you ultimately need is a piece of java code listing the rwx
> permissions for user, group and others that is not using ls
> internally, is that correct?
> If "RawLocalFileSystem" is not HDFS, do you really need to use
> hadoop-util for that?
> Can you tell us more about your use case?
>
> Regards,
> LLoyd
>
>
>
>
> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>>> Using ls to figure out permissions is a bad design anyway, so I would
>>> not be surprised if this hardcode was reported as a bug.
>>
>> Of course, I have no idea why it was implemented like this. I assume
>> it was written at some point in time where Java didn't provide the
>> needed APIS (?)
>>
>> Implementing the permission check without relying in ls at all is also
>> a solution for the problem I have :)
>>
>>> LLoyd
>>>
>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I am experiencing some crashes when using spark over local files (mainly
for
>>> > testing). Some operations fail with
>>> >
>>> > java.lang.RuntimeException: Error while running command to get file
>>> > permissions : java.io.IOException: Cannot run program "/bin/ls": error=2,
No
>>> > such file or directory
>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>> >         at
>>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514)
>>> >         at
>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489)
>>> >         at
>>> > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292)
>>> >
>>> > etcetera...
>>> >
>>> > Which seems to be related to Shell.java in org.apache.hadoop-util, that
uses
>>> > ls -ld to figure out file permissions (that is in
>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead of
just
>>> > calling ls, Shell .java calls /bin/ls, which is usually available, but in
>>> > certain circumstances might not. Regardless of the reasons not to have ls
in
>>> > /bin, hardcoding the directory bans users from using the standard mechanisms
>>> > to decide which binaries to run in their systems (in this case, $PATH),
so I
>>> > wonder if there is a particular reason why that path has been hardcoded
to
>>> > an absolute path instead to something resolvable using$PATH.
>>> >
>>> > Or in other words, is this a bug or a feature?
>>> >
>>> > Best
>>> >
>>> > --
>>> > Samuel
>>
>>
>>
>>
>> --
>> Samuel



-- 
Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message