hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Namikaze Minato <lloydsen...@gmail.com>
Subject Re: Dependency on absolute path /bin/ls in Shell.java
Date Fri, 11 Dec 2015 13:56:04 GMT
So what you ultimately need is a piece of java code listing the rwx
permissions for user, group and others that is not using ls
internally, is that correct?
If "RawLocalFileSystem" is not HDFS, do you really need to use
hadoop-util for that?
Can you tell us more about your use case?

Regards,
LLoyd




On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>> Using ls to figure out permissions is a bad design anyway, so I would
>> not be surprised if this hardcode was reported as a bug.
>
> Of course, I have no idea why it was implemented like this. I assume
> it was written at some point in time where Java didn't provide the
> needed APIS (?)
>
> Implementing the permission check without relying in ls at all is also
> a solution for the problem I have :)
>
>> LLoyd
>>
>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
>> > Hi,
>> >
>> > I am experiencing some crashes when using spark over local files (mainly for
>> > testing). Some operations fail with
>> >
>> > java.lang.RuntimeException: Error while running command to get file
>> > permissions : java.io.IOException: Cannot run program "/bin/ls": error=2, No
>> > such file or directory
>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>> >         at
>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>> >         at
>> > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593)
>> >         at
>> > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51)
>> >         at
>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514)
>> >         at
>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489)
>> >         at
>> > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292)
>> >
>> > etcetera...
>> >
>> > Which seems to be related to Shell.java in org.apache.hadoop-util, that uses
>> > ls -ld to figure out file permissions (that is in
>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead of just
>> > calling ls, Shell .java calls /bin/ls, which is usually available, but in
>> > certain circumstances might not. Regardless of the reasons not to have ls in
>> > /bin, hardcoding the directory bans users from using the standard mechanisms
>> > to decide which binaries to run in their systems (in this case, $PATH), so I
>> > wonder if there is a particular reason why that path has been hardcoded to
>> > an absolute path instead to something resolvable using$PATH.
>> >
>> > Or in other words, is this a bug or a feature?
>> >
>> > Best
>> >
>> > --
>> > Samuel
>
>
>
>
> --
> Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message