hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Namikaze Minato <lloydsen...@gmail.com>
Subject Re: Dependency on absolute path /bin/ls in Shell.java
Date Sat, 12 Dec 2015 11:29:22 GMT
Then it is enough :)

Regards,
LLoyd

On 11 December 2015 at 23:27, Samuel <samuelrivas@gmail.com> wrote:
>> My question was, which spark command are you using, and since you
>> already did the analysis, which function of Shell.java is this spark
>> code using?
>
> Sorry, I misunderstood you. It was something using RawLocalFileSystem
> to load parquet files. The problem seemed to go away after I upgraded
> to spark 1.5.2 (that was in spark 1.4.0).
>
> I don't have the code with me now, but if the information is useful in
> any way I can provide an example later.
>
> Otherwise, there seems to be a ticket already to reimplement that code
> path, so that is good enough for me.
>
> Best
>
>>
>> Regards,
>> LLoyd
>>
>> On 11 December 2015 at 15:43, Samuel <samuelrivas@gmail.com> wrote:
>> > I am not using hadoop-util directly, it is Spark code what uses it
>> > (i,e. not directly under my control).
>> >
>> > Regarding ls, for my particular use case it is fine if you use "ls"
>> > instead of "/bin/ls".
>> >
>> > However, I do agree that using ls to fetch file permissions is
>> > incorrect, so a better solution (in terms of code quality) would be
>> > not to use ls at all.
>> >
>> >
>> > On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com>
wrote:
>> >> So what you ultimately need is a piece of java code listing the rwx
>> >> permissions for user, group and others that is not using ls
>> >> internally, is that correct?
>> >> If "RawLocalFileSystem" is not HDFS, do you really need to use
>> >> hadoop-util for that?
>> >> Can you tell us more about your use case?
>> >>
>> >> Regards,
>> >> LLoyd
>> >>
>> >>
>> >>
>> >>
>> >> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>> >>>> Using ls to figure out permissions is a bad design anyway, so I
would
>> >>>> not be surprised if this hardcode was reported as a bug.
>> >>>
>> >>> Of course, I have no idea why it was implemented like this. I assume
>> >>> it was written at some point in time where Java didn't provide the
>> >>> needed APIS (?)
>> >>>
>> >>> Implementing the permission check without relying in ls at all is also
>> >>> a solution for the problem I have :)
>> >>>
>> >>>> LLoyd
>> >>>>
>> >>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com>
wrote:
>> >>>> > Hi,
>> >>>> >
>> >>>> > I am experiencing some crashes when using spark over local
files (mainly for
>> >>>> > testing). Some operations fail with
>> >>>> >
>> >>>> > java.lang.RuntimeException: Error while running command to
get file
>> >>>> > permissions : java.io.IOException: Cannot run program "/bin/ls":
error=2, No
>> >>>> > such file or directory
>> >>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>> >>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>> >>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>> >>>> >         at
>> >>>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
>> >>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>> >>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>> >>>> >         at
>> >>>> > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593)
>> >>>> >         at
>> >>>> > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51)
>> >>>> >         at
>> >>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514)
>> >>>> >         at
>> >>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489)
>> >>>> >         at
>> >>>> > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292)
>> >>>> >
>> >>>> > etcetera...
>> >>>> >
>> >>>> > Which seems to be related to Shell.java in org.apache.hadoop-util,
that uses
>> >>>> > ls -ld to figure out file permissions (that is in
>> >>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that
instead of just
>> >>>> > calling ls, Shell .java calls /bin/ls, which is usually available,
but in
>> >>>> > certain circumstances might not. Regardless of the reasons
not to have ls in
>> >>>> > /bin, hardcoding the directory bans users from using the standard
mechanisms
>> >>>> > to decide which binaries to run in their systems (in this case,
$PATH), so I
>> >>>> > wonder if there is a particular reason why that path has been
hardcoded to
>> >>>> > an absolute path instead to something resolvable using$PATH.
>> >>>> >
>> >>>> > Or in other words, is this a bug or a feature?
>> >>>> >
>> >>>> > Best
>> >>>> >
>> >>>> > --
>> >>>> > Samuel
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Samuel
>> >
>> >
>> >
>> > --
>> > Samuel
>
>
>
>
> --
> Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message