hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samuel <samuelri...@gmail.com>
Subject Re: Dependency on absolute path /bin/ls in Shell.java
Date Fri, 11 Dec 2015 22:27:11 GMT
> My question was, which spark command are you using, and since you
> already did the analysis, which function of Shell.java is this spark
> code using?

Sorry, I misunderstood you. It was something using RawLocalFileSystem
to load parquet files. The problem seemed to go away after I upgraded
to spark 1.5.2 (that was in spark 1.4.0).

I don't have the code with me now, but if the information is useful in
any way I can provide an example later.

Otherwise, there seems to be a ticket already to reimplement that code
path, so that is good enough for me.

Best

>
> Regards,
> LLoyd
>
> On 11 December 2015 at 15:43, Samuel <samuelrivas@gmail.com> wrote:
> > I am not using hadoop-util directly, it is Spark code what uses it
> > (i,e. not directly under my control).
> >
> > Regarding ls, for my particular use case it is fine if you use "ls"
> > instead of "/bin/ls".
> >
> > However, I do agree that using ls to fetch file permissions is
> > incorrect, so a better solution (in terms of code quality) would be
> > not to use ls at all.
> >
> >
> > On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com> wrote:
> >> So what you ultimately need is a piece of java code listing the rwx
> >> permissions for user, group and others that is not using ls
> >> internally, is that correct?
> >> If "RawLocalFileSystem" is not HDFS, do you really need to use
> >> hadoop-util for that?
> >> Can you tell us more about your use case?
> >>
> >> Regards,
> >> LLoyd
> >>
> >>
> >>
> >>
> >> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
> >>>> Using ls to figure out permissions is a bad design anyway, so I would
> >>>> not be surprised if this hardcode was reported as a bug.
> >>>
> >>> Of course, I have no idea why it was implemented like this. I assume
> >>> it was written at some point in time where Java didn't provide the
> >>> needed APIS (?)
> >>>
> >>> Implementing the permission check without relying in ls at all is also
> >>> a solution for the problem I have :)
> >>>
> >>>> LLoyd
> >>>>
> >>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com> wrote:
> >>>> > Hi,
> >>>> >
> >>>> > I am experiencing some crashes when using spark over local files
(mainly for
> >>>> > testing). Some operations fail with
> >>>> >
> >>>> > java.lang.RuntimeException: Error while running command to get
file
> >>>> > permissions : java.io.IOException: Cannot run program "/bin/ls":
error=2, No
> >>>> > such file or directory
> >>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> >>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
> >>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
> >>>> >         at
> >>>> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
> >>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
> >>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
> >>>> >         at
> >>>> > org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:593)
> >>>> >         at
> >>>> > org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.java:51)
> >>>> >         at
> >>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:514)
> >>>> >         at
> >>>> > org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:489)
> >>>> >         at
> >>>> > org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$anon$1$$anonfun$12.apply(newParquet.scala:292)
> >>>> >
> >>>> > etcetera...
> >>>> >
> >>>> > Which seems to be related to Shell.java in org.apache.hadoop-util,
that uses
> >>>> > ls -ld to figure out file permissions (that is in
> >>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that instead
of just
> >>>> > calling ls, Shell .java calls /bin/ls, which is usually available,
but in
> >>>> > certain circumstances might not. Regardless of the reasons not
to have ls in
> >>>> > /bin, hardcoding the directory bans users from using the standard
mechanisms
> >>>> > to decide which binaries to run in their systems (in this case,
$PATH), so I
> >>>> > wonder if there is a particular reason why that path has been hardcoded
to
> >>>> > an absolute path instead to something resolvable using$PATH.
> >>>> >
> >>>> > Or in other words, is this a bug or a feature?
> >>>> >
> >>>> > Best
> >>>> >
> >>>> > --
> >>>> > Samuel
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Samuel
> >
> >
> >
> > --
> > Samuel




-- 
Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message