hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samuel <samuelri...@gmail.com>
Subject Re: Dependency on absolute path /bin/ls in Shell.java
Date Fri, 11 Dec 2015 22:27:09 GMT
Thanks for the information. If you are already aware of the problem,
that is enough for me :)

Best

On 11 December 2015 at 18:49, Chris Nauroth <cnauroth@hortonworks.com> wrote:
> Hello Samuel,
>
> Issue HADOOP-11935 tracks an improvement to re-implement this code path
> using either JNI to OS syscalls or the JDK 7 java.nio.file APIs (probably
> the latter).
>
> https://issues.apache.org/jira/browse/HADOOP-11935
>
>
> For right now, I don't see a viable workaround besides ensuring that the
> command is accessible at /bin/ls on your system.
>
> --Chris Nauroth
>
>
>
>
> On 12/11/15, 8:05 AM, "Namikaze Minato" <lloydsensei@gmail.com> wrote:
>
>>My question was, which spark command are you using, and since you
>>already did the analysis, which function of Shell.java is this spark
>>code using?
>>
>>Regards,
>>LLoyd
>>
>>On 11 December 2015 at 15:43, Samuel <samuelrivas@gmail.com> wrote:
>>> I am not using hadoop-util directly, it is Spark code what uses it
>>> (i,e. not directly under my control).
>>>
>>> Regarding ls, for my particular use case it is fine if you use "ls"
>>> instead of "/bin/ls".
>>>
>>> However, I do agree that using ls to fetch file permissions is
>>> incorrect, so a better solution (in terms of code quality) would be
>>> not to use ls at all.
>>>
>>>
>>> On 11 December 2015 at 14:56, Namikaze Minato <lloydsensei@gmail.com>
>>>wrote:
>>>> So what you ultimately need is a piece of java code listing the rwx
>>>> permissions for user, group and others that is not using ls
>>>> internally, is that correct?
>>>> If "RawLocalFileSystem" is not HDFS, do you really need to use
>>>> hadoop-util for that?
>>>> Can you tell us more about your use case?
>>>>
>>>> Regards,
>>>> LLoyd
>>>>
>>>>
>>>>
>>>>
>>>> On 11 December 2015 at 13:05, Samuel <samuelrivas@gmail.com> wrote:
>>>>>> Using ls to figure out permissions is a bad design anyway, so I would
>>>>>> not be surprised if this hardcode was reported as a bug.
>>>>>
>>>>> Of course, I have no idea why it was implemented like this. I assume
>>>>> it was written at some point in time where Java didn't provide the
>>>>> needed APIS (?)
>>>>>
>>>>> Implementing the permission check without relying in ls at all is also
>>>>> a solution for the problem I have :)
>>>>>
>>>>>> LLoyd
>>>>>>
>>>>>> On 11 December 2015 at 09:19, Samuel <samuelrivas@gmail.com>
wrote:
>>>>>> > Hi,
>>>>>> >
>>>>>> > I am experiencing some crashes when using spark over local files
>>>>>>(mainly for
>>>>>> > testing). Some operations fail with
>>>>>> >
>>>>>> > java.lang.RuntimeException: Error while running command to get
file
>>>>>> > permissions : java.io.IOException: Cannot run program "/bin/ls":
>>>>>>error=2, No
>>>>>> > such file or directory
>>>>>> >         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
>>>>>> >         at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
>>>>>> >         at org.apache.hadoop.util.Shell.run(Shell.java:188)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:3
>>>>>>81)
>>>>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:467)
>>>>>> >         at org.apache.hadoop.util.Shell.execCommand(Shell.java:450)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem
>>>>>>.java:593)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem.access$100(RawLocalFileSystem.
>>>>>>java:51)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermiss
>>>>>>ionInfo(RawLocalFileSystem.java:514)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermissi
>>>>>>on(RawLocalFileSystem.java:489)
>>>>>> >         at
>>>>>> >
>>>>>>org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$buildScan$1$$an
>>>>>>on$1$$anonfun$12.apply(newParquet.scala:292)
>>>>>> >
>>>>>> > etcetera...
>>>>>> >
>>>>>> > Which seems to be related to Shell.java in org.apache.hadoop-util,
>>>>>>that uses
>>>>>> > ls -ld to figure out file permissions (that is in
>>>>>> > RawLocalFileSystem.loadPermissionsInfo). The problem is that
>>>>>>instead of just
>>>>>> > calling ls, Shell .java calls /bin/ls, which is usually available,
>>>>>>but in
>>>>>> > certain circumstances might not. Regardless of the reasons not
to
>>>>>>have ls in
>>>>>> > /bin, hardcoding the directory bans users from using the standard
>>>>>>mechanisms
>>>>>> > to decide which binaries to run in their systems (in this case,
>>>>>>$PATH), so I
>>>>>> > wonder if there is a particular reason why that path has been
>>>>>>hardcoded to
>>>>>> > an absolute path instead to something resolvable using$PATH.
>>>>>> >
>>>>>> > Or in other words, is this a bug or a feature?
>>>>>> >
>>>>>> > Best
>>>>>> >
>>>>>> > --
>>>>>> > Samuel
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Samuel
>>>
>>>
>>>
>>> --
>>> Samuel
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>>For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>



-- 
Samuel

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message