47

I can use ~ instead of /home/username/ to point to a file path when, for example, unzipping a .zip file.

However, today when I followed the same way to run a RNN example in terminal, tensorflow.python.framework.errors_impl.NotFoundError was thrown.

$ python ptb_word_lm.py --data_path=~/anaconda2/lib/python2.7/site-packages/tensorflow/models-master/tutorials/rnn/simple-examples/data/ --model=small 
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
Traceback (most recent call last):
  File "ptb_word_lm.py", line 374, in <module>
    tf.app.run()
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "ptb_word_lm.py", line 321, in main
    raw_data = reader.ptb_raw_data(FLAGS.data_path)
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/models-master/tutorials/rnn/ptb/reader.py", line 73, in ptb_raw_data
    word_to_id = _build_vocab(train_path)
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/models-master/tutorials/rnn/ptb/reader.py", line 34, in _build_vocab
    data = _read_words(filename)
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/models-master/tutorials/rnn/ptb/reader.py", line 30, in _read_words
    return f.read().decode("utf-8").replace("\n", "<eos>").split()
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 106, in read
    self._preread_check()
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/python/lib/io/file_io.py", line 73, in _preread_check
    compat.as_bytes(self.__name), 1024 * 512, status)
  File "/home/hok/anaconda2/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/hok/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: ~/anaconda2/lib/python2.7/site-packages/tensorflow/models-master/tutorials/rnn/simple-examples/data/ptb.train.txt

Then I replaced ~ with /home/username/, and it worked properly.

Why couldn't I use ~ instead of /home/username/ to point to the file path when runing a RNN example?

Could you tell me in detail?

wjandrea
  • 14,236
  • 4
  • 48
  • 98
JNing
  • 603
  • http://stackoverflow.com/questions/1685894/home-directory-expansion-within-an-argument – muru Mar 07 '17 at 08:25
  • Does ~ always equal $HOME – Stéphane Chazelas Mar 07 '17 at 10:51
  • @OskarSkog Shouldn't the shell expand the ~ before the argument is passed to python? Just like the shell would expand backslash escapes in the path, or remove quotes if the path was quoted. – micheal65536 Mar 08 '17 at 14:17
  • 2
    Unlike $VARIABLES, the ~ is only expanded at the beginning of a string. – alexis Mar 08 '17 at 14:54
  • @OskarSkog, "Python does not know what ~ means" implies that an issue is specific to Python lacking a piece of functionality, setting up an unreasonable expectation that such functionality (of performing expansion after being exec'd) should be widely available in UNIX tools. – Charles Duffy Mar 08 '17 at 16:10
  • @OskarSkog, similarly, "in python, it's like everything is quoted with single quotes" is just untrue if that's intended to be a comparison against any other tool. echo ~ and python ~ will both expand the argument in precisely the same way, and echo --foo=~ and python --foo=~ will both fail to expand in exactly the same way, but "it's like everything is quoted with single quotes" reads as if Python somehow suppresses shell expansion behavior that would otherwise happen pre-invocation. – Charles Duffy Mar 08 '17 at 16:12
  • @OskarSkog, ...so I literally don't know how your statement can be read to not be claiming that Python is somehow at fault for this behavior. (For someone coming from Windows, where command-line parsing from a string into an argument vector is done by the command being invoked rather than the shell doing the invoking, the difference in behavior being on a per-executable basis can even make sense!) – Charles Duffy Mar 08 '17 at 16:16
  • Difference between $HOME and ~, Why can't I cd to a quoted tilde ('~')?, Why doesn't the tilde (~) expand inside double quotes? – phuclv Dec 02 '19 at 10:11
  • 1
    Possible duplicate of Why can't I cd to a quoted tilde ('~')? – phuclv Dec 02 '19 at 10:12
  • 2
    @phuclv The reason tilde expansion isn't performed here has nothing to do with quoting, so I don't think this is a duplicate of a question about how quoting suppresses tilde expansion. There is some overlap in material that would answer these questions, but I don't think they qualify as duplicates. – Eliah Kagan Dec 02 '19 at 18:20

4 Answers4

51

You need to understand that ~ is normally expanded by the shell; the programs you call never see it, they see the full pathname as inserted by bash. But this only happens when the tilde is at the start of an argument (and is not quoted).

If the Python program you are running uses a module like getopt to parse its commandline, you can give the argument of the --data-path option as a separate "word" to allow tilde expansion:

$ python ptb_word_lm.py --data_path ~/anaconda2/lib/python2.7/...

In your own code, you can use getopt or argparse for argument processing, and could also manually expand tildes as @JacobVlijm's answer suggested.

PS. The tilde is also expanded at the start of a shell variable assignment expression like DIRNAME=~/anaconda2; although the tilde in your question also follows an equals sign, this usage doesn't have special meaning for the shell (it's just something passed to a program) and doesn't trigger expansion.

alexis
  • 1,028
39

Tilde expansion in python

The answer is short & simple:

python does not expand ~ unless you use:

import os
os.path.expanduser('~/your_directory')

See also here:

os.path.expanduser(path)
On Unix and Windows, return the argument with an initial component of ~ or ~user replaced by that user‘s home directory.

On Unix, an initial ~ is replaced by the environment variable HOME if it is set; otherwise the current user’s home directory is looked up in the password directory through the built-in module pwd. An initial ~user is looked up directly in the password directory.

Jacob Vlijm
  • 83,767
  • 12
    In general, you should never assume that tilde expansion is done at an OS level, it is something that unix shells (and not all of them!) do for you. – farsil Mar 07 '17 at 10:10
  • 1
    I think the more relevant issue is lined out in alexis' answer: the position of ~ in the shell argument list. – David Foerster Mar 07 '17 at 12:28
  • @farsil, I disagree. Programs can be made portable, but when you run them from the command line, you do so on a specific system. And let's not forget that this is askubuntu.com, and Ubuntu is always Unix (as far as we know :-) – alexis Mar 07 '17 at 12:52
  • 1
    @alexis: Ubuntu doesn't do tilde expansion at OS level either. It's still shell functionality. – user2357112 Mar 07 '17 at 21:59
  • 1
    Methinks you're splitting hairs. Nobody said the kernel is doing it. The point is, it's not done by the program that takes the arguments. – alexis Mar 07 '17 at 23:45
  • I agree with David: while the answer is proper as far as Python goes, the key issue here is tilde expansion done by the shell (or rather the lack of it in OP's case due to tilde's placement). Now, it should also be pointed out, alexis, that tilde expansion can be done by the program (as is shown in the documentation quote Jacob provided) and can be of particular use when HOME variable is unset (which would be a rare case, but plausible). Also, while this is Ask Ubuntu and we can assume Unix-like environment safely, I think portability and posix-compliance should be upheld more – Sergiy Kolodyazhnyy Mar 08 '17 at 06:48
  • Posix compliance, sure. I didn't mean to argue for Ubuntu-only features, just that it's ok to assume Unix. And tilde expansion is not a Windows feature, so there's no expectation that it ought to work there. – alexis Mar 08 '17 at 08:37
  • @alexis: You said "I disagree" to the statement "you should never assume that tilde expansion is done at an OS level". That certainly sounds like you're saying the OS does it. – user2357112 Mar 08 '17 at 19:59
13

Tilde expansion is only done in a few contexts that vary slightly between shells.

While it is performed in:

var=~

Or

export var=~

in some shells. It's not in

echo var=~
env var=~ cmd
./configure --prefix=~

in POSIX shells.

It is in bash though when not in POSIX conformance mode (like when called as sh, or when POSIXLY_CORRECT is in the environment):

$ bash -c 'echo a=~'
a=/home/stephane
$ POSIXLY_CORRECT= bash -c 'echo a=~'
a=~
$ SHELLOPTS=posix bash -c 'echo a=~'
a=~
$ (exec -a sh bash -c 'echo a=~')
a=~

However that's only when what's on the left of the = is shaped like an unquoted valid variable name, so while it would be expanded in cmd prefix=~, it would not be in cmd --prefix=~ (as --prefix is not a valid variable name) nor in cmd "p"refix=~ (because of that quoted p) nor in var=prefix; cmd $var=~.

In zsh, you can set the magic_equal_subst option for ~ to be expanded after any unquoted =.

$ zsh -c 'echo a=~'
a=~
$ zsh -o magic_equal_subst -c 'echo a=~'
a=/home/stephane
$ zsh -o magic_equal_subst -c 'echo --a=~'
--a=/home/stephane

In the case of ~ (as opposed to ~user), you can just use $HOME instead:

cmd --whatever="$HOME/whatever"

~ expands to the value of $HOME. If $HOME is not set, behaviour varies between shells. Some shells query the user database. If you want to take that into account, you could do (and that's also what you would have to do for ~user):

dir=~ # or dir=~user
cmd --whatever="$dir/whatever"

In any case, in shells other than zsh remember you need to quote variable expansions!

Stéphane Chazelas
  • 1,366
  • 1
    Bash's reference manual seems to say that tildes are expanded only on variable assignments and at the start of a word, so expanding it onecho a=~ seems to contradict the manual. – ilkkachu Mar 08 '17 at 10:21
  • @ilkkachu, yes the manual is incomplete. It also doesn't specify clearly in what context ~ will be expanded (what is meant by "word"). See the link at the top of the answer for more details. – Stéphane Chazelas Mar 08 '17 at 16:25
7

~ has particular expansion rules, which your command doesn't satisfy. Specifically, it is expanded only when unquoted, either at the beginning of a word (e.g. python ~/script.py) or at the beginning of a variable assignment (e.g. PYTHONPATH=~/scripts python script.py). What you have is --data_path=~/blabla which is a single word in shell terms, so expansion is not performed.

An immediate fix is to use $HOME shell variable, which follows regular variable expansion rules:

python ptb_word_lm.py --data_path=$HOME/blabla
Dmitry Grigoryev
  • 1,942