JuliaCall reinstalling PythonCall every time

[henry2004y]

Hi,

Two users of my Julia pkg via JuliaCall have encountered the same issue when multithreading on the Python side was involved. They were mainly Python users who haven't used Julia before. After they installed JuliaCall via pip install juliacall, they had no issue calling Julia in a serial program. However, when they used multithreading on the Python side, they told me Julia tried to repeatly install PythonCall everytime with lengthy error messages.

The easy fix would be to manually install PythonCall in Julia. I haven't encountered this issue myself previously because I have all the pkgs already installed on the Julia side.

Is this behaviour expected? Or should we mention this in the doc?

[henry2004y]

Maybe I got wrong info. Will reopen this if I can reproduce the issue. From what I've checked, installing PythonCall in Julia is needed to avoid adding the pkg every time, but I don't think this has anything to do with multithreading.

[henry2004y]

I tested with several different setups. On our cluster with multiple Python version installed, I selected the one that Julia used and use that specific pip to install JuliaCall. By adding PythonCall in Julia I got no issues. Interestingly, if I add this Python version to my $PATH with export PATH=/proj/$USER/miniconda3/envs/conda_jl/bin:$PATH, I see the following checking message every time:

Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from juliacall import Main as jl
[juliapkg] Locating Julia ^1.6
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] Using Julia 1.7.2 at /home/hongyang/proj/bin/julia
[juliapkg] Using Julia project at /home/hongyang/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`

I feel like this has something to do with other executable under the same path, e.g. pip or conda. But I'm not exactly sure. This may take 4 ~ 20 s each time.

[cjdoris]

Hmm I've occasionally had this issue in the past but can't seem to reproduce it right now. Can you update juliacall and juliapkg to their latest versions (0.8.0 and 0.1.3), run

import logging; logging.basicConfig(level=logging.DEBUG); import juliacall

and send me the output please?

[henry2004y]

Hmm, not sure if I've done this correctly, but I see no difference compared to from juliacall import Main as jl

Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging; logging.basicConfig(level=logging.DEBUG); import juliacall
[juliapkg] Locating Julia ^1.6
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] Using Julia 1.7.2 at /home/hongyang/proj/bin/julia
[juliapkg] Using Julia project at /home/hongyang/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`
>>> exit()

And randomly Python may be happy without showing me this.

[cjdoris]

Are you sure juliapkg is at v0.1.3? It should output something like

DEBUG:juliapkg:no meta file
[juliapkg] Locating Julia ^1.6
......

[henry2004y]

Sorry I used juliapkg v0.1.2. I upgraded to v0.1.3 via pip install juliapkg --upgrade, and here's the output in a new login session

hongyang@turso02:~$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging; logging.basicConfig(level=logging.DEBUG); import juliacall
DEBUG:juliapkg:sys.path changed ['', '/proj/hongyang/miniconda3/envs/conda_jl/bin', '/proj/hongyang/.local', '/home/hongyang/proj/analysator', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python39.zip', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python3.9', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python3.9/lib-dynload', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python3.9/site-packages'] to ['', '/home/hongyang', '/proj/hongyang/.local', '/home/hongyang/proj/analysator', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python39.zip', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python3.9', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python3.9/lib-dynload', '/proj/hongyang/miniconda3/envs/conda_jl/lib/python3.9/site-packages']
[juliapkg] Locating Julia ^1.6
[juliapkg] Using Julia 1.7.2 at /home/hongyang/proj/bin/julia
[juliapkg] Using Julia project at /home/hongyang/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`
>>> exit()
hongyang@turso02:~$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging; logging.basicConfig(level=logging.DEBUG); import juliacall
DEBUG:juliapkg:directory changed '/home/hongyang'
[juliapkg] Locating Julia ^1.6
[juliapkg] Using Julia 1.7.2 at /home/hongyang/proj/bin/julia
[juliapkg] Using Julia project at /home/hongyang/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Project.toml`
  No Changes to `/proj/hongyang/.julia/environments/pyjuliapkg/Manifest.toml`

[cjdoris]

Ah right ok, so juliapkg re-resolves whenever any directory in sys.path gets modified, in order to detect changes when packages are installed. You have your home directory in sys.path, and this gets modified frequently, hence resolving every time.

Presumably you have your home directory in PYTHONPATH, I'd recommend removing this.

[henry2004y]

Thank you! I can confirm that after removing my previous PYTHONPATH settings I don't see this resolving message anymore. I will share this information with my colleague to see if this fix his issue as well.