LoadError: InitError during Package Build

[pedromxavier]

Thanks again for this package!

I'm currently trying to build my recently released package Anneal.jl and this error is showing up during precompilation. One must run ] precompile to see the message below:

ERROR: LoadError: InitError: failed process: Process(`'C:\Users\pedroxavier\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p 'C:\Users\pedroxavier\.julia\environments\v1.7\.CondaPkg\env' --override-channels --no-channel-priority pip 'python >=3.5,<4' 'python >=3.7,<4' -c anaconda -c conda-forge`, ProcessExited(3221226505)) [3221226505]

A way to fix this is to plug the command between backticks in a Julia run call.

run(`'C:\Users\pedroxavier\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\pedroxavier\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -y -p 'C:\Users\pedroxavier\.julia\environments\v1.7\.CondaPkg\env' --override-channels --no-channel-priority pip 'python >=3.5,<4' 'python >=3.7,<4' -c anaconda -c conda-forge`)

This way, micro mamba gets installed and everything works fine. The problem is that my package won't precompile directly without this trick. I guess it has something to do with privileges, chmod or something like that.

To reproduce this issue, it is enough to run

] add Anneal
] precompile

Anneal is available in the General Registry, so the first line will error after a little while and ] precompile will reveal the message above.

[cjdoris]

How strange. I can reproduce this on your package, and on a blank package also using PythonCall.

There are a few ways forward:

1. Figure out why the micromamba call is failing. I have run it again with really high verbosity, and the output looks normal until it stops, no error messages or anything. Looks like it stops just before it tries to download the conda repodata. It's not clear if this is even a problem with CondaPkg or Julia or MicroMamba.
2. If it is possible to detect that a package is being loaded just to be precompiled, I could add a check for this into PythonCall and have it skip most of its initialisation.
3. Modify PythonCall to not do any initialisation until you actually do something with it. This is something I've mused about for a while, and is doable, and would work around this issue.

Options 2 and 3 are really just workarounds, but might be nice to have anyway. Option 1 is the real solution, but I'm not sure how to approach it, I may need to ask for help on discourse.

[pedromxavier]

Lazy installation as in 3 seems to be the easiest path to follow, but also one of the most painful issues I've dealt with while trying to use PyCall.

I've runned as admin and the problem is still there.

There is one more weird fact: CI's are running just fine!

[cjdoris]

You don't need to run the micromamba command yourself - even if ] precompile fails you can still do using Anneal and it will precompile successfully. I think the issue is isolated to precompiling packages ahead of time, as opposed to precompiling them just-in-time when you actually use them. This probably explains why CI works OK - it just doesn't precompile ahead of time.

Option 3 is certainly doable, but will be the most disruptive and I don't want to rush into it. Options 1 and 2 solve the problem without changing any outward behaviour so I'd rather do those first if possible.

[cjdoris]

BTW this only affects Windows - on Mac and Linux it precompiles fine. Though it has pointed out an undesirable effect, which is that CondaPkg resolves all its dependencies while precompiling, which hides its logs.

[pedromxavier]

Hello again, @cjdoris!

I've noticed that some of PythonCall's dependecies had new releases and tried to build my packages again, having no trouble this time 😄. Do you think this issue is solved for good?

[cjdoris]

No idea but I'm glad it's working! I didn't do anything specifically to fix this. What updated?

[pedromxavier]

Hmmm then it must have something to do with some kind of caching 🤔 I installed them in many environments within v1.7 and also on another installation of v1.6 (both on windows) and no problems occured

[cjdoris]

The precompiled module will be cached in ~/.julia/compiled - deleting the right directory in there (or the whole directory) should reproduce the issue.

[cjdoris]

Unless it's magically fixed. I possibly updated the version of MicroMamba bundled with MicroMamba.jl since you filed this issue, that may have solved it.

[YuliyaCl]

Hello!
I have the same error during adding PythonCall to the project - Precompiling of PythonCall failed with the same error as in the issue.
I've tried to fix it using pedromxavier's recommendation but the error still exist.

The full error text:

Failed to precompile testWFDB [c64484d1-08e2-4c4d-8ee9-fe33f95c1a54] to C:\Users\yzh\.julia\compiled\v1.7\testWFDB\jl_1FAC.tmp.
    CondaPkg Found dependencies: C:\Users\yzh\.julia\packages\PythonCall\XgP8G\CondaPkg.toml
    CondaPkg Resolving changes
             + python
    CondaPkg Installing packages
warning  libmamba Cannot lock 'C:\Users\yzh\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
    Waiting for other mamba process to finish
error    libmamba Path already locked by the same PID: 'C:\Users\yzh\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
warning  libmamba LockFile creation for path 'C:\Users\yzh\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache' failed, continuing without it
error    libmamba Path already locked by the same PID: 'C:\Users\yzh\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\cache'
ERROR: LoadError: InitError: failed process: Process(`'C:\Users\yzh\.julia\artifacts\d628738c351341856bc6e8abe94cd6c8d8a7bf66\Library\bin\micromamba.exe' -r 'C:\Users\yzh\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root' create -q -y -p 'c:\Users\yzh\.julia\dev\testWFDB\.CondaPkg\env' --override-channels --no-channel-priority "python[version='>=3.5,<4']" -c conda-forge`, ProcessExited(3221226505)) [3221226505]

Stacktrace:
  [1] pipeline_error
    @ .\process.jl:531 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base .\process.jl:446
  [3] run(::Cmd)
    @ Base .\process.jl:444
  [4] _run(io::IO, cmd::Cmd, args::Any; flags::Any)
    @ CondaPkg C:\Users\yzh\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:283
  [5] _resolve_conda_install(io::Any, conda_env::Any, specs::Any, channels::Any; create::Any)
    @ CondaPkg C:\Users\yzh\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:191
  [6] resolve(; force::Bool, io::IO, interactive::Bool, dry_run::Bool)
    @ CondaPkg C:\Users\yzh\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:395
  [7] resolve()
    @ CondaPkg C:\Users\yzh\.julia\packages\CondaPkg\Olo5Y\src\resolve.jl:288
  [8] envdir()
    @ CondaPkg C:\Users\yzh\.julia\packages\CondaPkg\Olo5Y\src\env.jl:70
  [9] init_context()
    @ PythonCall.C C:\Users\yzh\.julia\packages\PythonCall\XgP8G\src\cpython\context.jl:56
 [10] __init__()
    @ PythonCall.C C:\Users\yzh\.julia\packages\PythonCall\XgP8G\src\cpython\CPython.jl:21
 [11] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base .\loading.jl:768
 [12] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base .\loading.jl:854
 [13] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1097
 [14] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1013
 [15] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:997
 [16] include
    @ .\Base.jl:418 [inlined]
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1318
 [18] top-level scope
    @ none:1
 [19] eval
    @ .\boot.jl:373 [inlined]
 [20] eval(x::Expr)
    @ Base.MainInclude .\client.jl:453
 [21] top-level scope
    @ none:1
during initialization of module C
in expression starting at c:\Users\yzh\.julia\dev\testWFDB\src\testWFDB.jl:1
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types C:\Users\yzh\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\Types.jl:68
 [2] precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API C:\Users\yzh\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1362
 [3] precompile
   @ C:\Users\yzh\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1013 [inlined]
 [4] #precompile#220
   @ C:\Users\yzh\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1011 [inlined]
 [5] precompile()
   @ Pkg.API C:\Users\yzh\AppData\Local\Programs\Julia-1.7.1\share\julia\stdlib\v1.7\Pkg\src\API.jl:1011
 [6] top-level scope
   @ REPL[10]:1

Do you have any ideas what I need to check...? I've used PyCall.jl in another project, maybe some conflicts occured.

Thanks.

UPD 11/04/22: deleting of cache folders inside ....\.julia\scratchspaces solved the problem.