Fail to `#include <git2.h>` in virtualenv

[Graham42]

Install process works in a virtualenv but when I say import pygit2 I get a stack trace telling me it can't import git2.h

With python 3:

<path/to/my/virtualenv>/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/__pycache__/_cffi__x4f8db466x390834cc.c:185:18: fatal error: git2.h: No such file or directory
 #include <git2.h>
                  ^
compilation terminated.
...

Same thing with python 2:

<path/to/my/virtualenv>/lib/python2.7/site-packages/pygit2-0.21.3-py2.7-linux-x86_64.egg/pygit2/__pycache__/_cffi__x245ec18bx390834cc.c:185:18: fatal error: git2.h: No such file or directory
 #include <git2.h>
                  ^
compilation terminated.
...

[uniphil]

this issue is over my head, but after some digging, I can get it a little further before it breaks

1. install libgit2 and pygit2 in a virtualenv
2. export the includes dir: (venv)$ export C_INCLUDE_PATH=$VIRTUAL_ENV/include
3. (venv)$ python -c "import pygit2" fails with a new error:

/usr/bin/ld: cannot find -lgit2
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
  File "/usr/lib64/python3.4/distutils/unixccompiler.py", line 194, in link
    self.spawn(linker + ld_args)
  File "/usr/lib64/python3.4/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/usr/lib64/python3.4/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/usr/lib64/python3.4/distutils/spawn.py", line 162, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/ffiplatform.py", line 48, in _build
    dist.run_command('build_ext')
  File "/usr/lib64/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 448, in build_extensions
    self.build_extension(ext)
  File "/usr/lib64/python3.4/distutils/command/build_ext.py", line 535, in build_extension
    target_lang=language)
  File "/usr/lib64/python3.4/distutils/ccompiler.py", line 717, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/usr/lib64/python3.4/distutils/unixccompiler.py", line 196, in link
    raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/__init__.py", line 35, in <module>
    from .config import Config
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/config.py", line 32, in <module>
    from .errors import check_error
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/errors.py", line 29, in <module>
    from .ffi import ffi, C
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/pygit2-0.21.3-py3.4-linux-x86_64.egg/pygit2/ffi.py", line 59, in <module>
    include_dirs=include_dirs, library_dirs=library_dirs)
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/api.py", line 340, in verify
    lib = self.verifier.load_library()
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/verifier.py", line 74, in load_library
    self._compile_module()
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/verifier.py", line 139, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/ffiplatform.py", line 25, in compile
    outputfilename = _build(tmpdir, ext)
  File "/home/phil/.virtualenvs/venvgitdev/lib/python3.4/site-packages/cffi-0.8.6-py3.4-linux-x86_64.egg/cffi/ffiplatform.py", line 51, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.ffiplatform.VerificationError: LinkError: command 'gcc' failed with exit status 1

edit: http://stackoverflow.com/questions/5329638/usr-bin-ld-cannot-find makes me think an argument needs to be given in the actual g++ command that cffi is generating. Can we do that with an environment variable or something?

[uniphil]

@Graham42 after skimming some pygit2 code I got it to work. It is required to have LIBGIT2 exported as an environment variable at run time, not just when installing.

(venv)$ export LIBGIT2=$VIRTUAL_ENV
(venv)$ python -c "import pygit2"
(venv)$ 

no errors.

This is a little bit annoying, as previously (with v0.20.x, I think) exporting $LIBGIT2 was only required at build time, and then things just worked.

[jdavid]

I am looking into this

[jdavid]

When installing pygit2 an extension is generated by cffi with a name like _cffi__xc9c1b883xa706f225.so
What happens is that at run time cffi does not find this extension, so it tries to generate it again, and then you get the build error from the original comment, since the include dirs are not properly defined.

The name of the extension module is a hash from a number of sources including the parameters passed to the cffi verify method, https://github.com/libgit2/pygit2/blob/master/pygit2/ffi.py#L52

In other words, for the name to be the same, so cffi finds the extension, the parameters passed to verify must be the same. That's solution number (1), to be sure, one way or another, that verify is called with the same parameters.

Solution number (2) is to use modulename, https://cffi.readthedocs.org/en/release-0.8/#reference-verifier but there is a caveat, https://cffi.readthedocs.org/en/release-0.8/#warning-about-modulename

[jdavid]

After some testing I have not found any practical caveat, every time I call build it regenerates the cffi extension, even when nothing changes.

So I have made the extension name constant, pygit2_cffi, and this fixes the issue for me. Could you give it a try?

[Graham42]

Works! Thanks very much!

[carlosmn]

After some testing I have not found any practical caveat, every time I call build it regenerates the cffi extension, even when nothing changes.

It never regenerates the extension for me (at least on OSX, which is where I'm working atm), even if the C header changes, so when adding functionality via cffi, you need to remember to clean the cache dir. It's definitely better to have easier deploys, but I just wanted to point out that it is some extra work when developing.

[jdavid]

Ok, I'm working on this.

[jdavid]

@carlosmn could you check it with latest commit?

[carlosmn]

Yep, now I can change decl.h without problems.

[wakenmeng]

excuse me, I got the cffi verification error and don't know where to find help. I run 'python setup.py build' on Mac os , virtualenv , and keep get the cffi verify error. Really appreciate it if you can give me some help. The same error when run pip install pygit2(v0.21.4).
same situation in ubuntu 12.04.
Here's the log:
cffi.ffiplatform.VerificationError: importing '/usr/local/var/lib/pyenv/versions/test_env/build/pygit2/pygit2/pycache/pygit2_cffi_18eab927xbf062fb5.so': dlopen(/usr/local/var/lib/pyenv/versions/test_env/build/pygit2/pygit2/pycache/pygit2_cffi_18eab927xbf062fb5.so, 2): Library not loaded: @rpath/libgit2.0.21.2.dylib

Referenced from: /usr/local/var/lib/pyenv/versions/test_env/build/pygit2/pygit2/pycache/pygit2_cffi_18eab927xbf062fb5.so

Reason: image not found

[wakenmeng]

I just solve the problem, the reason is I didn't deal well with the rpath on mac, on ubuntu it's easy which do what the installation doc says sudo ldconfig, I set the DYLD_LIBRARY_PATH to the libgit2 in virtuanenv , and it was solved. Sorry to bother, I'll leave this comment here to help someone 'dumb' like me.