run vulnerablecode behind proxy

[tardyp]

Looking at deploying this in a proxy protected environment I figured that the import will not work behind proxy

the rootcause is pygit2 which does not support proxy. There is experimental support in the mainline, but it is not yet released.
libgit2/pygit2#1063

In general, I would respectfully question the use of pygit2 in favor gitpython.

Being backed by standard git makes it in my experience more reliable in all environments.

Would you accept a PR switching this dependency?

[sbs2001]

@tardyp

the rootcause is pygit2 which does not support proxy

That's a nice find. Needs to be fixed soon.

respectfully question the use of pygit2 in favor gitpython

I'm not super familiar with the pygit2 code we have here, but I guess at the time pygit2 was used due to existing familiarity it was used to sketch out a quick working prototype. We can always iterate and improve.

Would you accept a PR switching this dependency?

Yes !! That would be great.

@pombredanne ping to you

[tardyp]

PR done, with unit test working for now. Pubishing for early review

I'll be working this afternoon in running it with some real data

[pombredanne]

In general, I would respectfully question the use of pygit2 in favor gitpython.

Being backed by standard git makes it in my experience more reliable in all environments.

Would you accept a PR switching this dependency?

Sure thing. The alternative would be to possibly look into FetchCode https://github.com/nexB/fetchcode/ which is using the extracted guts of pip for the parts that deals with version control systems (it but cannot deal much with checkouts though, but may not be an issue) e.g. it uses the git CLI directly, but either way is fine.

In particular I am not sure FetchCode deals with proxies (yet)

[tardyp]

The data_source.py module actually has incremental fetch ability and change detection code.

However, it looks like the caller actually do not care, and working directories are always recreated from scratch for all runs of import.

This may not be a huge issue.

FWYI there is another proxy issue with the package code, which I will send a PR soon.

[Hritik14]

it looks like the caller actually do not care, and working directories are always recreated from scratch for all runs of import.

This is configurable here
https://github.com/nexB/vulnerablecode/blob/3688d89a88014f2e04475fda0179362e9ab54eae/vulnerabilities/data_source.py#L241-L243

[tardyp]

This is configurable here

right, but nobody uses it except for tests. Am I wrong?

[Hritik14]

All the parameters are passed via the data_source_cfg in importer_yielder.py. If desired, it could be set to not destroy the directories it is creating. It defaults to destroying them though.