Add In-Memory Repository Support

[MicahZoltu]

I am building an application that needs to interact with git (clone repo, create repo, push, etc.) but does not have direct access to a disk. I would like to be able to work with a git repo entirely in-memory.

As an aside, I would also like to be able to work with a git repository that is written out to a non-file datastore such as a database.

In general, it would be nice if the logic in Repository was abstracted out away from the disk. It appears that IRepository is not tightly coupled with disk (assuming all of the various methods that take a path are referring to a relative path). However, there appears to be quite a bit of git specific logic bound up in the Repository implementation which makes it difficult to a novice contributor to implement a custom datastore engine (e.g., in-memory, database, etc.).

Researching this topic very briefly, it appears that the underlying libgit2 supports non-filesystem repositories, so I don't believe this is a technical limitation with libgit2.

[nulltoken]

@Zoltu Indeed, libgit2 exposes the concepts of pluggable backends: to my knowledge there are 3 of them: object database, reference database and configuration storage.

• The object database backend is already available through LibGit2Sharp (though the API may still be improved). You can find a sample implementation at https://github.com/nulltoken/libgit2sharp.voron
• The reference database isn't done yet. There's an ongoing PR (Expose libgit2 reference database backend for managed code refdb impls. #401) to work on this, but it's very outdated.
• Nothing exist yet regarding the configuration storage from the LibGit2Sharp standpoint.

Repository implementation which makes it difficult to a novice contributor to implement a custom datastore engine (e.g., in-memory, database, etc.).

Would you feel like giving a hand, we'd be more than happy in helping you making this happen!

[MicahZoltu]

References:
https://github.com/libgit2/libgit2-backends
http://blog.deveo.com/your-git-repository-in-a-database-pluggable-backends-in-libgit2/
https://github.com/libgit2/libgit2/blob/master/include/git2/sys/odb_backend.h
https://github.com/libgit2/libgit2/blob/master/include/git2/sys/refdb_backend.h

[nulltoken]

https://github.com/libgit2/libgit2/blob/master/include/git2/sys/config.h

[MicahZoltu]

I started reviewing the code for implementation and I wondered what would be the most "clean" example of the correct way to do the C/C# interop for this project? Looking over the code, I see a few subtly different patterns at work and want to be a good citizen if I add my own.

If there is no well established pattern to follow my instinct is to build a layer that is as close as possible to the C API (including snake case naming convention), probably built with an interop code generator. Even the source files in this layer would map to the libgit2 source files, rather than the C# convention of one file per class/struct.

For structures that hold callbacks I would put delegates in the structure (rather than separate) in order to keep the mapping from C# to C as direct as possible. For functions I would put them in the same file as the structures (to match the libgit2 C way of doing things) and wrap them all up in a partial NativeMethods class.

After this I would then create C# constructs that hide away all of the C memory management and follow more reasonable C# paradigms. Finally, a layer on top of that which exposes things in a user friendly way.

However, I haven't done interop for a long time so this way of doing things may be sub-par and I want to be a good contributor (increase chance of PR acceptance) if I decide to follow-through with this feature.

[nulltoken]

@Zoltu The interop layer is indeed the less polished part of the codebase. The Proxy layer is there to avoid polluting too much the remaining parts of the code.

One thing about the callbacks vs delegates, we've had issues in the past when using callbacks. Those were hard to track and not easily reproducible. See the following for further references:

• https://github.com/libgit2/libgit2sharp/pull/348/files

• libgit2sharp/LibGit2Sharp/Network.cs

  Lines 106 to 113 in 8c48d39

  	// It is OK to pass the reference to the GitCallbacks directly here because libgit2 makes a copy of
  	// the data in the git_remote_callbacks structure. If, in the future, libgit2 changes its implementation
  	// to store a reference to the git_remote_callbacks structure this would introduce a subtle bug
  	// where the managed layer could move the git_remote_callbacks to a different location in memory,
  	// but libgit2 would still reference the old address.
  	//
  	// Also, if GitRemoteCallbacks were a class instead of a struct, we would need to guard against
  	// GC occuring in between setting the remote callbacks and actual usage in one of the functions afterwords.

However, I haven't done interop for a long time so this way of doing things may be sub-par and I want to be a good contributor (increase chance of PR acceptance) if I decide to follow-through with this feature.

I don't see any problem with your approach. Only one advice when it comes to interop: open your PR early and heavily rely on the CI servers. Most of the interop issues will be put under the light thanks to them.

[takeucda]

I'm curious, this issue seems a little outdated, but were there any updates?
I wanted to use a custom reference database backend to implement a file-free git repository in my system.

[LarsKemmann]

Second @takeucda -- we're looking at writing some custom backends using a document DB (Azure Cosmos DB) and it would be good to know if something is missing that would keep us from being able to do that. (And what we might have to do to hack around it. :))

[ethomson]

There should be no problem storing objects in a custom database, but we still haven't added reference or configuration support. libgit2 supports these, so you can always add these as native extensions if you want, or we'd love to see a PR for managed references / configuration support.

(#1226 has started the reference db support.)

[Zoxive]

I have a branch that revived #1226 and got all the tests working. Ive intended to make a PR just havent found time yet. Ill see if i can make some time to get it created.

[ethomson]

@Zoxive 👍 That would be awesome if you would push that up at your leisure! 😁

[LarsKemmann]

@ethomson Is it that the required interfaces don't exist yet in libgit2sharp, or what would be included in implementing that?

[Zoxive]

@LarsKemmann libgit2sharp does not have the implementations yet. libgit2 supports pluggable backends all already.

My branch adds support for the missing refs backend to libgit2sharp.
master...Zoxive:zoxive/refdb

I'll be making a PR soon when I get it updated to latest and tested.

[solvingj]

@Zoxive @ethomson , It looks like this got held up. Any chance this will make it in?