Feedback for dynamic tool selection 🚀

[SamMorrowDrums]

📢 🔔 we have been experimenting with the ability to let the LLM find and select tools from the GitHub MCP server, rather than having them all enabled up-front.

The idea here is that too many tools causing LLM models to experience tool confusion, and we have a lot of tools to add to cover our variety of GitHub products.

With --dynamic-toolsets enabled, you start with only 4 tools:

You try something like:

Can you search for issues assigned to me on GitHub?

And it will check and enable the tools it needs, so you end up with the tools you need enabled:

We want your feedback 🙏

• Did you encounter any issues?
• What software and what models did you try it with?
• What did you try to achieve?

Brief Demo

trim.DDCAF156-384D-42BA-9DC6-BFA970AEB077.MOV

To test this out

Use the :latest or v0.2.0 docker tag, and add the dynamic toolsets env:

Image: ghcr.io/github/github-mcp-server:latest
Env: GITHUB_DYNAMIC_TOOLSETS=1

Make sure you run the following to upgrade the server when using Docker

docker pull ghcr.io/github/github-mcp-server:latest

You can create or edit your configuration to try it out. Here is an example of a .vscode/mcp.json file in your workspace that will enable the feature:

{
    "inputs": [
        {
            "id": "github-pat",
            "type": "promptString",
            "description": "Github Personal Access Token",
            "password": true
        }
    ],
    "servers": {
        "github-mcp-server": {
            "type": "stdio",
            "command": "docker",
            "args": [
                "run", "-i", "--rm", "-e", "GITHUB_DYNAMIC_TOOLSETS=1", "-e", "GITHUB_PERSONAL_ACCESS_TOKEN", "ghcr.io/github/github-mcp-server:main",
                "./github-mcp-server", "stdio"
            ],
            "env": {
                "GITHUB_PERSONAL_ACCESS_TOKEN": "${input:github-pat}"
            }
        },
    }
}       

Tool Configuration

The GitHub MCP Server supports enabling or disabling specific groups of functionalities via the --toolsets flag. This allows you to control which GitHub API capabilities are available to your AI tools.

Available Toolsets

The following sets of tools are available (all are on by default):

Toolset	Description
repos	Repository-related tools (file operations, branches, commits)
issues	Issue-related tools (create, read, update, comment)
users	Anything relating to GitHub Users
pull_requests	Pull request operations (create, merge, review)
code_security	Code scanning alerts and security features
experiments	Experimental features (not considered stable)

Specifying Toolsets

To reduce the available tools, you can pass an allow-list in two ways:

1. Using Command Line Argument:

   github-mcp-server --toolsets repos,issues,pull_requests,code_security

2. Using Environment Variable:

   GITHUB_TOOLSETS="repos,issues,pull_requests,code_security" ./github-mcp-server

The environment variable GITHUB_TOOLSETS takes precedence over the command line argument if both are provided.

Any toolsets you specify will be enabled from the start, including when --dynamic-toolsets is on.

You might want to do this if the model is confused about which tools to call and you only require a subset.

Using Toolsets With Docker

When using Docker, you can pass the toolsets as environment variables:

docker run -i --rm \
  -e GITHUB_PERSONAL_ACCESS_TOKEN=<your-token> \
  -e GITHUB_TOOLSETS="repos,issues,pull_requests,code_security,experiments" \
  ghcr.io/github/github-mcp-server

The "all" Toolset

The special toolset all can be provided to enable all available toolsets regardless of any other configuration:

./github-mcp-server --toolsets all

Or using the environment variable:

GITHUB_TOOLSETS="all" ./github-mcp-server

Dynamic Tool Discovery

Instead of starting with all tools enabled, you can turn on Dynamic Toolset Discovery.
This feature provides tools that help the MCP Host application to discover and enable sets of GitHub tools only when needed.
This helps to avoid situations where models get confused by the shear number of tools available to them, which varies by model.

Using Dynamic Tool Discovery

When using the binary, you can pass the --dynamic-toolsets flag.

./github-mcp-server --dynamic-toolsets

When using Docker, you can pass the toolsets as environment variables:

docker run -i --rm \
  -e GITHUB_PERSONAL_ACCESS_TOKEN=<your-token> \
  -e GITHUB_DYNAMIC_TOOLSETS=1 \
  ghcr.io/github/github-mcp-server

Let us know how you get on!

[gillisandrew]

Could there also be filtering of tools based on the scope of the token? Could improve the tool selection of the agent and reduce the chances of encountering authorization errors.

[SamMorrowDrums]

The official release of this is out (although feedback is still being gathered) https://github.com/github/github-mcp-server/releases/tag/v0.2.0