Azure DevOps Testing Tracking Issue

[varsanojidan]

Note to testers

Suggested approach:

• Copy this template and paste a new comment on this issue. Update your comment as you progress with the testing. Add in-line notes so that we can triage this and follow up with issues as and where needed

Setting up Azure DevOps + Sourcegraph

1. Add this codehost config, the credentials can be found in our password manager (under Azure DevOps temporary test credentials) (it is encouraged to try creating your own test credentials as that is also part of the site admin workflow)

{
"AZUREDEVOPS": [
    {
      "url": "https://dev.azure.com",
      "username": "<username>",
      "token": "<token>",
      "projects": ["sgtestazure/sgtestazure"]
    }
  ]
}

Additionally add the following to your site config

"experimentalFeatures": {
    "azureDevOps": "enabled"
  },

2. Setup ngrok

Azure DevOps Repository syncing

• Verify specifying repositories in the codehost config either through orgs or projects both work for you, and return repos as expected.
• Play around with exclude, either exclude repos by name(projectName/repoName) or by pattern and verify it behaves accordingly.
• Add projects and repositories that have many spaces in the name (one already exists in the sgtestazure org, but you can also make your own). Make sure repositories are cloned as expected. Navigate through the Sourcegraph UI, clicking as many links as you can associated with these projects/repos, verify they all link correctly and display well.

Azure DevOps OAuth provider testing scenarios

• Configure Azure auth provider by following the docs (https://docs.sourcegraph.com/admin/config/authorization_and_authentication#azure-devops-services)
• Set allowSignup to false in the auth provider config
• Try to sign up with a new account on Sourcegraph, this should fail
• Remove allowSignup from the auth provider config
• Try to sign up with a new account on Sourcegraph, this should succeed
• Logout and login with an existing account on Sourcegraph
• Set allowOrgs to: ["some-org-that-does-not-exist"] in the auth provider config
• Try logging in to Sourcegraph with Azure DevOps, this should fail
• Add an org to allowOrgs that already exists and your Azure DevOps user is a part of, for example sgtestazure in the auth provider config
• Try logging in to Sourcegraph with Azure DevOps, this should succeed
• Visit Account security page and remove the Azure DevOps connection, this should succeed
• Now try to add the account connection back from the Account Security page, this should succeed

Azure DevOps permissions syncing testing scenarios

• Configure permissions syncing for Azure DevOps following the docs (https://docs.sourcegraph.com/admin/external_service/azuredevops#permissions-syncing)
• Create a new org (or use an existing one that is being synced) and add another user with Basic as the access level (See 1Password shared vault Source - Repository Management Vault for test credentials)
• Enable Third-party application access via OAuth for the org that you just created
• Sync the newly created org by updating the code host config
• Configure Azure DevOps code host connection to sync at least two orgs
• Verify that the test user can access the repos in this org
• Remove access from one of the projects in the org and verify that the repos of this project can not be accessed on Sourcegraph
• Change access level (under Organization settings -> Users) from Basic to Stakeholder on Azure DevOps, trigger a permissions sync and verify that the user can no longer access the repo of this org
• Change access level from Stakeholder to back to Basic, trigger a permissions sync and verify that the user can access the repo of this org again
• Remove the user from the org and verify that the user can no longer access the repo from this org
• Create a public repo in this org and verify that the user can access the repo even though they are not a part of this org

Azure DevOps Batch Changes

Setup:

1. Create an Azure DevOps Personal Access Token with the following scopes:

• Organization:All accessible organizations
• Code:Full
• Code:Status
• Pull Request Thread:Read & Write
• User Profile:Read

2. Run sg start batches in your local dev env.
3. On the sourcegraph UI, go to Settings -> Batch Changes, and add your credentials for ADO (same as above)
4. Create a new Batch Change
5. Use the example Hello World Batch Spec, hit Preview Workspaces, then Run Batch Spec
6. Setup BC webhooks for ADO following this guide.
7. Hit Preview, Apply, Select All, Publish Changesets, and enjoy playing around with some Batch Changes 😄

Scenarios:

• Setup the basic batch changes scenario described above, mess around with the changesets, publish as draft, create comments, close Pull Requests, Open Pull Requests, verify the PR in ADO as well as the Changeset in Batch Changes are updated correctly.
• Change the review state of the PR in ADO, make sure the review state in BC is updated automatically and correctly (via webhooks): Approved -> Approved, all else -> changes requested.
• Make an API call to create some check statuses on your PRs. This will not update the Changeset automatically, but verify that once you hit the refresh icon at the bottom of the Changeset, that the Checkstate displays correctly.
• Try different combinations of reviews/check states and make sure the Changeset always reflects it correctly.
• Try to retest the scenarios Kelli listed.
• Try to break it, do anything you think you can to try to produce an error, record down any errors or misbehavior you notice.

[mrnugget]

I just noticed that ADO doesn't show up in the list of code hosts for the new setup wizard:

Not a big problem, but you should talk to @vovakulikov to make sure this will work with the new setup experience flow if the feature flag is on.

[mrnugget]

Feedback on the new code host connection screen

Feedback on setting up auth provider

It's really confusing that it's called clientID in our schema but App ID on their end. I left a comment here https://github.com/sourcegraph/sourcegraph/pull/48822/files#r1129480449 saying the same thing.

[indradhanush]

Starting to test on Dogfood. Let's see how far I can get.

Azure DevOps Repository syncing

• Verify specifying repositories in the codehost config either through orgs or projects both work for you, and return repos as expected.
• Play around with exclude, either exclude repos by name(projectName/repoName) or by pattern and verify it behaves accordingly.
• Add projects and repositories that have many spaces in the name (one already exists in the sgtestazure org, but you can also make your own). Make sure repositories are cloned as expected. Navigate through the Sourcegraph UI, clicking as many links as you can associated with these projects/repos, verify they all link correctly and display well.

Azure DevOps OAuth provider testing scenarios

• Configure Azure auth provider by following the docs (https://docs.sourcegraph.com/admin/config/authorization_and_authentication#azure-devops-services)
• Set allowSignup to false in the auth provider config
• Try to sign up with a new account on Sourcegraph, this should fail
• Remove allowSignup from the auth provider config
• Try to sign up with a new account on Sourcegraph, this should succeed
• Logout and login with an existing account on Sourcegraph
• Set allowOrgs to: ["some-org-that-does-not-exist"] in the auth provider config
• Try logging in to Sourcegraph with Azure DevOps, this should fail
• Add an org to allowOrgs that already exists and your Azure DevOps user is a part of, for example sgtestazure in the auth provider config
• Try logging in to Sourcegraph with Azure DevOps, this should succeed
• Visit Account security page and remove the Azure DevOps connection, this should succeed
• Now try to add the account connection back from the Account Security page, this should succeed

Azure DevOps permissions syncing testing scenarios

• Configure permissions syncing for Azure DevOps following the docs (https://docs.sourcegraph.com/admin/external_service/azuredevops#permissions-syncing)
• Create a new org (or use an existing one that is being synced) and add another user with Basic as the access level (See 1Password shared vault Source - Repository Management Vault for test credentials)
• Enable Third-party application access via OAuth for the org that you just created
• Sync the newly created org by updating the code host config
• Configure Azure DevOps code host connection to sync at least two orgs
• Verify that the test user can access the repos in this org
• Remove access from one of the projects in the org and verify that the repos of this project can not be accessed on Sourcegraph
• Change access level (under Organization settings -> Users) from Basic to Stakeholder on Azure DevOps, trigger a permissions sync and verify that the user can no longer access the repo of this org
• Change access level from Stakeholder to back to Basic, trigger a permissions sync and verify that the user can access the repo of this org again
• Remove the user from the org and verify that the user can no longer access the repo from this org
• Create a public repo in this org and verify that the user can access the repo even though they are not a part of this org

Azure DevOps Batch Changes

Setup:

1. Create an Azure DevOps Personal Access Token with the following scopes:

• Organization:All accessible organizations
• Code:Full
• Code:Status
• Pull Request Thread:Read & Write
• User Profile:Read

2. Run sg start batches in your local dev env.
3. On the sourcegraph UI, go to Settings -> Batch Changes, and add your credentials for ADO (same as above)
4. Create a new Batch Change
5. Use the example Hello World Batch Spec, hit Preview Workspaces, then Run Batch Spec
6. Setup BC webhooks for ADO following this guide.
7. Hit Preview, Apply, Select All, Publish Changesets, and enjoy playing around with some Batch Changes 😄

Scenarios:

• Setup the basic batch changes scenario described above, mess around with the changesets, publish as draft, create comments, close Pull Requests, Open Pull Requests, verify the PR in ADO as well as the Changeset in Batch Changes are updated correctly.
• Change the review state of the PR in ADO, make sure the review state in BC is updated automatically and correctly (via webhooks): Approved -> Approved, all else -> changes requested.
• Make an API call to create some check statuses on your PRs. This will not update the Changeset automatically, but verify that once you hit the refresh icon at the bottom of the Changeset, that the Checkstate displays correctly.
• Try different combinations of reviews/check states and make sure the Changeset always reflects it correctly.
• Try to retest the scenarios Kelli listed.
• Try to break it, do anything you think you can to try to produce an error, record down any errors or misbehavior you notice.

List of bugs

• No helpful example from exclude key unlike the projects key

[indradhanush]

Bugs with exclude

1. Exclude org/project/repo not working

Edit: Using pattern worked

Edit2: This is working locally for me on latest main but not on dogfood. Maybe it's not on the latest change then. But the config editor still complains about the string not matching the pattern.

2. Exclude repo from the repository settings page does not work.

3. No helpful example from exclude key unlike the projects key

The same behaviour is also missing for the orgs key (not captured in GIF).

[sashaostrikov]

I did test setting a code host, everything works as expected 👍🏻
I also did a batch changes run without any problems.
Good stuff @varsanojidan @indradhanush!