Skip to content

feat(conformance): Add test execution instruction to the guide. #878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat(conformance): Add test execution instruction to the guide. #878

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
8c9463f
Add test execution instruction to the guide.
SinaChavoshi May 27, 2025
7ce6fd1
separate test setup to a different read me file.
SinaChavoshi May 27, 2025
6310fdf
fix prerequisites
SinaChavoshi May 30, 2025
8190e64
update overview
SinaChavoshi May 30, 2025
5794920
remove cluster Setup and Verification
SinaChavoshi May 30, 2025
2ec4965
use your_gatewayclass_name
SinaChavoshi May 30, 2025
fee10bf
move guide to site-src/guides/
SinaChavoshi May 30, 2025
16d8ebc
add a link from implementers guide
SinaChavoshi May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions conformance/reports/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This directory stores conformance reports submitted by various implementations o

This folder stores conformance reports organized first by the version of the Gateway API Inference Extension specification they were tested against, and then by the specific conformance profile (e.g., Gateway, EPP, Model Server):

```
|-- conformance/reports
| |-- v0.3.0 # Example extension version
| | |-- gateway # Conformance profile/category
Expand All @@ -22,6 +23,7 @@ This folder stores conformance reports organized first by the version of the Gat
| | | |-- ...
| |-- v0.4.0 # Future extension version
| | |-- ...
```

## Implementation Submissions

Expand Down
44 changes: 44 additions & 0 deletions site-src/guides/conformance-tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@

Copy link
Contributor

@danehans danehans May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mkdocs cannot find this doc without updating mkdocs.yml. Here's a docs structure to consider:

diff --git a/mkdocs.yml b/mkdocs.yml
index 1741fd1..675d837 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -65,7 +65,9 @@ nav:
           - Adapter Rollout: guides/adapter-rollout.md
           - InferencePool Rollout: guides/inferencepool-rollout.md
       - Metrics: guides/metrics.md
-    - Implementer's Guide: guides/implementers.md
+    - Implementer Guides:
+      - Getting started: guides/implementers.md
+      - Conformance Tests: guides/conformance-tests.md
   - Performance:
     - Benchmark: performance/benchmark/index.md
     - Regression Testing: performance/regression-testing/index.md

# Test Setup and Execution

This document provides steps to run the Gateway API Inference Extension conformance tests.

## Prerequisites: External Istio/Envoy Setup
Copy link
Contributor

@danehans danehans May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace Prerequisites: External Istio/Envoy Setup with Prerequisites


Before running the conformance tests, you need a Kubernetes cluster (e.g., GKE) with [LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer) support. Refer to [The Quickstart Guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/) for more details.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove (e.g., GKE)

Copy link
Contributor

@danehans danehans May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Running the EPP or EPP mock is a prerequisite. See Conformance: Add Mock EPP #893 for details. Leave this prerequisite as a TODO with a link to the issue so we can merge this PR.

Since this section will contain multiple prerequisites, consider using a numbered list.


## Running the Conformance Tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: s/Running the Conformance Tests/Running Conformance Tests/


1. **Choose an Implementation**:
Install an [existing implementation](https://gateway-api-inference-extension.sigs.k8s.io/implementations/gateways/) or run tests against your implementation after completing the [implementer's guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/implementers/#implementers-guide).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO this step should be in the Prerequisites section.


2. **Clone the Repository**:
Create a local copy of the Gateway API Inference Extension repository:
```bash
git clone https://github.com/kubernetes-sigs/gateway-api-inference-extension.git
cd gateway-api-inference-extension
```

3. **Execute Tests**:
Run the following command to execute all available tests. Replace <your_gatewayclass_name> with the GatewayClass used by the implementation under test.
Copy link
Contributor

@danehans danehans May 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/<your_gatewayclass_name>/<your_gatewayclass_name>/ (otherwise the text does not get rendered by mkdocs).


```bash
go test ./conformance -args -gateway-class <your_gatewayclass_name>
```

### Test Execution Options

* **Speeding up Reruns**: For repeated runs, you can add the flag `-cleanup-base-resources=false`. This will preserve resources such as namespaces and gateways between test runs, speeding up the process.
```bash
go test ./conformance -args -gateway-class <your_gatewayclass_name> -cleanup-base-resources=false
```

* **Running Specific Tests**: To run a specific test, you can reference the test name by using the `-run-test` flag. For example:
```bash
go test ./conformance -args -gateway-class <your_gatewayclass_name> -run-test HTTPRouteMultipleGatewaysDifferentPools
```

* **Detailed Logging**: To view detailed logs, you can enable logging mode by adding the `-v` as well as `-debug` flags.
```bash
go test -v ./conformance -args -debug -gateway-class <your_gatewayclass_name> -cleanup-base-resources=false -run-test HTTPRouteMultipleGatewaysDifferentPools
```
2 changes: 1 addition & 1 deletion site-src/guides/implementers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,4 @@ Here are some tips for testing your controller end-to-end:

### Conformance Tests

A set of conformance tests will be developed soon to help verify that a controller is working as expected. This guide will be updated once we have more information. Stay tuned!
See [Conformance Test Setup and Execution](https://gateway-api-inference-extension.sigs.k8s.io/guides/conformance-tests).