Easier debugging / test-specific configurations

[jhildenbiddle]

Is your feature request related to a problem? Please describe.

I typically run all tests headless, but I often have failing tests that that require visually inspecting the page. Currently, this involves the following:

1. Find my jest-playwright.config.js file
2. Modify my browsers to include only one browser (since I generally don't need to visually inspect in multiple browsers)
3. Modify my launchOptions to set headless:false and occasionally devtools:true

Once the test passes, I then have to undo all of the changes made above.

What would greatly speed up this process is a way to launch a single test in non-headless mode, possibly with devtools enabled, for one-or-more browsers and/or devices. Ideally, this would be as quick as adding .only or .skip to a test, although this would obviously take some pre-configuration.

Describe the solution you'd like

The verbose-but-flexible way would be to allow passing configuration options with each test. Using the existing jestPlaywright.skip implementation as inspiration, perhaps something like this could work:

jestPlaywright.config({ 
  browser: 'chromium',
  launchOptions: {
    headless: false,
    devtools: true,
  }
}, () => {
  // Tests...
})

This could be made less obnoxious by storing the non-headless/debug configuration separately and passing it to the .config() method:

const jestPlaywrightDebugConfig = {
  browser: 'chromium',
  launchOptions: {
    headless: false,
    devtools: true,
  }
};

jestPlaywright.config(jestPlaywrightDebugConfig, () => {
  // Tests...
})

Better yet, the non-headless/debug configuration could be set in jest-playwright.config.js, allowing a much simpler .debug() method to be used without passing additional parameters:

// jest-playwrite.config.js
module.exports = {
  browsers: [
    'chromium',
    'firefox',
    'webkit',
  ],
  debugOptions: {
    browsers: [
      'chromium'
    ],
    launchOptions: {
      headless: false,
      devtools: true
    }
  }
};

// my.test.js
jestPlaywright.debug(() => {
  // Tests...
})

[mmarkelov]

@jhildenbiddle thank you for your suggestion. Seems like that it can be very helpful. I'll take it

[jhildenbiddle]

Excellent! Thanks for considering the change, @mmarkelov.

[mmarkelov]

@jhildenbiddle just look through this issue and made #233
You will be able to do something like that:

    test('test1', async () => {
        await jestPlaywright.config({
            browser: 'chromium',
            launchType: "LAUNCH",
            launchOptions: {
                headless: false,
                devtools: true,
            }
        }, async ({page}) => {
            // page is coming from function params
            await page.goto('https://github.com/')
            const title = await page.title()
            expect(title).toBe('Google')
        })
    })

So you can config playwright only inside the specific test. Personally I don't think that it's perfect decision, I just tried some another opportunities, like using jestPlaywright.config function inside describe, but cause it's sync. there is no ability to use it with async config function.

Also I don't know how we should config it, should we merge passing options with existing?

[mmarkelov]

@jhildenbiddle also we just discussed some another ways to implement this feature with @mxschmitt
Maybe we can test it with CLI, for example:

// tests:
describe('Simple tests',  () => {
    beforeAll(async () => {
        await page.goto('https://google.com/')
    })

    test('passed', async () => {
        const title = await page.title()
        await expect(title).toBe('Google')
    })

    test('failed', async () => {
        const title = await page.title()
        await expect(title).toBe('')
    })

    afterAll(async () => {
        await page.close()
    });
})

So you got one failed test for chromium for example.

After it you can run only failed test with DEBUG flag:

DEBUG=true BROWSER=chromium jest -t 'failed'

So jest-playwright will run with some constant debug options:

            launchOptions: {
                headless: false,
                devtools: true,
            }

So you will be able to also run tests for failed describe block for example.

But you couldn't pass another playwright options with your CLI, I suppose it won't be critical

[jhildenbiddle]

First, thank you @mmarkelov for your efforts. They are very much appreciated. Whatever we end up with, having the ability to modify configurations for describe and test blocks will be extremely helpful.

Both of the proposed implementations (new method & CLI option) are interesting because they serve different purposes: the config() method allows initiating debug mode for individual tests within test files while the CLI option allows scripts and IDE helpers to initiate debug mode for multiple describe and/or test blocks.

Implementation 1

test('test1', async () => {
  await jestPlaywright.config({
    // ...
  }, async ({page}) => {
    // ...
  })
});

This is essentially what I was hoping for. While it may be limited to single test blocks only, I still think this is approach has value because it allows me to run my tests in watch+headless mode, have a test fail, wrap the failed test in the code described above, then have access to a browser with dev tools ready for debugging. I didn't have to stop my watch task or deal with browser windows for every test after changing my main configuration. After I fix the test I just remove the jestPlaywright method and continue working.

As stated in the first comment though, I'd want to reuse a debug configuration automatically and simplify the code I need to wrap my tests with using a separate method. Passing a config is still useful in other scenarios though (like trying a different screen size or specifying a browser), so I can see value in both jestPlaywright.config (for adjusting the standard configuration per test) and a jestPlaywright.debug methods (for using a separate debug-focused config). Ideally the debug() method would also (optionally) accept a configuration as the first argument, allowing overrides of properties for the debug-focused config as well:

// Default debug configuration (stored in jest-playwright settings)
test('test1', async () => {
  await jestPlaywright.debug(async ({page}) => {
    // ...
  })
});

// Debug configuration with `browser` override
test('test2', async () => {
  await jestPlaywright.config({ browser: 'webkit' }, async ({page}) => {
    // ...
  })
});

// Regular configuration with `browser` override
test('test2', async () => {
  await jestPlaywright.config({ browser: 'chromium' }, async ({page}) => {
    // ...
  })
});

Other thoughts:

• What is the value of this inside of the jestPlaywright method? My concern is that this is often referred to within tests to access things like the test title, so it's important to maintain the value of this from test() when executing test code inside of the jestPlaywright method.

• If the page object is global and the config() method is synchronous (via await), why do we need to receive ({page}) inside the config method? Why can't we do this:

  await jestPlaywright.debug(async () => {
    await page.goto('https://google.com/');
    // ...
  })

• Would it be possible to extend Jest's test object to allow something like this:

  // Defaults debug configuration
  test.jestPlaywrightDebug('test1', async () => {
    // ...
  });
  
  // Modified debug configuration
  test.jestPlaywrightDebug({ /* ... */ }, 'test2', async () => {
    // ...
  });
  
  // Modified standard configuration
  test.jestPlaywrightConfig({ /* ... */ }, 'test3', async () => {
    // ...
  });

Implementation 2

DEBUG=true BROWSER=chromium jest -t 'failed'

I would use this functionality to create VSCode debug scripts which already contain both the file context and (when selected) the describe/test name context, allowing me to click a button in VSCode to debug the current file or describe/test block without touching the CLI:

// .vscode/launch.json
{
  // Use IntelliSense to learn about possible attributes.
  // Hover to view descriptions of existing attributes.
  // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug current test file",
      "env": {
        "DEBUG": "true",
        "BROWSER": "chromium"
      },
      "runtimeExecutable": "npm",
      "runtimeArgs": [
        "run-script", 
        "test:jest"
      ],
      "args": [
        "--", 
        "-i", 
        "${relativeFile}",
      ],
      "console": "integratedTerminal"
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug selected test name",
      "env": {
        "DEBUG": "true",
        "BROWSER": "chromium"
      },
      "runtimeExecutable": "npm",
      "runtimeArgs": [
        "run-script", 
        "test:jest"
      ],
      "args": [
        "--", 
        "-i", 
        "${relativeFile}",
        "-t",
        "${selectedText}"
      ],
      "console": "integratedTerminal"
    }
  ]
}

Obviously this hasn't been tested, but I don't see why it wouldn't work. A few additional thoughts:

• I'm a macOS user so I can't confirm this, but I believe the CLI syntax above won't work on Windows. I'm sure there's a wy to accomplish the same thing, but I thought it was worth mentioning if/when docs are updated.

• Using a generic DEBUG name for an environment variable could have unintended side effects. For example, Playwright already supports a DEBUG environment variable for debugging browser launches. Something more specific like JEST_PLAYWRIGHT_DEBUG would be safer even though it's more to type.

• Where would the debug settings be stored? Can I modify them (I hope so), or are they fixed (aside from the BROWSER list shown in the example)? How would I set nested options like { launchOptions: { headless: false, devtools: true } } on the CLI? My preference would be define debug-focused options in jest.config.js (per Allow jest-playwright settings in jest.config.js #226) and just do this:

  JEST_PLAYWRIGHT_DEBUG=true jest -t 'failed'

Apologies for the lengthy post. Hope some of this is helpful. :)

[trusktr]

If the page object is global and the config() method is synchronous (via await), why do we need to receive ({page}) inside the config method? Why can't we do this:

Because if it shared the same page as with other tests, it would have to make the browser not headless for the particular test using jestPlaywright.debug. How would it dynamically make the browser non-headless for that test, then headless again after that test is finished?

My guess is that may not be possible without some OS-specific hacking beyond the current capabilities. The easy solution is to make a new non-headless browser instance, and give you that page.

test.jestPlaywrightDebug('test1', async () => {
  // ...
});

An API like that would be much more ergonomic! I'd say make it just test.playwrightDebug as we know we're in Jest already

[mmarkelov]

@jhildenbiddle thank you for your comment. I think that we can start from the first implementation. It's almost done.
I think that it will be possible to use two helper functions: config and debug:

// Default debug configuration 
test('test1', async () => {
  await jestPlaywright.debug(async ({page}) => {
    // ...
  })
});

// Configuration will be used from the first argument
test('test2', async () => {
  await jestPlaywright.config({ ...options }, async ({page}) => {
    // ...
  })
});

Would it be possible to extend Jest's test object

Yeah, I agree that it will be much cleaner. But I'm not sure that it's possible to override test. I didn't find any way to do it. But I asked about this opportunity, so I'll make it if I can.

If we are talking about CLI implementation is about your playwright configuration. So you will be able to use just custom debug options. But I think that it will be possible to use jest.config.js or jest-playwright.config.js to pass them. Maybe something like this:

// jest-playwright.config.js
module.exports = {
  ...,
  debugOptions: { ... }
}

Something more specific like JEST_PLAYWRIGHT_DEBUG would be safer even though it's more to type.

Yeah, sure!

So to sum up, I will work on the first implementation. It will have two helper functions config and debug. For now you should use them inside test block, but if I find out any ways to extend test I will rewrite this in this way. The next step will be some additional research to use CLI for debugging, cause there is some extra question about this implementation

[jhildenbiddle]

@mmarkelov

Excellent! I'm happy to test an in-progress branch before release if you'd like.

Thank you for all the hard work. Much appreciated.

[mmarkelov]

@jhildenbiddle I just find out the way to extend global test. So there will be just two helpers on it:

// Defaults debug configuration
test.jestPlaywrightDebug('test1', async () => {
  // ...
});

// Modified standard configuration
test.jestPlaywrightConfig({ /* ... */ }, 'test2', async () => {
  // ...
});

I hope that we can merge the first version of it soon. But it will be a bit unstable - I need to implement some checks on these helpers and also I should check if test.jestPlaywrightConfig will work fine - it should override jest-playwright configuration

[mmarkelov]

@jhildenbiddle And also I need to find out the way to run these kind of test only once if the browser param is passed:

// jest-playwright.config.js
module.exports = {
    ...,
    browsers: ['chromium', 'firefox'],
}

// test

// This test should run only once
test.jestPlaywrightConfig({ browser: 'chromium' }, 'test2', async () => {
  // ...
});