New fixture: unique temporary directory per session

[nicoddemus]

I would like to propose adding a new session_tmpdir fixture.

This fixture provides a py.path to a unique temporary directory, but all workers see the same directory for the entirety of the test session. After the session ends, the directory is deleted as usual.

This fixture would be very useful as an integration point for workers, allowing them to share information if required.

[RonnyPfannschmidt]

Should be a tmpdir factory attribute

[nicoddemus]

Hmm interesting idea. Could you elaborate why do you think that's better, implemention and ux wise?

[RonnyPfannschmidt]

well - as we currently have no sane communication point, i would go as far as defferring this until we have more transient fixtures

also we don't have a well defined concept of a "test session"

in terms of pytest each worker has a own session - so it makes no sense to intermix as of now at all

i would propose to defer this or beginning with a ml discussion on terms and scopes (else we wil lhave a mess)

[nicoddemus]

Valid concern about the implementation, but I have given it some thought and it seems very trivial at first glance, basically master creates the directory and forwards it to the workers using the normal config object, and each worker just provides a fixture which returns the py.local object that points to that directory.

But you do have a point about using "session" to name it though, it may be confusing because session already has a different semantic for that name in pytest.

Given the bit about the implementation and your concerns about the name, how about shared_tmpdir then? After all, it is just a shared directory between all workers.

[RonnyPfannschmidt]

btw - next problem - this can never ever work with ssh

[RonnyPfannschmidt]

so we will have quite some fun figuring failure and xfail modes for it

[nicoddemus]

btw - next problem - this can never ever work with ssh

Hmm indeed, this is a real problem, but I think I have a clean solution, see below:

so we will have quite some fun figuring failure and xfail modes for it

Not sure if you mean "how to fail if one uses shared_tmpdir in a remote pytest-xdist session (ssh)", but if you do then an easy way to handle it would be for master to detect that some workers are using remote connections (based on their popen setting) and just send None as shared_tmpdir to all workers so they can fail gracefully with "shared_tmpdir fixture can't be used in remote sessions".

This can be a known limitation of the fixture, and won't really be surprising if someone uses it in a remote session.

[RonnyPfannschmidt]

its a nice degradion

[drake-mer]

That's not a solution - rather an ugly hack (will all due respect). I think xdist should integrate properly the meaning of 'session', 'module' and 'function' pytest scopes.

[nicoddemus]

@elijahbal ideally yes, but given that workers live in different processes, sharing fixtures in a way that conforms to what normal code expects (e.g., that changes in a fixture object would reflect in all the other workers) is very hard.

[RonnyPfannschmidt]

@elijahbal with regard to that - no its not sanely possible and should never be a design goal

[drake-mer]

Allright I get it. I thought I could imagine some workaround. It would have been very convenient to do so but I understand it's a hard path to take and if you are right (I believe you are) this use case should be handled in a totally different way.

Thanks for answering.