Build directory for editable installations

[dnicolodi]

The build directory for editable build is currently .mesonpy/editable/build in the source directory. I don't see a reason why the build directory is nested two levels deep into otherwise empty directories . A flatter directory structure could be used.

While doing so, attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

Isn't there prior art for choosing the name of the build directory? I don't like something specific to meson-python much. Python uses __pycache__ for its cache. Is it allowed to add tool specific stuff in there? distutils uses a build folder, IIRC. What about other build backends?

It would be nice to have this decided before we release a version with the editable support.

EDIT: reworded for politeness

[rgommers]

Isn't there prior art for choosing the name of the build directory?

I'd say build/ in the root dir is the one more or less standard name - not just distutils/setuptools, but also Sphinx (also that also uses _build sometimes). Not sure about other backends, never had a need to actually use them.

The downside of build is that it may clash with a standard choice for using Meson directly (meson setup build, ninja -C build is already mostly muscle memory for me).

Reusing __pycache__ is a no no I'd say.

Also attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

Hmm, that is an interesting point.

[rgommers]

I'd say that whatever we choose here is a private implementation detail though - the user has no business doing anything with it beyond adding it to .gitignore, so we should be fine tweaking it later.

[rgommers]

Thinking about this some more:

• a hidden dir isn't nice, it's already tricked me once or twice, and I kinda knew it should be there
• build is the standard name
• separate interpreters can already be supported via the builddir option (I tested it with pip install -e . --no-build-isolation --config-settings=builddir=build-py310 and it works)
• build clashing with the distutils default may actually be a good thing; having in-tree remnants from a distutils build can in some cases cause problems

tl;dr let's use build?

[dnicolodi]

I like build too. I like supporting multiple interpreters out of the box. Having a two level directory structure solves this: build/cp311 or build/cpython-3.11/ but I don't know if this is a common enough use case that is worth supporting.

[rgommers]

I'd prefer to not special-case interpreters I think. Different builds can also happen with different compilers (e.g., using GCC and Clang in parallel), different sets of build dependencies (on SciPy testing with 2 numpy versions is much more interesting than testing with 2 cpython versions), and so on.

Plus it does bring up questions like "what if we have regular and a debug 3.11 interpreters".

Matrix'ed environment testing is for tools like tox/nox or custom scripts I'd say, and they can easily pass a builddir for anything like this.

[FFY00]

Now it is something like .mesonpy/editable/build in the source directory. The two dirst levels are kind of useless, and can be removed.

Please avoid this kind of wording. If you don't see a purpose, then simply say that. Saying that the directories are useless is not constructive, and doesn't really provide any value to the conversation. It's just a stab at whoever made that choice, which in this case it was me.
If you don't understand the decision, then try to learn about it, instead of attacking it.

This is not the first time you have used this kind of language, and it is not the first time I have asked you to stop, so, please, please, make an effort to avoid it.

It is not even an issue just for this project, but both Python and all PyPA projects are under the PSF CoC, where this type of continued behavior is not allowed.

I have been keeping an open mind, and trying to be charitable towards you and the way you word things, but it has been more and more difficult lately (particularly bad examples are #359 and #358). I also tried to reach out personally, to try to sort out any friction that may exist between us, but without luck.

This repo hasn't adopted the PSF CoC, but please consider this a warning for your future interactions in such places. Even though we haven't adopted the PSF CoC, our interactions have started to become an issue, and I, personally, cannot continue like this.

Regardless, I think everyone here is trying to do their best, and I am very thankful for your contributions to meson-python so far. Unfortunately, communication and interpersonal relationships are not easy, so sometimes trying your best is not enough. If you have any suggestions for how we can improve our relationship, I would love to hear, otherwise, please just be a bit more mindful of how you word things.

Also attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

+1 on this.

Isn't there prior art for choosing the name of the build directory? I don't like something specific to meson-python much. Python uses __pycache__ for its cache. Is it allowed to add tool specific stuff in there? distutils uses a build folder, IIRC. What about other build backends?

I think the common behavior for this sort of stuff is exactly what we have currently, and that's why I choose it. A hidden directory for the tool, and then whatever we need to have inside. In this case, I put the build directory as a subdir so that we can still use the main directory for other stuff we may need.

Python uses __pycache__ for its cache. Is it allowed to add tool specific stuff in there?

Adding stuff to __pycache__ is a very bad idea IMO.

a hidden dir isn't nice, it's already tricked me once or twice, and I kinda knew it should be there

This is common practice, though. We could perhaps add a message of where the builddir is placed in the end of build_editable?

tl;dr let's use build?

I'd very much prefer to use builddir instead.

---

Overall my main objection against this proposal is that it is not clear at all for a user that finds a build directory in their project that that directory is being used by the editable install. On top of that, it will likely conflict with already existing builddirs.

You could solve that by naming it builddir-editable. I am -0.5 on the proposal with that change, and -1 as-is.

[FFY00]

Other possible option would be putting the build directory on the installation path, instead of the project source. This would also allow installations to different interpreters without any conflicts.

[dnicolodi]

@FFY00 I considered for quite a while whether I should respond in the merit of the accusation of not being considerate in my communication. I really wanted to just ignore it: I'm not contributing to open source for the drama and I really don't have emotive energy to sink into this kind of arguments. This is the reason why I didn't even read your email on the subject: I don't have the emotional resources for it. However, you really seem convinced that my choice of words here and elsewhere is a personal attack toward you, so much that it constitutes a violation of community code of conduct. I cannot let pass this characterization of what I wrote.

Can you please tell me what you find offensive in me pointing out that a technical minutia does not serve anymore any purpose? Would the use of the words "do not serve anymore any purpose" instead of "are kind of useless" have made a difference in your interpretation of my statement? I'm clearly not a native English speaker, but AFAIK, "useless" just means "not fulfilling the intended purpose" and does not carry any judgement on who put the object in place. What if the word "wrong" would have been used instead? Would you consider stating that something is wrong or broken or that does not function as intended a stab at who implemented it?

It did not even cross my mind to give any judgment on who coded the solution in this way. We continuously make decision that will be revealed to be wrong down the road. If this would not be the case, software would be without bugs. I don't think that pointing out that (with more information and in mutated circumstanced) some of these decisions appear wrong should be seen as a judgment of the person that made these decisions.

In this specific instance, I know why the directory structure was chosen to be what it is. It was a decision that made sense when it was made. However, with mutated circumstances (the new implementation of the editable wheels) this specific directory structure does not serve anymore the intended purpose. I think it is not controversial to state that, because there is no more installation directory for editable wheels, having the build component in the .mesonpy/editable/build/ path does not have any use: : it is kind of useless.

If you don't understand the decision, then try to learn about it, instead of attacking it.

This kind of statements bring a technical discussion to the personal level and are not useful to the consensus building process. In particular, the "you don't understand" argument not followed by an explanation is so bad that I feel it is almost stereotypical of bad arguments. It inherently describes one side of the discussion as being right and the other wrong: it reads pretty much as "I have the knowledge and you don't. I'm going to dismiss everything else you have to say because the product of ignorance. Once you will inform yourself, you will inevitably agree with me". Please note how this logically reduces to "You don't understand till you don't agree with me", which is dangerously close to "I am right, and you are wrong". Consider this a warning for your future interactions with fellow human beings: this kind of arguments tend to do not be received wheel.

[rgommers]

I'm sorry to see this interaction, as well as the pattern of lots of back and forth in general. It's safe to say that things could go more smoothly in communication on this project. I hope that we all strive to be open, kind and collaborative. If we'd try to be a little less strongly worded in our opinions (myself included), I think that would improve things. We're all pretty competent developers here, but what often seems to be the problem is we missed some use case or have some silent assumption in our heads that turns out to be incomplete.

Regarding reviews and decisions, it'd be great to distinguish more between what's important/blocking, and what's really not. This issue is a good example: we can make a small improvement in usability here, but it really doesn't matter all that much. And a similar thing applies to any review comments: style comments, grammar nits, using a set vs. a tuple or other such things that don't change how code works, etc. - let's just try to not comment on those, and if we do indicate that we understand that it's not critical by adding "minor" or "nit"?

To bring that back to a more neutral example: my last doc PR (gh-338) attracted 44 comments - nothing particularly bad about any single comment, but the overall effect is that it's energy-draining and the end result of the doc content really didn't change all that much compared to the initial version. If we'd just merged it as is and spent the remaining time on other docs work, we'd have accomplished more with less friction.

To address directly the CoC comments:

• We don't have a CoC right now, and this is not the right time to discuss introducing one. Once things settle down I'd be happy to take the lead on that though, since it's good practice to have one (ideally we'd have all that is listed under Community Standards). In the meantime, let's all try to be kind/respectful/professional/<choose your preferred term here>.
• I don't see the quoted sentence as a personal attack.
  • It is strongly worded and, as stated above, it'd be useful to tone that down a little. But that goes for all of us, and for quite a few other review interactions.

[rgommers]

Also attaching a Python API tag to the build directory name would allow to have multiple parallel installations of the same package for different interpreters.

+1 on this.

Okay, you both agree on this, so let's go that way.

For the other choices, how about I make an editorial decision here? Seems preferred over lots more discussion.

[dnicolodi]

I proposed tagging the build directory with the python interpreter API tag in some form because the editable wheel documentation page prominently warns (or at least I remember it this way) to the fact that being able to have editable wheels for multiple interpreters as a limitation. I thought that, if the limitation was felt so important, adding a tiny amount of complexity to allow it could have been a good compromise.

However, I'm a big fan of keeping things simple unless necessary, thus I'm happy without this feature and a paragraph in the documentation that mentions how to achieve the same using the builddir config setting.

I'm fine with @rgommers picking the build directory name that he prefers. I opened this issue only because the nested directories are not required to exist anymore and I thought we could simplify what we present to the users before releasing editable wheel support in the next release.