[lit] TestTimes.py is racy - file access should be atomic / there should be a lock

[LebedevRI]

It does not do anything to ensure that there isn't a second instance of lit
that is also using the same .lit_test_times.txt. They may happen to write
into the same file at the same time, and now you've got corrupted .lit_test_times.txt,
and lit would not start, and you don't know what's wrong or how to fix it.

CC @davemgreen @davezarzycki

[davezarzycki]

I'm not sure this is the fault of lit really. The build results directory in general assumes non-conflicting use (the configure phase, building, testing, etc). A user has to have intimate knowledge of their project internals to know that sharing a build results directory between otherwise oblivious concurrent use is safe. Thoughts?

[LebedevRI]

I agree that it's not //intended// use case, and i did not do it on purpose,
but i think this is not the first report of this happening. I think the fact that
it just completely breaks (and can not be run again without manual intervention,
which again requires deep understanding of the problem), is not a great UX.

[davezarzycki]

Agreed. This isn't urgent for me, but if I find some time I'll try to figure out how to do file locks in Python.

[LebedevRI]

Thanks!