Improve handling of sketch source file handling

[matthijskooijman]

Prompted by arduino/Arduino#4335 I noticed that the way that sketch source files are handled in arduino-builder should probably be improved. There's a few sides to this:

• Instead of accepting a single .ino file, arduino-builder should probably also accept the sketch directory (since you're compiling the entire sketch, not just he .ino file passed). In general, it's probably the most flexible to accept either the directory, or any file inside (not limited to the .ino file).
• In addition to the main sketch, it should accept one (or perhaps any number) of "override" directories. Files in these override directories are used in favor of the original ones, but any files missing from the override directories are just used from the original sketch directory. This is useful when the IDE files have unsaved changes, which now cause the entire sketch to be written to a temporary directory and that sketch is compiled instead. This is wasteful (since also unchanged files are copied) and prevents arduino-builder from properly detecting wether a file was changed (this is currently solved by doing a contents comparison, but timestamps are probably better).
• Instead of doing content comparison for files to decide if they changed, we should probably just check timestamps. The previous change allows this, since only changed files are written.
• When actually compiling the sketch, arduino-builder compiles any source files present in the build directory (which are copied there, because they need to be in the same directory as the preprocessed .ino files). However, arduino-builder does not currently remove any source files no longer present in the original sketch. This means that deleting a file from the sketch causes it to still be compiled by arduino-builder. The IDE has some code to delete source files from the build directory when a file is deleted through the IDE, but this is really the wrong place to solve this. To properly solve this, either arduino-builder should delete any "extra" files when starting a build (taking care of keeping the .o files around) or it should just ignore the extra files (e.g. compile only files present in the original sketch, instead of listing files in the build directory). I tend towards the latter option, since the former will also delete any extra files generated by the commands in platform.txt, which prevents a third-party platform.txt from doing some things, like keeping a cache of sorts between compile runs). The downside of the latter approach is that it keeps extra files around in the build directory, but that should only be until the IDE is closed, so I don't think it's a big problem.
• When copying sketch source files into the build directory for building, arduino-builder should probably add #line directives, so the compiler knows the original filename (which can help with error messages and debugging symbols).

Some of these might be somewhat separate, but I mostly wanted to get these thoughts out of my head so I won't forget about them later :-)

[matthijskooijman]

To implement some of these things, it is probably needed to slightly rework the compilation process, by first listing files in the (original) sketch and then running the rest of the preparation and mbuild process based on that list, instead of repeatedly listing files in different directories.

[digimeas]

Hi!
1 Please don't "clean out" my folder without asking, and
2 Please don't ask. :-)

3 An easy way to specify extra #include directories would be nice.

Thanks,
Doug

[matthijskooijman]

My proposal was indeed to not clean out any files automatically (except for an automatically created temporary build directory when the IDE is closed, which already happens). Your third request is off-topic for this issue, IIRC there is another one that requests something similar to that that you might want to subscribe to.

[slady]

Hi, I created issue 323 add #line directive to source files that is related to your proposal and contains a part of your original text.