Advice on using (mocking?) dependencies of the Library Under Test

[jgfoster]

I'd like to test my use of the LiquidCrystal library

I believe that I can use other libraries and that they should "just work" because they would make calls to the base Arduino code that you currently implement. For example, displaying something on the LCD should set appropriate pins and I could check the pins using the existing Godmode interface. But with that approach, I'd have to reverse engineer the meaning of the pins and watch for appropriate values (in fact, I've tried this at https://github.com/jgfoster/Arduino/tree/master/LCD1602).

Alternatively, it would be nice to have something like the Serial interface available through Godmode. For that I suppose I could take the existing library code and modify it to keep track of high-level events and make them available for analysis. Is that the right approach or am I missing something obvious?

If I did create such a mock library, would it be something that should be contributed back to this project?

[ianfixes]

Have you looked at https://github.com/ianfixes/arduino_ci/blob/master/REFERENCE.md#serial-data ?

If you're talking about something that could provide a more semantic explanation of raw bits being sent to a device, the closest thing I have to what you're talking about is the ability to read pins as streams of bytes instead of streams of bits. That's crudely defined in the form of a toAscii method: https://github.com/ianfixes/arduino_ci/blob/master/REFERENCE.md#pin-history-as-ascii

This is already messy, because the grouping of bits into bytes depends on the offset of the first byte's first bit, and endian-ness in the stream.

More long term, I think the path forward is going to be a more robust approach to Godmode. Does #145 cover some of what you're looking for?

[jgfoster]

The Serial example isn't quite what I want since Serial is part of the built-in Arduino system. At first I thought that a better example would be SoftwareSerial since it is in the list of external libraries. I see that there are tests for this "library", but it appears that it isn't really one of the Arduino Libraries, but is in some sense built-in. I also see that, instead of using the "real" library, you have a substitute SoftwareSerial.h that deals directly with GodmodeState instead of letting the official code read and write to pins.

I'm sure this is more useful, but it doesn't give me an example of how to interact with a true Arduino Library that has C++ code in the location returned by bundle exec arduino_library_location.rb. I see that there is a discussion of custom versions of external arduino libraries, but I can't find any examples.

I have a custom library that uses LiquidCrystal. It works from the Arduino IDE with a trivial sketch, but doesn't work from arduino_ci:

Unit testing test.cpp with g++... 

Last command:  $ g++ -std=c++0x -o /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/unittest_test.cpp.bin -DARDUINO=100 -g -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize=address -D__AVR_ATmega328P__ -I/Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/vendor/bundle/ruby/2.6.0/gems/arduino_ci-0.3.0/cpp/arduino -I/Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/vendor/bundle/ruby/2.6.0/gems/arduino_ci-0.3.0/cpp/unittest -I/Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/src /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/vendor/bundle/ruby/2.6.0/gems/arduino_ci-0.3.0/cpp/arduino/Arduino.cpp /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/vendor/bundle/ruby/2.6.0/gems/arduino_ci-0.3.0/cpp/arduino/Godmode.cpp /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/vendor/bundle/ruby/2.6.0/gems/arduino_ci-0.3.0/cpp/arduino/stdlib.cpp /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/vendor/bundle/ruby/2.6.0/gems/arduino_ci-0.3.0/cpp/unittest/ArduinoUnitTests.cpp /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/src/HelloLCD.cpp /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/test/test.cpp

/Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/src/HelloLCD.cpp:4:10: fatal error: 'LiquidCrystal.h' file not found
#include <LiquidCrystal.h>
         ^~~~~~~~~~~~~~~~~
1 error generated.
/Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/test/test.cpp:3:10: fatal error: 'LiquidCrystal.h' file not found
#include <LiquidCrystal.h>
         ^~~~~~~~~~~~~~~~~
1 error generated.
...Unit testing test.cpp with g++                                              ✗

The library is present in the expected directory:

jfoster@pitcairn HelloLCD % ls -alF `bundle exec arduino_library_location.rb`
total 24
drwxr-xr-x  11 jfoster  staff   352 Sep  9 10:58 ./
drwxr-xr-x@ 12 jfoster  staff   384 Sep  9 09:20 ../
-rw-r--r--@  1 jfoster  staff  6148 Sep  9 08:55 .DS_Store
drwxr-xr-x  14 jfoster  staff   448 Sep  8 18:00 Blink/
lrwxr-xr-x   1 jfoster  staff    57 Sep  9 10:52 DoSomething@ -> /Users/jfoster/code/arduino_ci/SampleProjects/DoSomething
drwxr-xr-x  13 jfoster  staff   416 Sep  9 10:57 HelloLCD/
drwxr-xr-x   7 jfoster  staff   224 Sep  8 21:46 LiquidCrystal/
drwxr-xr-x   8 jfoster  staff   256 Sep  5 20:13 RingBuffer/
lrwxr-xr-x   1 jfoster  staff    59 Sep  9 10:58 TestSomething@ -> /Users/jfoster/code/arduino_ci/SampleProjects/TestSomething
drwxr-xr-x   7 jfoster  staff   224 Sep  8 11:36 USBHost/
-rw-r--r--   1 jfoster  staff    87 Sep  2 18:51 readme.txt
jfoster@pitcairn HelloLCD % 

It seems to me that the g++ command (above) does not provide an include directory or a reference to the desired C++ file.

It looks like we are supposed to be able to use an external library, but I can't get it to work. Any advice?

[ianfixes]

The first thing that I notice is /Users/jfoster/code/arduino_ci/SampleProjects/HelloLCD/src/HelloLCD.cpp, which sets off a red flag because I expect HelloLCD to be in the directory specified by bundle exec arduino_library_location.rb.

Beyond that, it sounds like you're asking how to test a library that depends on another 3rd-party Arduino library. I suspect my documentation on this is misleading. I had set things up so that you (according to REFERENCE.md) specify the library dependencies in the configuration file.

Snip of that text:

unittest:

  # These dependent libraries will be installed
  libraries:
    - "abc123"
    - "def456"

These files get included in C++ compilation (code here and then here).

So in your case the dependent library would be LiquidCrystal.

My comment should be updated

- # These dependent libraries will be installed
+ # These dependent libraries will be included from Arduino's "Libraries" directory during compilation.
+ #    If they don't exist there already, the Arduino IDE will be used to attempt to install them
    libraries:
      - "abc123"
      - "def456"

But there's probably a better place in the docs to explain all of the behavior related to dependencies on other 3rd-party libraries.

Is this answering the entire question or just part of it?

[jgfoster]

I thought I had tried all that and it didn't work, but now, reading and following the instructions carefully (!), it seems to be finding the files. I'll keep working at it. Thanks for your help!

[jgfoster]

Part of the reason for developing inside arduino_ci is that I anticipate making changes to the testing framework. In particular, I'd like to examine your suggestion that "the path forward is going to be a more robust approach to Godmode." As to #145, it seems sufficiently vague that I'm not sure how it would handle my situation. I think I'll pursue "the simplest thing that could possibly work" and see where it leads me.

[ianfixes]

It sounds like we can close this issue.

As far as work on Godmode is concerned, there are 2 facets being discussed right now. The first is that (currently) arduino_ci has no real capability of mimicking real-world hardware: bits take time to flip, functions take time to run, a circuit might be expected to produce certain effects on input pins, and so on.

The other issue is as you noted -- validation of a library that makes calls to a hardware driver can expect to run into a LOT of bits that aren't necessarily easy to verify in the current framework. Anything involving hardware that acts in asynchronous fashion would be cumbersome to say the least.

Having better use cases would lead to arduino_ci developing more ergonomic test patterns, but it's a chicken-and-egg problem for now (not enough adoption of this project to see where its real weaknesses are).

[jgfoster]

My only question is where would like questions related to testing LiquidCrystal? Is it okay to keep a conversation going on a closed issue (it's fine with me)? For example:

I find that I'd like a way to distinguish between when code is compiled from the Arduino IDE and when code is compiled from arduino_ci. Ideal would be something like #define ARDUINO_CI but the closest I've come is MOCK_PINS_COUNT. Thoughts?

[ianfixes]

Sounds like it's premature to close this issue 😄 I'm also opening #159 to track the discussion of constants.

So if I'm understand the question here, it's that you have one library that depends on another library, and you want to mock one out?

One way I can imagine doing this is to put mock classes directly in the library you wish to mock, so that anyone using the library can have access to them. That would mean writing the library such that it could handle dependency injection of some kind.

Failing that, you could create the mock library in its own Arduino library directory and declare mock versions of all the things you'd need access to. In that setup, your configuration would look something like this:

compile:
  libraries:
    - "LiquidCrystal"

unittest:
  libraries:
    - "LiquidCrystal Mock"

That at least gives you the ability to mock a library without any access to the original.

[jgfoster]

I've come up with hybrid approach that seems to work (step 1 of "make it work, make it right, make it fast"). I'm subclassing the standard library to create a testable object, and using a check in my primary program to decide whether to use the production or the test version (this is where #159 will be helpful). So, in a sense this isn't a pure mock (but could be turned into one by not being a subclass). If you think this could be done in another way that is more consistent with the general approach (and possibly be included here), let me know.

The code is LiquidCrystal_CI and a video describing it is available. Advice and comments are welcome, but I'll close this since we at least have a way to use/mock standard libraries.