lib manager overrides core libraries included with 3rd party boards

[soundanalogous]

I'm encountering several library compatibility issues lately. I believe these are due in part to the library manager and that if you update a library via the lib manager (the Servo library for example) that is also included by 3rd party boards (via the board manager or 3rd party installers such as the Teensy installer) the library installed by the lib manager will take priority.

A couple of specific examples:
Arduino Zero required a newer Servo library than was included in the latest Arduino release. This could be resolved by using the library manager to install Servo v1.1.0. However this version of Servo was incompatible with non SAMD architectures (although this appears to have been resolved in Servo v1.1.1). However 3rd party boards such as Teensy include their own version of the Servo library. Updating Servo via lib manager however will now include that version rather than the version of the lib included with the 3rd party board. Same problem with Intel Galileo and Servo library.

I've noticed the same incompatibility issues across boards with the Ethernet and SPI libraries as well (I believe it's related to this issue: #3961).

So there are actually 2 issues here:

1. lib manager overrides core libraries included with 3rd party and board manager installed boards
2. core libraries are not compatible across all official arduino boards (when they should be) - due to issues in core library support for new boards (is anyone testing this before release?)

Some possible solutions:

• Give priority to core libraries included with 3rd party boards (installed by means other than the boards manager) and by boards installed and updated via the boards manager. Include those libraries instead of core libraries updated via the library manager. Maybe there would be a clever way to override on a case-by case basis (for example I wouldn't want the Teensy version of Firmata included when the latest Firmata version supports all Teensy boards).
• Consider including core libraries in all architecture variants. For example include Servo in samd instead of trying to maintain a all in one version for all avr, sam and samd, especially when new boards are available before they are fully compatible with the core software.
• Consider a way for the library manager to be architecture aware. This would enable updating a library for only certain architectures.

[lmihalkovic]

I've been playing with the IDE code for a couple days. One of the things I was considering was to change the way libraries are stored. The current layout is something like this:

libraries
   libABC
       file1
       file2
   libDEF
       file1
       file2

I was thinking about following the unix model and doing something like:

libraries
   libABC
       1.0.1
           file1
           file2
       1.0.8
           file1
           file2
   libDEF
       2.0.0
           file1
           file2

This would take care of keeping multiple versions of any given library. The problem is that it requires a new mechanism to let the compiler know which one to use.

{ 
  ' this is a simplified JSON file following the Ninja format
  depends [
    {
      name: libABC
      version: 1.0.8
    }
  ]
}

No matter what, I may need something like that for the other changes I am making. In the following screenshot, the 'references' section contains a list of the system libraries that the sketch uses. This test sketch only requires Wire but the list is still missing filtering code (the 'libraries' section shows the libraries stored locally inside the sketch).

#4050

[NicoHood]

This makes sense. However I would suggest to integrate git somehow. Or only apply this to extrnal downloaded libraries via download manager. So that you can develop you libraries via git like we are used to.

[lmihalkovic]

Yes, my thought for that multi-versions scheme is only for the lib manager, to address cases where someone might have different sketches that would require different versions of the same lib. Or for the case where some uses a given version, and bumping up presents a risk until things are fully tested. Then the lib manager cen be instructed to download the newer version, but keeping the existing one.

The sketch-local libraries do not follow this proposed layout, because by definition they would be version controlled (could be fun to have a bare minumum GIT/HG/SVN awareness in the ide).

In all cases, displaying the lib version in the outline view (I think it makes sense to christen it like that!?) is just informative. Ideally, switching to a different version in the REFERENCE section could be done with a simple right-click context menu showing all the versions that exist on the machine. Then it is easy to recompile/test. And then go to lib manager and tell it to delete a given version if it is no longer used... Having all the sketches inside a single parent folder means it is even possible to verify which versions are no longer cureently in use. It would also be nice to be able to say 'make a copy of this lib into my sketch-local dependencies (like in my example where I took the ADFNeoPixels lib and started modifying it).

Then the last nice step would be to right click on a sketch-local lib, thereby pushing it descriptor to someplace that the lib-manager can see, to 'Puclish Library'. The publishing process would require validation the descriptor
validating the presense of keywd
Adding a 'stars' evaluation system to the lib manager (pre-req to be traced back to github account for eg) would then give users feedback about the libs.

I'm a newb to arduino, so maybe some of this is already there and i don't know it.

Future improvement thouhts include:

• a right-click somewhere to help create the keyword.txt file when editing a sketch-local lib (it is easy to tell in my code if the file being edited is a sketch file or a lib file, so it is easy to select-something/right-click/click-DefineAsKeywordX)
• a couple of wizards/templates for keywords.txt/library.properties

....time permitting

criticisms most welcome for any/all of it!!!

[per1234]

1. lib manager overrides core libraries included with 3rd party and board manager installed boards

Library Manager installs to sketchbook/libraries so the issue is actually that libraries in the sketchbook/libraries folder now take precedence over bundled libraries. This appears to have been caused by arduino/arduino-builder@7f7eb7a or maybe arduino/arduino-builder@4e6db8c the issue doesn't occur in Arduino IDE 1.6.6 2015/10/29 11:41 and the problem occurs in Arduino IDE 1.6.6 2015/10/30 09:58.

The order of priority used to be(1.6.6 2015/10/26 and earlier): bundled library for selected 3rd party board > sketchbook/libraries > Arduino/libraries

Edit: this information is correct but caused by a bug introduced by arduino-builder.

[ffissore]

@NicoHood and @lmihalkovic you're going off topic. Please open a separate issue to discuss your enhancement proposal
@soundanalogous and @per1234 what you're experiencing is the correct and expected IDE behaviour. Libraries installed through library manager get installed in SKETCHBOOK/libraries, which is where libraries have the highest priority. This is to give user/developer an easy way to override a core/builtin library, whatever the reason might be (she's developing a new version of it, she needs some customization)
As for the incompatibility of Serve with other cores, I'm assigning this issue to @cmaglie who knows more about core stuff and who can deal with it

[per1234]

@ffissore I just edited my previous comment to state this but in case you miss it:
The order of priority used to be(1.6.6 2015/10/26 and earlier): bundled library for selected 3rd party board > sketchbook/libraries > Arduino/libraries

Edit: this information is correct but caused by a bug introduced by arduino-builder.

[ffissore]

@per1234 that was a bug, fixed with arduino/arduino-builder#38. try with 1.6.5 (which doesn't ship arduino-builder): it works like recent hourly builds (which bundle arduino-builder)

[ffissore]

For the record, correct order is
built in libs, parent platform (if any) libs, target platform libs, sketchbook libs

[per1234]

Ok, so then the issue is that once a library is updated using Library Manager it is installed to sketchbook/libraries so the behavior changes from the 3rd party bundled library taking precedence to the Library Manager installed library taking precedence. The solution would be for Library Manager to install built in libs to the arduino15/libraries folder and for that folder to be higher in priority than built in libs but lower in priority than the rest.

[ffissore]

Could you rewrite my priority list above so that it features your proposed lib?

[per1234]

In order of lowest to highest priority: built in libs, updated built in libs(arduino15/libraries folder), parent platform (if any) libs, target platform libs, sketchbook libs

This issue is a major problem for 3rd party cores that use bundled libraries that replace built in libraries because the replacement library needs to be only used when the associated board is selected so requiring the user to install the library as is into the sketchbook/libraries folder is not an option. The replacement library names could be changed but that will require the user to change the includes in their sketches back and forth depending on which board they are using. An example of this is Mighty 1284P where Ethernet, Firmata, GSM, SD, and Servo libraries were patched to make them compatible with ATmega1284P.