core/lib search results must be the same as from the IDE 1.x

[kittaakos]

Describe the problem

There is an inconsistency between the CLI lib search output and IDE 1.x's Library Manager functionality.

The issue originally popped up here: arduino/arduino-ide#1442

IDE2 wants to produce the same library search results as IDE 1.x. When searching for the SparkFun_u-blox_GNSS term, the following results should be returned by the CLI:

The CLI returns with one result only, where the name matches:

./arduino-cli lib search "SparkFun_u-blox_GNSS" --format json:

{
  "libraries": [
    {
      "name": "SparkFun u-blox Arduino Library",
      "releases": {
        "1.8.10": {
          "author": "SparkFun Electronics \u003ctechsupport@sparkfun.com\u003e",
          "version": "1.8.10",
          "maintainer": "SparkFun Electronics \u003csparkfun.com\u003e",
          "sentence": "Library for I2C and Serial Communication with u-blox modules",
          "paragraph": "An Arduino Library to enable both I2C and Serial communication for both NMEA reception and binary UBX sending to u-blox modules. Useful for interfacing to the \u003ca href=\"https://www.sparkfun.com/products/15136\"\u003eSparkFun GPS-RTK2\u003c/a\u003e ZED-F9P, \u003ca href=\"https://www.sparkfun.com/products/14980\"\u003eSparkFun GPS-RTK\u003c/a\u003e NEO-M8P-2, the \u003ca href=\"https://www.sparkfun.com/products/15210\"\u003eSparkFun SAM-M8Q\u003c/a\u003e, and the \u003ca href=\"https://www.sparkfun.com/products/15193\"\u003eSparkFun ZEO-M8Q\u003c/a\u003e. Library also works with other u-blox based boards. \u003cbr\u003e\u003cbr\u003eThe ZED-F9P and NEO-M8P-2 modules are top-of-the-line modules for high accuracy GNSS and GPS location solutions including RTK. The ZED-F9P is unique in that it is capable of both rover and base station operations allowing the module to become a base station and produce RTCM 3.x correction data.",
          "website": "https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library",
          "category": "Sensors",
          "architectures": [
            "*"
          ],
          "types": [
            "Contributed"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/sparkfun/SparkFun_u_blox_Arduino_Library-1.8.10.zip",
            "archive_filename": "SparkFun_u_blox_Arduino_Library-1.8.10.zip",
            "checksum": "SHA-256:0d6225cf780357b9565abb52e03ab68efe9c6ae84e187d74a991e53bac350162",
            "size": 327202,
            "cache_path": "libraries"
          }
        },
        "1.8.11": {
          "author": "SparkFun Electronics \u003ctechsupport@sparkfun.com\u003e",
          "version": "1.8.11",
          "maintainer": "SparkFun Electronics \u003csparkfun.com\u003e",
          "sentence": "DEPRECATED Library for I2C and Serial Communication with u-blox modules\u003cbr/\u003e\u003cbr/\u003e",
          "paragraph": "Please note: this library is deprecated. Please migrate to \u003ca href=\"https://github.com/sparkfun/SparkFun_u-blox_GNSS_Arduino_Library\"\u003ev2.0\u003c/a\u003e. You can find the migration instructions \u003ca href=\"https://github.com/sparkfun/SparkFun_u-blox_GNSS_Arduino_Library#migrating-to-v20\"\u003ehere\u003c/a\u003e.\u003cbr/\u003e\u003cbr/\u003eAn Arduino Library to enable both I2C and Serial communication for both NMEA reception and binary UBX sending to u-blox modules. Useful for interfacing to the \u003ca href=\"https://www.sparkfun.com/products/15136\"\u003eSparkFun GPS-RTK2\u003c/a\u003e ZED-F9P, \u003ca href=\"https://www.sparkfun.com/products/15005\"\u003eSparkFun GPS-RTK\u003c/a\u003e NEO-M8P-2, the \u003ca href=\"https://www.sparkfun.com/products/15210\"\u003eSparkFun SAM-M8Q\u003c/a\u003e, and the \u003ca href=\"https://www.sparkfun.com/products/15193\"\u003eSparkFun ZOE-M8Q\u003c/a\u003e. Library also works with other u-blox based boards.\u003cbr/\u003e\u003cbr/\u003eThe ZED-F9P and NEO-M8P-2 modules are top-of-the-line modules for high accuracy GNSS and GPS location solutions including RTK. The ZED-F9P is unique in that it is capable of both rover and base station operations allowing the module to become a base station and produce RTCM 3.x correction data.\u003cbr/\u003e",
          "website": "https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library",
          "category": "Sensors",
          "architectures": [
            "*"
          ],
          "types": [
            "Contributed"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/sparkfun/SparkFun_u_blox_Arduino_Library-1.8.11.zip",
            "archive_filename": "SparkFun_u_blox_Arduino_Library-1.8.11.zip",
            "checksum": "SHA-256:d7367e582f3e3d336d435998f45c81a90c674be574972004e4bd44dc44f48024",
            "size": 327469,
            "cache_path": "libraries"
          }
        },
        "1.8.7": {
          "author": "SparkFun Electronics \u003ctechsupport@sparkfun.com\u003e",
          "version": "1.8.7",
          "maintainer": "SparkFun Electronics \u003csparkfun.com\u003e",
          "sentence": "Library for I2C and Serial Communication with u-blox modules",
          "paragraph": "An Arduino Library to enable both I2C and Serial communication for both NMEA reception and binary UBX sending to u-blox modules. Useful for interfacing to the \u003ca href=\"https://www.sparkfun.com/products/15136\"\u003eSparkFun GPS-RTK2\u003c/a\u003e ZED-F9P, \u003ca href=\"https://www.sparkfun.com/products/14980\"\u003eSparkFun GPS-RTK\u003c/a\u003e NEO-M8P-2, the \u003ca href=\"https://www.sparkfun.com/products/15210\"\u003eSparkFun SAM-M8Q\u003c/a\u003e, and the \u003ca href=\"https://www.sparkfun.com/products/15193\"\u003eSparkFun ZEO-M8Q\u003c/a\u003e. Library also works with other u-blox based boards. \u003cbr\u003e\u003cbr\u003eThe ZED-F9P and NEO-M8P-2 modules are top-of-the-line modules for high accuracy GNSS and GPS location solutions including RTK. The ZED-F9P is unique in that it is capable of both rover and base station operations allowing the module to become a base station and produce RTCM 3.x correction data.",
          "website": "https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library",
          "category": "Sensors",
          "architectures": [
            "*"
          ],
          "types": [
            "Contributed"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/sparkfun/SparkFun_u_blox_Arduino_Library-1.8.7.zip",
            "archive_filename": "SparkFun_u_blox_Arduino_Library-1.8.7.zip",
            "checksum": "SHA-256:f9c7a6c5d150a71903a75a676948ea2625f81b733a6e2aba00a44bd2297727b7",
            "size": 207836,
            "cache_path": "libraries"
          }
        },
        "1.8.8": {
          "author": "SparkFun Electronics \u003ctechsupport@sparkfun.com\u003e",
          "version": "1.8.8",
          "maintainer": "SparkFun Electronics \u003csparkfun.com\u003e",
          "sentence": "Library for I2C and Serial Communication with u-blox modules",
          "paragraph": "An Arduino Library to enable both I2C and Serial communication for both NMEA reception and binary UBX sending to u-blox modules. Useful for interfacing to the \u003ca href=\"https://www.sparkfun.com/products/15136\"\u003eSparkFun GPS-RTK2\u003c/a\u003e ZED-F9P, \u003ca href=\"https://www.sparkfun.com/products/14980\"\u003eSparkFun GPS-RTK\u003c/a\u003e NEO-M8P-2, the \u003ca href=\"https://www.sparkfun.com/products/15210\"\u003eSparkFun SAM-M8Q\u003c/a\u003e, and the \u003ca href=\"https://www.sparkfun.com/products/15193\"\u003eSparkFun ZEO-M8Q\u003c/a\u003e. Library also works with other u-blox based boards. \u003cbr\u003e\u003cbr\u003eThe ZED-F9P and NEO-M8P-2 modules are top-of-the-line modules for high accuracy GNSS and GPS location solutions including RTK. The ZED-F9P is unique in that it is capable of both rover and base station operations allowing the module to become a base station and produce RTCM 3.x correction data.",
          "website": "https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library",
          "category": "Sensors",
          "architectures": [
            "*"
          ],
          "types": [
            "Contributed"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/sparkfun/SparkFun_u_blox_Arduino_Library-1.8.8.zip",
            "archive_filename": "SparkFun_u_blox_Arduino_Library-1.8.8.zip",
            "checksum": "SHA-256:a8c150dedf162de22f6df817dcb1d10528a898c04dfabcbcf4808ba693fadc83",
            "size": 313900,
            "cache_path": "libraries"
          }
        },
        "1.8.9": {
          "author": "SparkFun Electronics \u003ctechsupport@sparkfun.com\u003e",
          "version": "1.8.9",
          "maintainer": "SparkFun Electronics \u003csparkfun.com\u003e",
          "sentence": "Library for I2C and Serial Communication with u-blox modules",
          "paragraph": "An Arduino Library to enable both I2C and Serial communication for both NMEA reception and binary UBX sending to u-blox modules. Useful for interfacing to the \u003ca href=\"https://www.sparkfun.com/products/15136\"\u003eSparkFun GPS-RTK2\u003c/a\u003e ZED-F9P, \u003ca href=\"https://www.sparkfun.com/products/14980\"\u003eSparkFun GPS-RTK\u003c/a\u003e NEO-M8P-2, the \u003ca href=\"https://www.sparkfun.com/products/15210\"\u003eSparkFun SAM-M8Q\u003c/a\u003e, and the \u003ca href=\"https://www.sparkfun.com/products/15193\"\u003eSparkFun ZEO-M8Q\u003c/a\u003e. Library also works with other u-blox based boards. \u003cbr\u003e\u003cbr\u003eThe ZED-F9P and NEO-M8P-2 modules are top-of-the-line modules for high accuracy GNSS and GPS location solutions including RTK. The ZED-F9P is unique in that it is capable of both rover and base station operations allowing the module to become a base station and produce RTCM 3.x correction data.",
          "website": "https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library",
          "category": "Sensors",
          "architectures": [
            "*"
          ],
          "types": [
            "Contributed"
          ],
          "resources": {
            "url": "https://downloads.arduino.cc/libraries/github.com/sparkfun/SparkFun_u_blox_Arduino_Library-1.8.9.zip",
            "archive_filename": "SparkFun_u_blox_Arduino_Library-1.8.9.zip",
            "checksum": "SHA-256:d8a3eef76d2a66e7b9578a38752ba0e5e1859cea26aa73780fab4777a62f2088",
            "size": 321812,
            "cache_path": "libraries"
          }
        }
      },
      "latest": {
        "author": "SparkFun Electronics \u003ctechsupport@sparkfun.com\u003e",
        "version": "1.8.11",
        "maintainer": "SparkFun Electronics \u003csparkfun.com\u003e",
        "sentence": "DEPRECATED Library for I2C and Serial Communication with u-blox modules\u003cbr/\u003e\u003cbr/\u003e",
        "paragraph": "Please note: this library is deprecated. Please migrate to \u003ca href=\"https://github.com/sparkfun/SparkFun_u-blox_GNSS_Arduino_Library\"\u003ev2.0\u003c/a\u003e. You can find the migration instructions \u003ca href=\"https://github.com/sparkfun/SparkFun_u-blox_GNSS_Arduino_Library#migrating-to-v20\"\u003ehere\u003c/a\u003e.\u003cbr/\u003e\u003cbr/\u003eAn Arduino Library to enable both I2C and Serial communication for both NMEA reception and binary UBX sending to u-blox modules. Useful for interfacing to the \u003ca href=\"https://www.sparkfun.com/products/15136\"\u003eSparkFun GPS-RTK2\u003c/a\u003e ZED-F9P, \u003ca href=\"https://www.sparkfun.com/products/15005\"\u003eSparkFun GPS-RTK\u003c/a\u003e NEO-M8P-2, the \u003ca href=\"https://www.sparkfun.com/products/15210\"\u003eSparkFun SAM-M8Q\u003c/a\u003e, and the \u003ca href=\"https://www.sparkfun.com/products/15193\"\u003eSparkFun ZOE-M8Q\u003c/a\u003e. Library also works with other u-blox based boards.\u003cbr/\u003e\u003cbr/\u003eThe ZED-F9P and NEO-M8P-2 modules are top-of-the-line modules for high accuracy GNSS and GPS location solutions including RTK. The ZED-F9P is unique in that it is capable of both rover and base station operations allowing the module to become a base station and produce RTCM 3.x correction data.\u003cbr/\u003e",
        "website": "https://github.com/sparkfun/SparkFun_Ublox_Arduino_Library",
        "category": "Sensors",
        "architectures": [
          "*"
        ],
        "types": [
          "Contributed"
        ],
        "resources": {
          "url": "https://downloads.arduino.cc/libraries/github.com/sparkfun/SparkFun_u_blox_Arduino_Library-1.8.11.zip",
          "archive_filename": "SparkFun_u_blox_Arduino_Library-1.8.11.zip",
          "checksum": "SHA-256:d7367e582f3e3d336d435998f45c81a90c674be574972004e4bd44dc44f48024",
          "size": 327469,
          "cache_path": "libraries"
        }
      }
    }
  ],
  "status": 1
}

To reproduce

Compare the library search output of the CLI and the Arduino IDE 1.x. See the description for more details.

Expected behavior

The same lib/core search result is produced by the CLI and IDE 1.x.

Arduino CLI version

arduino-cli Version: 0.27.1 Commit: a900cfb Date: 2022-09-06T16:44:27Z

Operating system

macOS

Operating system version

12.5.1

Additional context

No response

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the nightly build
• My report contains all necessary details

[kittaakos]

Another alternative solution could be if IDE2 converted the http://librarymanager/All#SparkFun_u-blox_GNSS URL to the SparkFun u-blox GNSS query term. Note the _ to conversion, but then the IDE2 library manager query field will be populated with SparkFun u-blox GNSS and not SparkFun_u-blox_GNSS; hence the same behavior between IDE 1.x and IDE2 cannot be preserved. See arduino/arduino-ide#1442 (comment).

[cmaglie]

for reference here's how the Arduino IDE 1.8.x do the search, I'm going to implement the same search:

  private boolean filterCondition(ContributedLibraryReleases lib) {
    if (selectedCategoryFilter != null && !selectedCategoryFilter.test(lib)) {
      return false;
    }

    ContributedLibrary latest = lib.getLatest();
    String compoundTargetSearchText = latest.getName() + " "
                                      + latest.getParagraph() + " "
                                      + latest.getSentence();
    if (latest.getProvidesIncludes() != null) {
      compoundTargetSearchText += " " + latest.getProvidesIncludes();
    }
    if (!stringContainsAll(compoundTargetSearchText, selectedFilters)) {
      return false;
    }

    return true;
  }

  /**
   * Check if <b>string</b> contains all the substrings in <b>set</b>. The
   * compare is case insensitive.
   *
   * @param string
   * @param filters
   * @return <b>true<b> if all the strings in <b>set</b> are contained in
   *         <b>string</b>.
   */
  private boolean stringContainsAll(String string, String filters[]) {
    if (string == null) {
      return false;
    }

    if (filters == null) {
      return true;
    }

    for (String filter : filters) {
      if (!string.toLowerCase().contains(filter.toLowerCase())) {
        return false;
      }
    }

    return true;
  }

[kittaakos]

Thank you for taking care of it!

Where is the _ (underscore) to (space) escaping in the logic? The problem described in arduino/arduino-ide#1442 is unrelated to the predicate logic you have posted here. Please let me know if more input is required. I would like to avoid implementing something unnecessary.

[cmaglie]

Where is the _ (underscore) to (space) escaping in the logic?

Very good point... mmmmmmm

[cmaglie]

ok I think I found it:

  public void applyFilter() {
    String[] filteredText = new String[0];
    if (!showingHint) {
      String filter = getText().toLowerCase();
        
      // Replace anything but 0-9, a-z, or : with a space
      filter = filter.replaceAll("[^\\x30-\\x39^\\x61-\\x7a^\\x3a]", " ");
      
      filteredText = filter.split(" ");
    } 
    onFilter(filteredText);
  }

so in the query we must Replace anything but 0-9, a-z, or : with a space