Add support for "extra", e.g. private or local licenses

[pombredanne]

At times, someone may have some special notices that may not be worthy of landing in the main scancode licenses dataset. These could be your own proprietary notices or some composite license that you like to have named as such or it could be some advanced fixes that you maintain locally and you do not want yet to upgrade to a newer develop version of scancode. Regardless of the why this is needed, it would be great to have an option to reference an extra, local directory of license texts and rules that would be included in the license detection perimeter.

This could simply take the form of a new command line option such as --extra-licenses or --additional-licenses that would point to a directory with the same structure as the license data set e.g with a licenses and a rules subdirectory and when using this option, these extra licenses would validated and loaded in the index the same way that the standard ones are.

Some internal caveats to consider is the license index creation and cache invalidation:

should there be a separate index built or not? and if so then a separate cache might be needed.
If a single index is used, then the cache (and its cache hash) should be built considering these extra licenses files (in addition to the actual scancode source code files and its license and rule files) which means that if you run scancode once without extra, an index cache is built.

If you run with --extra that cache would be invalidated and a new cache built considering the extra. And if you run scancode again without extra or with a different extra, the cache would be again invalidated and rebuilt. This is not a major issue but may come with small perf penalty when switching usage between with or without extra or using different extra.

[DennisClark]

Transferring the following comments from #913.
A business may discover multiple commercial and other proprietary licenses in its product code, or in third-party products that it is evaluating for usage. These notices often appear as proprietary copyright and/or license notices in source file headers. Since the information in those license notices might be sensitive, it would be very useful to:

Collect these notices+detection rules in a custom directory path, which could then be used in addition to the standard ScanCode licenses by a runtime option.

Designating a custom directory path outside of the standard ScanCode directory would help to manage this custom collection and help to protect it from getting wiped out by a ScanCode upgrade.

[pombredanne]

@richardfontana FYI

[pombredanne]

@tsteenbe @sschuberth @haikoschol ping as this was of interest to you too

[pombredanne]

I am merging the ticket #2471here to have a single one

Custom License Rules folder #2471 from @tardyp
Short Description
Our internal code has copyright headers that we would like to properly categorize.
We don't think it make sense to upstream those rules, and we want to avoid forking scancode.
Thus we would like to add an option to scan code to provide a folder path which would contain custom .yml + RULE files.

In #2471 (comment) I wrote:

Now if these are just a few proprietary license and headers, it could be well worth adding them to scancode anyway.

And to implement this feature here are some thoughts:

A) the base approach to get these the extra rules in scancode:

1. a directory that contains extra rules and you could point to with some command line argument
2. a "plugin" where we package extra licenses and rules in a Python package and that can be installed as some private extra locally.

I am leaning towards 2. as otherwise this may be complicated to deploy this.

B) how these rules and license would be consumed:

1. they could be merged in scancode main index
2. they could be included in their own secondary index (with either A.1 or A.2) and the detection would run using this (or these) extra indexes either before of after the main index, and the matched results merged

I am not sure which is best.

in #2471 (comment) @tardyp wrote:

I didn't see #480, as I only focused my search on the keyword RULES.

I like what I see there, especially the idea from @DennisClark to automatically create this custom folder based on Unknown License findings.

In my first scans with scancode, we end up with big pile of unknown license, which is normal as we want to use scancode to make sure our proprietary software is not mixed up with open-source, and that our devs use packaging techniques to compose software.

I spent some time yesterday to experiment with the source code of scancode, and indeed dicovered the huge license library and the need to cache the index.

I am not sure if for custom license there is really a usecase where those number will be so big that they need to be cached as well. The needed cache module refactoring seems quite scary to me.

What I like with secondary index is that we could skip primary matching all together if the secondary index match score is high enough.

This could open the path to a quick scan mode that we could put in the pre-commit CI.

in @tardyp wrote:

incorrect detection is treated as a bug

I don't say it is incorrect detection, as those are mostly files, which are our proprietary license, and I don't expect scancode to magically detect it.
We have our own spdx identifiers, and scancode detects that as unknown-spdx, maybe that could be enhanced

Question: if you were to use a secondary index in your case, would you see an exclusive us of that index for a given scan run and not the main one? or would you see the use of boths at the same time?

I would use both.

For each file, if the secondary index detects with 100% score that this is our copyright, don't bother run the rest of the rules.
If a file does not match one of our proprietary licenses, we try to detect based on the primary database (and we afford this takes 250ms per file)

[pombredanne]

Here is an idea for a high level design:

• we can have multiple licenses and rules data sets. Each is packaged as a plugin and needs to declare itself with a name that must be unique. This applies also to the standard built-in rules data sets.
• at run time, we can select with an option which license indexes to use, e.g. one more more with a UX more or less like: --license-index=builtin,foo,bar and a default to builtin

With this the code would deal with the specifics of creating and caching the indexes and merging and filtering the results as appropriate.

This could therefore support not only private indexes, but also special purpose indexes such as indexes for other languages #139 and focused indexes on commercial and proprietary licenses for instance.

[tardyp]

I like this design proposal. I would like to also introduce a simple test mechanism that would lower the bar for adoption as much as possible.
I would propose an data/examples directory alongside the licenses and rules directory.

this directory will contain a list of .yml files and source files associated to the test

• data/examples/acme.c # contains the file to be scanned
• data/examples/acme.yml # contains the results that are supposed to be found:

    - 
      rule:
        identifier: "spdx-license-identifier: commercial-license-acme"
      score: 100
    - 
      rule:
        identifier: "commercial-license-acme.RULE"
      score: 100

This would help the non-expert to confidently write rules without having to run a full scan each time, and help tracability of the custom rules.

[pombredanne]

@tardyp re:

I like this design proposal. I would like to also introduce a simple test mechanism that would lower the bar for adoption as much as possible.
I would propose an data/examples directory alongside the licenses and rules directory.

This makes perfect sense. Could this be essentially the same as in https://github.com/nexB/scancode-toolkit/tree/develop/tests/licensedcode/data/datadriven/lic1 ? we have a pair of test file and expected results in YAML. This could be enhanced to also support expecting a score (or score range?) and a certain license or rule.

[tardyp]

Cool. I was sure that there was already something like this in scancode...

So looks like all you have to do is to do create a test like this:

https://github.com/nexB/scancode-toolkit/blob/develop/tests/licensedcode/test_detection_datadriven1.py

I see that this is implemented here:

https://github.com/nexB/scancode-toolkit/blob/develop/tests/licensedcode/licensedcode_test_utils.py#L173

shall we put licensedcode_test_utils in the main namespace? aren't you stripping (or willing to strip) the tests from the main wheel?

[pombredanne]

shall we put licensedcode_test_utils in the main namespace?

yes, with a little refactoring.

aren't you stripping (or willing to strip) the tests from the main wheel?

there are a few things to consider IMHO here:

1. we want to be able to run extra tests as contributed by an index plugin and we may want to avoid any additional test library dependencies (e.g. pytest, etc) as I would prefer to avoid having these as core deps. So the tests should be usable without the tests libraries.

2. the main, standard index tests can stay a special case for now as long the main index is provided builtin.