Skip to content

Support for multiple targets. #747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 2, 2022
Merged

Support for multiple targets. #747

merged 2 commits into from
May 2, 2022

Conversation

stephenbach
Copy link
Member

@stephenbach stephenbach commented Apr 27, 2022

THIS BREAKS BACKWARDS COMPATIBILITY WITH v0.2.x. Now, Template.apply now returns Tuple[str, List[str]] instead of Tuple[str, str]. The list of strings is the valid targets.

Users can add additional ||| to their templates to denote additional valid completions.

Example of the interface:
Screen Shot 2022-04-26 at 8 37 27 PM

@stephenbach stephenbach marked this pull request as ready for review April 27, 2022 16:07
@awebson awebson added the enhancement New feature or request label Apr 27, 2022
@jordiclive
Copy link
Contributor

@stephenbach @awebson , Can we merge this in? I have tested and written prompts for my two datasets with multi-target: #743 ,#740 (which also need review).

@cjlovering, This would require changes to eh. The Multi-ref Bleu and Multi-ref Rouge I have tested and seem to work, I can see Rouge is same as in GEM. I included a draft PR of a hacky way as I don't know the repo that well- but something like that

If this is done, I will try and implement SARI (as BLEU is pretty useless for Simplification) and some other GEM metrics for NLG.

@awebson
Copy link
Contributor

awebson commented May 2, 2022

Merging this now per discussion in our Monday (May 2) meeting. Thanks everyone!

@awebson awebson merged commit f6a0e21 into eval-hackathon May 2, 2022
@jon-tow jon-tow mentioned this pull request May 3, 2022
@stephenbach stephenbach deleted the multitarget branch May 18, 2022 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants