-
Notifications
You must be signed in to change notification settings - Fork 365
adding GEM/webnlg prompts #743
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
jordiclive
merged 12 commits into
bigscience-workshop:eval-hackathon
from
jordiclive:webnlg_GEM
May 8, 2022
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
d1caa4c
adding GEM/webnlg prompts and Russian prompts, so can use challenge s…
jordiclive 26bbbb8
add GEM
jordiclive 2ee8081
Update templates.yaml
jordiclive 398d749
Update templates.yaml
jordiclive d1b3379
merging latest eval-hackathon
jordiclive b40bd1b
Merge remote-tracking branch 'origin/eval-hackathon' into webnlg_GEM
jordiclive 6fefd98
Update templates.yaml
jordiclive eb7edb0
Update templates.yaml
jordiclive 2ce7938
Update templates.yaml
jordiclive e472b81
Update templates.yaml
jordiclive a034f8e
Update templates.yaml
jordiclive 29ff9bf
Update templates.yaml
jordiclive File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,114 @@ | ||
dataset: GEM/web_nlg | ||
subset: en | ||
templates: | ||
38342608-5cd7-4ce7-b2e1-905ecd7f4c80: !Template | ||
answer_choices: null | ||
id: 38342608-5cd7-4ce7-b2e1-905ecd7f4c80 | ||
jinja: "{{input | join(\"; \")}}\nThe above is a collection of relations. Write\ | ||
\ descriptive English that contains this information.\n{% for i in references\ | ||
\ %}\n ||| {{ i }} \n{% endfor %}" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- BLEU | ||
- ROUGE | ||
original_task: true | ||
name: implicit-graph-description | ||
reference: '' | ||
3e41305c-5461-4cf3-853d-8a6fb5747623: !Template | ||
answer_choices: null | ||
id: 3e41305c-5461-4cf3-853d-8a6fb5747623 | ||
jinja: "I will verbalize an abstract representation of a sentence in natural language.\ | ||
\ To do so, I will first show the representation and then the natural language.\ | ||
\ The text needs to include all of the information in the representation.\n\n\ | ||
{{input | join(\", \")}} {% for i in references %}\n ||| {{ i }} \n{% endfor\ | ||
\ %}" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- BLEU | ||
- ROUGE | ||
original_task: true | ||
name: PALM_prompt | ||
reference: used-in-PALM | ||
426b682e-e801-4e8d-9ac3-5b676c9d3da2: !Template | ||
answer_choices: null | ||
id: 426b682e-e801-4e8d-9ac3-5b676c9d3da2 | ||
jinja: "A semantic triple is the atomic data entity in the Resource Description\ | ||
\ Framework (RDF) data model. As its name indicates, a triple is a set of three\ | ||
\ entities that codifies a statement about semantic data in the form of subject\u2013\ | ||
predicate\u2013object expressions. (e.g., \"Bob | is | 35\", or \"Bob | knows\ | ||
\ | John\"). \n\nA graph can be formed from a set of these triples. An example\ | ||
\ is {{input | join(\", \")}}. \n\nWrite grammatical text expressing all the\ | ||
\ relations succinctly and fluently.\n{% for i in references %}\n ||| {{ i\ | ||
\ }} \n{% endfor %}\n\n" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- ROUGE | ||
- BLEU | ||
original_task: true | ||
name: very-explicit-description | ||
reference: '' | ||
471c94de-2db2-48bf-a61f-89aedad19f56: !Template | ||
answer_choices: null | ||
id: 471c94de-2db2-48bf-a61f-89aedad19f56 | ||
jinja: "A semantic triple is a set of three entities that codifies a statement\ | ||
\ about semantic data in the form of subject\u2013predicate\u2013object expressions.\ | ||
\ (e.g., \"Bob | is | 35\", or \"Bob | knows | John\"). \n\nA graph to do with\ | ||
\ {{category}} can be formed from a set of these triples. An example is {{input\ | ||
\ | join(\", \")}}. \n\nWrite grammatical text expressing all the relations\ | ||
\ succinctly and fluently.\n{% for i in references %}\n ||| {{ i }} \n{% endfor\ | ||
\ %}\n\n" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- ROUGE | ||
- BLEU | ||
original_task: false | ||
name: use-category | ||
reference: '' | ||
9415bd8a-685f-4fa4-803a-f09bd47d4603: !Template | ||
answer_choices: null | ||
id: 9415bd8a-685f-4fa4-803a-f09bd47d4603 | ||
jinja: "I am taking this tabular data, where each row is separated by a \"|\"\ | ||
\ and expresses a relation between an object and a predicate : {{input | join(\"\ | ||
, \")}}. \n\nNow, I will produce a description of the tabular data using English sentences. {%\ | ||
\ for i in references %}\n ||| {{ i }} \n{% endfor %}" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- ROUGE | ||
- BLEU | ||
original_task: true | ||
name: non-explicit-description | ||
reference: Data-to-Text | ||
af519494-2a6a-4d83-a440-ee6d24df00e3: !Template | ||
answer_choices: null | ||
id: af519494-2a6a-4d83-a440-ee6d24df00e3 | ||
jinja: "I will take the following graph formed from relations (triples) separated\ | ||
\ by \"|\": {{input | join(\", \")}}. \n\nThen, I will verbalise the triple\ | ||
\ set into plain text. {% for i in references %}\n ||| {{ i }} \n{% endfor\ | ||
\ %}" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- BLEU | ||
- ROUGE | ||
original_task: true | ||
name: explicit-graph-description | ||
reference: 'Explicit Graph Description: verbalization.' | ||
afeec167-f75f-4687-a775-1efde7d04780: !Template | ||
answer_choices: null | ||
id: afeec167-f75f-4687-a775-1efde7d04780 | ||
jinja: "{{input | join(\", \")}}. \n\nThe above is a set of subject | predicate\ | ||
\ | object expressions separated by commas: \nWrite all the information in proper\ | ||
\ sentences. {% for i in references %}\n ||| {{ i }} \n{% endfor %}" | ||
metadata: !TemplateMetadata | ||
choices_in_prompt: false | ||
metrics: | ||
- ROUGE | ||
- BLEU | ||
original_task: true | ||
name: explicit-graph-description2 | ||
reference: " subject\u2013predicate\u2013object " |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how do we know this prompt was used in palm?

all i could find in their paper is the below ..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe @sebastianGehrmann shared some NLG prompts for PALM.