Skip to content

Commit fb4c27d

Browse files
rbawdenRachel Bawdenstephenbach
authored
Add prompts for DiaBLa dataset (#759)
* added first templates for DiaBLa dataset rbawden/DiaBLa * declared previous_ref at beginning of templates * declared more variables at beginning of templates * declared more variables at beginning of templates * declared more variables at beginning of templates * declared more variables at beginning of templates * corrected ref for mt in one template * corrected ref for mt in one template * moved condition to just target side rather than around entire prompt * allow template even when no context (beginning of dialogue) * corrected templates that use past history * corrected multiple choice answer field * updated templates - simplified some targets and added translation completion tasks (simpler?) * corrected duplicate name * updated duplicate definition * corrected error of only two pipes in answer choices * corrected -2 index to -1 - duplicate defintiions * merged with eval-hackathon updates * corrected discriminate mt ref * removed directional templates and only keep both directions (analysis to be done separately later. Also some slight modifs such as removing excess words and quotes * simplified names, changed random to choice Co-authored-by: Rachel Bawden <[email protected]@users.noreply.github.com> Co-authored-by: Stephen Bach <[email protected]>
1 parent a0a7258 commit fb4c27d

File tree

2 files changed

+247
-1
lines changed

2 files changed

+247
-1
lines changed

‎promptsource/templates.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626

2727
# These are users whose datasets should be included in the results returned by
2828
# filter_english_datasets (regardless of their metadata)
29-
INCLUDED_USERS = {"Zaid", "craffel", "GEM", "aps", "khalidalt", "shanya"}
29+
INCLUDED_USERS = {"Zaid", "craffel", "GEM", "aps", "khalidalt", "shanya", "rbawden"}
3030

3131

3232
def highlight(input):

‎promptsource/templates/rbawden/DiaBLa/templates.yaml

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
dataset: rbawden/DiaBLa
2+
templates:
3+
2731216a-b994-48f9-aaf6-00c7038bbed5: !Template
4+
answer_choices: null
5+
id: 2731216a-b994-48f9-aaf6-00c7038bbed5
6+
jinja: "{% set first_lang=\"\" %}\n{% if dialogue_history|length > 0 %}\nGiven\
7+
\ the following dialogue between person A and person B:\n\n{% set first_lang=dialogue_history[-5:][0].utterance_meta.lang\
8+
\ %}{% for previous in dialogue_history[-5:] %}{% if previous.utterance_meta.lang\
9+
\ == first_lang %}A{% else %}B{% endif %}: {% if previous.utterance_meta.lang\
10+
\ == utterance_meta.lang %}{{ previous.orig }}{% else %}{{ previous.ref }}{%\
11+
\ endif %}\n{% endfor %}{% endif %} \nTranslate {% if utterance_meta.lang ==\
12+
\ first_lang %}A{% else %}B{% endif %}'s next utterance into {% if utterance_meta.lang\
13+
\ == \"english\" %}French{% else %}English{% endif %}: {{ orig }}\n\n||| {{\
14+
\ ref }}"
15+
metadata: !TemplateMetadata
16+
choices_in_prompt: false
17+
metrics:
18+
- BLEU
19+
original_task: true
20+
name: MT with dialogue context (5 same context ref)
21+
reference: MT with dialogue context (up to 5 previous sentences, same language
22+
ref)
23+
28ea04f4-338e-40cf-8730-4a794b5b64b2: !Template
24+
answer_choices: yes ||| no
25+
id: 28ea04f4-338e-40cf-8730-4a794b5b64b2
26+
jinja: "{% set options = [\"word choice\", \"grammar\", \"style\", \"coherence\"\
27+
, \"meaning\"] %}\n{% set label = range(0,5)|choice %}\n{% set reply=0 %}\n\
28+
{% set first_lang=\"\" %}\n{% if options[label] in utterance_meta.eval_problems\
29+
\ %}{% set reply=0 %}{% else %}{% set reply=1 %}{% endif %}\n{% if dialogue_history|length\
30+
\ > 0 %}\nGiven the following dialogue between person A and person B:\n\n{%\
31+
\ set first_lang=dialogue_history[-5:][0].utterance_meta.lang %}{% for previous\
32+
\ in dialogue_history[-5:] %}\n{% if previous.utterance_meta.lang == first_lang\
33+
\ %}A{% else %}B{% endif %}: {% if previous.utterance_meta.lang != utterance_meta.lang\
34+
\ %}{{ previous.orig }}{% else %}{{ previous.mt }}{% endif %}{% endfor %}{%\
35+
\ endif %} \n{% if utterance_meta.lang == first_lang %}A{% else %}B{% endif\
36+
\ %}: {{ mt }}\n\nDoes the last utterance contain a {{ options[label] }} problem,\
37+
\ {{ \"yes\" }} or {{ \"no\" }}?\n\n||| {{ [\"yes\", \"no\" ][reply] }}"
38+
metadata: !TemplateMetadata
39+
choices_in_prompt: true
40+
metrics:
41+
- Accuracy
42+
original_task: false
43+
name: Is the error present? (same lang)
44+
reference: Identify presence of notable errors given 5 context sentences (same
45+
lang mt)
46+
6a01fbe6-d5ec-4ad9-a2ee-3c48ed095885: !Template
47+
answer_choices: null
48+
id: 6a01fbe6-d5ec-4ad9-a2ee-3c48ed095885
49+
jinja: '{% set previous_ref = "" %}{% set other_lang = "" %}
50+
51+
{% if utterance_meta.lang == "french" %}{% set other_lang = "English" %}{% else
52+
%}{% set other_lang = "French" %}{% endif %}
53+
54+
{% if dialogue_history|length > 0 %}
55+
56+
"{% if utterance_meta.lang == dialogue_history[-1].utterance_meta.lang %}{{
57+
dialogue_history[-1].orig }}{% set previous_ref = dialogue_history[-1].ref %}{%
58+
else %}{{ dialogue_history[-1].ref }}{% set previous_ref = dialogue_history[-1].orig
59+
%}{% endif %}" translates into {{ other_lang }} as: {{ previous_ref }}{% endif
60+
%}
61+
62+
63+
"{{ orig }}" translates into {{ other_lang }} as: ||| {{ ref }}
64+
65+
'
66+
metadata: !TemplateMetadata
67+
choices_in_prompt: false
68+
metrics:
69+
- BLEU
70+
original_task: true
71+
name: MT analogy, lang given (1 same context)
72+
reference: MT task, translation by analogy (same language), target language provided
73+
842dc41a-8af0-4dca-8b55-a87026bfac31: !Template
74+
answer_choices: null
75+
id: 842dc41a-8af0-4dca-8b55-a87026bfac31
76+
jinja: 'Translate this into {% if utterance_meta.lang == "english" %}French{%
77+
else %}English{% endif %}: {{ orig }} ||| {{ ref }}'
78+
metadata: !TemplateMetadata
79+
choices_in_prompt: false
80+
metrics:
81+
- BLEU
82+
original_task: true
83+
name: MT sentence-level
84+
reference: Sentence-level
85+
93f5256d-bd93-4056-b466-152b55860d02: !Template
86+
answer_choices: null
87+
id: 93f5256d-bd93-4056-b466-152b55860d02
88+
jinja: '{% set first_lang="" %}
89+
90+
{% if dialogue_history|length > 0 %}
91+
92+
Given the following dialogue between person A and person B:
93+
94+
95+
{% set first_lang=dialogue_history[-5:][0].utterance_meta.lang %}{% for previous
96+
in dialogue_history[-5:] %}{% if previous.utterance_meta.lang == first_lang
97+
%}A{% else %}B{% endif %}: {{ previous.orig }}
98+
99+
{% endfor %}{% endif %}
100+
101+
Translate {% if utterance_meta.lang == first_lang %}A{% else %}B{% endif %}''s
102+
next utterance into {% if utterance_meta.lang == "english" %}French{% else %}English{%
103+
endif %}: {{ orig }}
104+
105+
106+
||| {{ ref }}'
107+
metadata: !TemplateMetadata
108+
choices_in_prompt: false
109+
metrics:
110+
- BLEU
111+
original_task: true
112+
name: MT with dialogue context (5 orig context)
113+
reference: MT with dialogue context (up to 5 previous sentences, original language)
114+
a7511f73-dd28-449f-bc6c-9609b736bb40: !Template
115+
answer_choices: null
116+
id: a7511f73-dd28-449f-bc6c-9609b736bb40
117+
jinja: '{% set target_lang = "english" %}{% if utterance_meta.lang == "english"
118+
%}{% set target_lang = "french" %}{% endif %}
119+
120+
{% for previous in dialogue_history[-2:] %}
121+
122+
{{ previous.orig }}{% endfor %}
123+
124+
{{ orig }}
125+
126+
127+
The {% if utterance_meta.lang == "english" %}French{% else %}English{% endif
128+
%} translation is:
129+
130+
{% for previous in dialogue_history[-2:] %}{% if previous.utterance_meta.lang
131+
== target_lang %}{{ previous.orig }}{% else %}{{ previous.mt }}{% endif %}
132+
133+
{% endfor %} ||| {{ ref }}'
134+
metadata: !TemplateMetadata
135+
choices_in_prompt: false
136+
metrics:
137+
- BLEU
138+
original_task: true
139+
name: MT complete translation (2 orig context)
140+
reference: MT task, complete the translation, 2 sentences of previous context
141+
(orig language)
142+
ac4c63da-32d2-40ac-aa7a-632e8ba42b4a: !Template
143+
answer_choices: A ||| B
144+
id: ac4c63da-32d2-40ac-aa7a-632e8ba42b4a
145+
jinja: '{% set label = [''A'',''B'']|choice %}
146+
147+
Which of the following translations of "{{ orig }}" is produced automatically?
148+
149+
{{ "A" }}) {% if label==''A'' %}{{ mt }}{% else %}{{ ref }}{% endif %}
150+
151+
{{ "B" }}) {% if label==''A'' %}{{ ref }}{% else %}{{ mt }}{% endif %}
152+
153+
|||{{ label }}'
154+
metadata: !TemplateMetadata
155+
choices_in_prompt: true
156+
metrics:
157+
- Accuracy
158+
original_task: false
159+
name: Which is automatic?
160+
reference: Identify MT output between MT and ref
161+
b61c81ec-29eb-47f8-a1c6-561264ac04f3: !Template
162+
answer_choices: null
163+
id: b61c81ec-29eb-47f8-a1c6-561264ac04f3
164+
jinja: '{% set first_lang="" %}
165+
166+
{% if dialogue_history|length > 0 %}
167+
168+
Given the following dialogue between person A and person B:
169+
170+
171+
{% set first_lang=dialogue_history[-5:][0].utterance_meta.lang %}{% for previous
172+
in dialogue_history[-5:] %}{% if previous.utterance_meta.lang == first_lang
173+
%}A{% else %}B{% endif %}: {% if previous.utterance_meta.lang == utterance_meta.lang
174+
%}{{ previous.orig }}{% else %}{{ previous.mt }}{% endif %}
175+
176+
{% endfor %}{% endif %}
177+
178+
Translate {% if utterance_meta.lang == first_lang %}A{% else %}B{% endif %}''s
179+
next utterance into {% if utterance_meta.lang == "english" %}French{% else %}English{%
180+
endif %}: {{ orig }}
181+
182+
183+
||| {{ ref }}'
184+
metadata: !TemplateMetadata
185+
choices_in_prompt: false
186+
metrics:
187+
- BLEU
188+
original_task: true
189+
name: MT with dialogue context (5 same context mt)
190+
reference: MT with dialogue context (up to 5 previous sentences, same language
191+
MT)
192+
e9f5dfda-42b5-4698-8e8d-9fc9924e2e29: !Template
193+
answer_choices: null
194+
id: e9f5dfda-42b5-4698-8e8d-9fc9924e2e29
195+
jinja: '{% set target_lang = "english" %}{% if utterance_meta.lang == "english"
196+
%}{% set target_lang = "french" %}{% endif %}
197+
198+
{% for previous in dialogue_history[-1:] %}
199+
200+
{{ previous.orig }}{% endfor %}
201+
202+
{{ orig }}
203+
204+
205+
The {% if utterance_meta.lang == "english" %}French{% else %}English{% endif
206+
%} translation is:
207+
208+
{% for previous in dialogue_history[-1:] %}{% if previous.utterance_meta.lang
209+
== target_lang %}{{ previous.orig }}{% else %}{{ previous.mt }}{% endif %}
210+
211+
{% endfor %} ||| {{ ref }}'
212+
metadata: !TemplateMetadata
213+
choices_in_prompt: false
214+
metrics:
215+
- BLEU
216+
original_task: true
217+
name: MT complete translation (1 orig context)
218+
reference: MT task, complete the translation, 1 sentence of previous context (orig
219+
language)
220+
eea8f47e-9bf5-4423-980b-58a9635c1f49: !Template
221+
answer_choices: null
222+
id: eea8f47e-9bf5-4423-980b-58a9635c1f49
223+
jinja: '{% set previous_ref = "" %}{% set other_lang = "" %}
224+
225+
{% if dialogue_history|length > 0 %}
226+
227+
{% if utterance_meta.lang == "french" %}{% set other_lang = "English" %}{% else
228+
%}{% set other_lang = "French" %}{% endif %}
229+
230+
"{% if utterance_meta.lang == dialogue_history[-1].utterance_meta.lang %}{{
231+
dialogue_history[-1].orig }}{% set previous_ref = dialogue_history[-1].ref %}{%
232+
else %}{{ dialogue_history[-1].ref }}{% set previous_ref = dialogue_history[-1].orig
233+
%}{% endif %}" translates as: {{ previous_ref }}{% endif %}
234+
235+
236+
"{{ orig }}" translates as: ||| {% if dialogue_history|length > 0 %}{{ ref }}{%
237+
endif %}
238+
239+
'
240+
metadata: !TemplateMetadata
241+
choices_in_prompt: false
242+
metrics:
243+
- BLEU
244+
original_task: true
245+
name: MT analogy, infer lang (1 same context)
246+
reference: MT task, translation by analogy (same language), infer target language

0 commit comments

Comments
 (0)