Skip to content

Commit 29afd13

Browse files
fyvoVictorSanh
fyvo
and
VictorSanh
authored
Eval hackathon - WMT (#752)
* test prompts for newsco * sample prompts for zero-shot mt * one more pattern for wmt14/en-fr * Fix an inverted fr-en prompt for wmt14 * Prompts for WMT14 de-en, adapted directly from fr-en * fixed duplicate uuids, add a new set of prompts * restore the original template * remove prompts in the French language to simplify copies * remove faulty german prompts * Add prompts for WMT14 hi-En * Prompts for wmt14 cs-en * prompts for wmt14-wmt19 all news tasks * added 4 new glm like prompts for wmt* tasks * Small change to the translate-* family of prompts - now Translate this from X into Y * fix names Co-authored-by: Victor Sanh <[email protected]>
1 parent 59cb306 commit 29afd13

38 files changed

+13110
-0
lines changed

‎promptsource/templates/wmt14/cs-en/templates.yaml

Lines changed: 345 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,345 @@
1+
dataset: wmt14
2+
subset: cs-en
3+
templates:
4+
027bdc9b-aed7-4509-94ea-38b138f14ddf: !Template
5+
answer_choices: null
6+
id: 027bdc9b-aed7-4509-94ea-38b138f14ddf
7+
jinja: 'Translate this from Czech into English: {{translation["cs"]}}
8+
9+
||| {{translation["en"]}}'
10+
metadata: !TemplateMetadata
11+
choices_in_prompt: false
12+
metrics:
13+
- BLEU
14+
original_task: true
15+
name: translate-cs-en-source+target
16+
reference: https://arxiv.org/pdf/1910.10683.pdf
17+
16da9664-30e5-4acc-aff6-21658158cf85: !Template
18+
answer_choices: null
19+
id: 16da9664-30e5-4acc-aff6-21658158cf85
20+
jinja: 'What is the Czech translation of: {{translation["en"]}}
21+
22+
||| {{translation["cs"]}}'
23+
metadata: !TemplateMetadata
24+
choices_in_prompt: false
25+
metrics:
26+
- BLEU
27+
original_task: true
28+
name: gpt-3-en-cs-target
29+
reference: GPT-3 paper
30+
3b3ca652-8414-4442-b55f-c2fc7630b071: !Template
31+
answer_choices: null
32+
id: 3b3ca652-8414-4442-b55f-c2fc7630b071
33+
jinja: 'Given the following source text in English: {{translation["en"]}} , a
34+
good Czech translation is:
35+
36+
||| {{translation["cs"]}}'
37+
metadata: !TemplateMetadata
38+
choices_in_prompt: false
39+
metrics:
40+
- BLEU
41+
original_task: true
42+
name: a_good_translation-en-cs-source+target
43+
reference: ''
44+
4266e99a-41fa-4c2d-8bbc-91317f42d9ff: !Template
45+
answer_choices: null
46+
id: 4266e99a-41fa-4c2d-8bbc-91317f42d9ff
47+
jinja: 'What is the English translation of the Czech sentence: {{translation["cs"]}}
48+
49+
||| {{translation["en"]}}'
50+
metadata: !TemplateMetadata
51+
choices_in_prompt: false
52+
metrics:
53+
- BLEU
54+
original_task: true
55+
name: gpt-3-cs-en-source+target
56+
reference: GPT-3 paper
57+
46621847-e018-43fe-b8d1-af439e130254: !Template
58+
answer_choices: null
59+
id: 46621847-e018-43fe-b8d1-af439e130254
60+
jinja: 'What is the Czech translation of the English sentence: {{translation["en"]}}
61+
62+
||| {{translation["cs"]}}'
63+
metadata: !TemplateMetadata
64+
choices_in_prompt: false
65+
metrics:
66+
- BLEU
67+
original_task: true
68+
name: gpt-3-en-cs-source+target
69+
reference: GPT 3 paper
70+
4c52dc6a-f16c-4f3c-99d0-274b4162a616: !Template
71+
answer_choices: null
72+
id: 4c52dc6a-f16c-4f3c-99d0-274b4162a616
73+
jinja: 'If the original version says: {{translation["en"]}}; then the Czech version
74+
should say:
75+
76+
||| {{translation["cs"]}}'
77+
metadata: !TemplateMetadata
78+
choices_in_prompt: false
79+
metrics:
80+
- BLEU
81+
original_task: true
82+
name: version-en-cs-target
83+
reference: ''
84+
5b87693b-ed41-4794-bb3e-b9bf7ada2689: !Template
85+
answer_choices: null
86+
id: 5b87693b-ed41-4794-bb3e-b9bf7ada2689
87+
jinja: 'Given the following passage: {{translation["en"]}} , a good Czech translation
88+
is: ||| {{translation["cs"]}}'
89+
metadata: !TemplateMetadata
90+
choices_in_prompt: false
91+
metrics:
92+
- BLEU
93+
original_task: true
94+
name: a_good_translation-en-cs-target
95+
reference: ''
96+
5e476fb7-5e6e-4732-a084-88e361a82de0: !Template
97+
answer_choices: null
98+
id: 5e476fb7-5e6e-4732-a084-88e361a82de0
99+
jinja: '{{translation["cs"]}} translates into English as:
100+
101+
||| {{translation["en"]}}'
102+
metadata: !TemplateMetadata
103+
choices_in_prompt: false
104+
metrics:
105+
- BLEU
106+
original_task: true
107+
name: translate_as_cs-en-target
108+
reference: ''
109+
744bab4b-44d1-4368-a98d-4a403c1a7e0e: !Template
110+
answer_choices: null
111+
id: 744bab4b-44d1-4368-a98d-4a403c1a7e0e
112+
jinja: 'Translate this into English: {{translation["cs"]}}
113+
114+
||| {{translation["en"]}}'
115+
metadata: !TemplateMetadata
116+
choices_in_prompt: false
117+
metrics:
118+
- BLEU
119+
original_task: true
120+
name: translate-cs-en-target
121+
reference: ''
122+
7fd9d8d7-0c72-42da-a029-ef4d9a9e5f72: !Template
123+
answer_choices: null
124+
id: 7fd9d8d7-0c72-42da-a029-ef4d9a9e5f72
125+
jinja: 'Given the following source text in Czech: {{translation["cs"]}} , a good
126+
English translation is: ||| {{translation["en"]}}'
127+
metadata: !TemplateMetadata
128+
choices_in_prompt: false
129+
metrics:
130+
- BLEU
131+
original_task: true
132+
name: a_good_translation-cs-en-source+target
133+
reference: ''
134+
809cc7a3-189b-4018-8f81-1e65c2f42271: !Template
135+
answer_choices: null
136+
id: 809cc7a3-189b-4018-8f81-1e65c2f42271
137+
jinja: 'Given the following passage: {{translation["cs"]}} , a good English translation
138+
is:
139+
140+
||| {{translation["en"]}}'
141+
metadata: !TemplateMetadata
142+
choices_in_prompt: false
143+
metrics:
144+
- BLEU
145+
original_task: true
146+
name: a_good_translation-cs-en-target
147+
reference: ''
148+
9023b083-ccf1-42a4-9589-5fffd3562649: !Template
149+
answer_choices: null
150+
id: 9023b083-ccf1-42a4-9589-5fffd3562649
151+
jinja: 'If the Czech version says: {{translation["cs"]}}; then the English version
152+
should say:
153+
154+
||| {{translation["en"]}}'
155+
metadata: !TemplateMetadata
156+
choices_in_prompt: false
157+
metrics:
158+
- BLEU
159+
original_task: true
160+
name: version-cs-en-source+target
161+
reference: ''
162+
92e3a4a7-e7c2-4ea1-abfb-be63069d13fa: !Template
163+
answer_choices: null
164+
id: 92e3a4a7-e7c2-4ea1-abfb-be63069d13fa
165+
jinja: 'Translate this into Czech: {{translation["en"]}}
166+
167+
||| {{translation["cs"]}}'
168+
metadata: !TemplateMetadata
169+
choices_in_prompt: false
170+
metrics:
171+
- BLEU
172+
original_task: true
173+
name: translate-en-cs-target
174+
reference: ''
175+
a63daae0-77ba-4790-a89d-2428193fbcdc: !Template
176+
answer_choices: null
177+
id: a63daae0-77ba-4790-a89d-2428193fbcdc
178+
jinja: 'What is the English translation of : {{translation["cs"]}}
179+
180+
||| {{translation["en"]}}'
181+
metadata: !TemplateMetadata
182+
choices_in_prompt: false
183+
metrics:
184+
- BLEU
185+
original_task: true
186+
name: gpt-3-cs-en-target
187+
reference: GPT-3 paper
188+
a63fd691-4f07-41da-a98c-a934e866e74e: !Template
189+
answer_choices: null
190+
id: a63fd691-4f07-41da-a98c-a934e866e74e
191+
jinja: '{{translation["cs"]}} = English:
192+
193+
||| {{translation["en"]}}'
194+
metadata: !TemplateMetadata
195+
choices_in_prompt: false
196+
metrics:
197+
- BLEU
198+
original_task: true
199+
name: xglm-cs-en-target
200+
reference: XGLM paper https://arxiv.org/abs/2112.10668
201+
ab264706-636e-4bd0-a762-860029b54f45: !Template
202+
answer_choices: null
203+
id: ab264706-636e-4bd0-a762-860029b54f45
204+
jinja: 'Translate this from English into Czech: {{translation["en"]}}
205+
206+
||| {{translation["cs"]}}'
207+
metadata: !TemplateMetadata
208+
choices_in_prompt: false
209+
metrics:
210+
- BLEU
211+
original_task: true
212+
name: translate-en-cs-source+starget
213+
reference: ''
214+
ac25c74b-1a81-409e-9c4f-bdc94912721c: !Template
215+
answer_choices: null
216+
id: ac25c74b-1a81-409e-9c4f-bdc94912721c
217+
jinja: ' {{translation["en"]}} translates into Czech as:
218+
219+
||| {{translation["cs"]}}'
220+
metadata: !TemplateMetadata
221+
choices_in_prompt: false
222+
metrics:
223+
- BLEU
224+
original_task: true
225+
name: translate_as_en-cs-target
226+
reference: ''
227+
b42858a8-35d7-4172-ace4-80df1f6a94e0: !Template
228+
answer_choices: null
229+
id: b42858a8-35d7-4172-ace4-80df1f6a94e0
230+
jinja: 'If the English version says: {{translation["en"]}}; then the Czech version
231+
should say:
232+
233+
||| {{translation["cs"]}}'
234+
metadata: !TemplateMetadata
235+
choices_in_prompt: false
236+
metrics:
237+
- BLEU
238+
original_task: true
239+
name: version-en-cs-source+target
240+
reference: ''
241+
c4629789-74ce-4378-8d03-f75b2327ad77: !Template
242+
answer_choices: null
243+
id: c4629789-74ce-4378-8d03-f75b2327ad77
244+
jinja: 'English: {{translation["en"]}} = Czech:
245+
246+
||| {{translation["cs"]}}'
247+
metadata: !TemplateMetadata
248+
choices_in_prompt: false
249+
metrics:
250+
- BLEU
251+
original_task: true
252+
name: xglm-en-cs-source-target
253+
reference: Adapted from XGLM paper on few shot evaluation https://arxiv.org/abs/2112.10668
254+
cdce6dd9-52ad-42c3-a691-8efb13ab535f: !Template
255+
answer_choices: null
256+
id: cdce6dd9-52ad-42c3-a691-8efb13ab535f
257+
jinja: 'Czech: {{translation["cs"]}} translates into English as:
258+
259+
||| {{translation["en"]}}'
260+
metadata: !TemplateMetadata
261+
choices_in_prompt: false
262+
metrics:
263+
- BLEU
264+
original_task: true
265+
name: translate_as_cs-en-source+target
266+
reference: ''
267+
dcefc4dc-560d-4bf5-8623-c99d535232b0: !Template
268+
answer_choices: null
269+
id: dcefc4dc-560d-4bf5-8623-c99d535232b0
270+
jinja: 'How do you say {{translation["cs"]}} in English?
271+
272+
||| {{translation["en"]}}'
273+
metadata: !TemplateMetadata
274+
choices_in_prompt: false
275+
metrics:
276+
- BLEU
277+
original_task: true
278+
name: how_to_say-cs-en-target
279+
reference: ''
280+
e13ee973-fce4-4d42-8ec5-b871c20eaf39: !Template
281+
answer_choices: null
282+
id: e13ee973-fce4-4d42-8ec5-b871c20eaf39
283+
jinja: 'Czech: {{translation["cs"]}} = English:
284+
285+
||| {{translation["en"]}}'
286+
metadata: !TemplateMetadata
287+
choices_in_prompt: false
288+
metrics:
289+
- BLEU
290+
original_task: true
291+
name: xglm-cs-en-source+target
292+
reference: XGLM paper https://arxiv.org/abs/2112.10668
293+
ee298b4c-8165-44cc-81b6-1fb18f4e9e02: !Template
294+
answer_choices: null
295+
id: ee298b4c-8165-44cc-81b6-1fb18f4e9e02
296+
jinja: 'English: {{translation["en"]}} translates into Czech as:
297+
298+
||| {{translation["cs"]}}'
299+
metadata: !TemplateMetadata
300+
choices_in_prompt: false
301+
metrics:
302+
- BLEU
303+
original_task: true
304+
name: translate_as_en-cs-source+target
305+
reference: ''
306+
f7af3145-3444-4183-921b-a549fef5e1b3: !Template
307+
answer_choices: null
308+
id: f7af3145-3444-4183-921b-a549fef5e1b3
309+
jinja: 'If the original version says: {{translation["cs"]}}; then the English
310+
version should say:
311+
312+
||| {{translation["en"]}}'
313+
metadata: !TemplateMetadata
314+
choices_in_prompt: false
315+
metrics:
316+
- BLEU
317+
original_task: true
318+
name: version-cs-en-target
319+
reference: ''
320+
f7d630ca-1791-4aaf-97b5-7c15e5eba296: !Template
321+
answer_choices: null
322+
id: f7d630ca-1791-4aaf-97b5-7c15e5eba296
323+
jinja: 'How do you say {{translation["en"]}} in Czech?
324+
325+
||| {{translation["cs"]}}'
326+
metadata: !TemplateMetadata
327+
choices_in_prompt: false
328+
metrics:
329+
- BLEU
330+
original_task: true
331+
name: how_to_say-en-cs-target
332+
reference: ''
333+
f83cb5c3-7989-49d9-a1cb-6fdc16b445e3: !Template
334+
answer_choices: null
335+
id: f83cb5c3-7989-49d9-a1cb-6fdc16b445e3
336+
jinja: '{{translation["en"]}} = Czech:
337+
338+
||| {{translation["cs"]}}'
339+
metadata: !TemplateMetadata
340+
choices_in_prompt: false
341+
metrics:
342+
- BLEU
343+
original_task: true
344+
name: xglm-en-cs-target
345+
reference: XGLM paper https://arxiv.org/abs/2112.10668

0 commit comments

Comments
 (0)