Skip to content

add prompts for conceptnet, squad, and google re subset - Lama Dataset #737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

KhalidAlt
Copy link
Contributor

No description provided.

@awebson awebson requested review from awebson and jzf2101 April 26, 2022 21:45
@jzf2101 jzf2101 self-assigned this Apr 26, 2022
@KhalidAlt KhalidAlt changed the base branch from main to eval-hackathon April 26, 2022 23:26
Copy link
Contributor

@awebson awebson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jess will review the other subsets and leave more comments soon, but some of the wording of your prompts are not very natural/grammatical to non-NLP experts. Please see our suggestions.

@jzf2101 jzf2101 changed the title add prompts for conceptnet, squad, and google re subset add prompts for conceptnet, squad, and google re subset - Lama Dataset Apr 27, 2022
@jzf2101 jzf2101 requested a review from awebson April 27, 2022 05:45
@KhalidAlt KhalidAlt requested review from jzf2101 and removed request for jzf2101 April 29, 2022 10:50
@jzf2101
Copy link
Collaborator

jzf2101 commented May 16, 2022

@KhalidAlt there's a build error could you try to resolve- we can't merge until the build error is fixed:

Given the following fact statement: {{masked_sentence}} What is the missing word to fill at [MASK]. ||| {{obj_label}}


	--------
	Split  train
	Example  {'uuid': 'd4f11631dde8a43beda613ec845ff7d1', 'sub': 'alive', 'obj': 'think', 'pred': 'HasSubevent', 'obj_label': 'think', 'masked_sentence': 'One of the things you do when you are alive is [MASK].', 'negated': ''}
	--------
Traceback (most recent call last):
  File "test/show_templates.py", line 65, in <module>
    if output[0].strip() == "" or (len(output) > 1 and output[1].strip() == ""):
AttributeError: 'list' object has no attribute 'strip'
Error: Process completed with exit code 1.

@jzf2101 jzf2101 requested review from VictorSanh and stephenbach May 16, 2022 16:55
@VictorSanh
Copy link
Member

@jzf2101 @KhalidAlt I have started looking into that. This is definitely weird. I can't reproduce that error when I run the tests locally (and I tried on two setups)... will keep you updated once I have debugged this.

@KhalidAlt there's a build error could you try to resolve- we can't merge until the build error is fixed:

Given the following fact statement: {{masked_sentence}} What is the missing word to fill at [MASK]. ||| {{obj_label}}


	--------
	Split  train
	Example  {'uuid': 'd4f11631dde8a43beda613ec845ff7d1', 'sub': 'alive', 'obj': 'think', 'pred': 'HasSubevent', 'obj_label': 'think', 'masked_sentence': 'One of the things you do when you are alive is [MASK].', 'negated': ''}
	--------
Traceback (most recent call last):
  File "test/show_templates.py", line 65, in <module>
    if output[0].strip() == "" or (len(output) > 1 and output[1].strip() == ""):
AttributeError: 'list' object has no attribute 'strip'
Error: Process completed with exit code 1.

@KhalidAlt
Copy link
Contributor Author

KhalidAlt commented May 17, 2022

Hi @VictorSanh, same for me. I tested (show_templates.py) locally, but it did not produce any error. I am not sure what is the reason for this issue. The same error shows up in other PRs : here and here. Please let me know if anything comes up. Thank you.

@jzf2101
Copy link
Collaborator

jzf2101 commented May 19, 2022

@KhalidAlt - per @stephenbach please merge eval-hackathon in your branch and push back

@jzf2101 jzf2101 requested review from awebson and removed request for awebson June 6, 2022 17:18
Copy link
Contributor

@awebson awebson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed some typos, but see the "consistent" prompt for the Squad subset for a critical bug.

Bring back latest changes into this branch
VictorSanh and others added 2 commits June 25, 2022 16:57
@VictorSanh VictorSanh self-assigned this Jun 25, 2022
Copy link
Member

@VictorSanh VictorSanh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integrated @jzf2101 and @awebson 's suggestions + merging back latest updates from eval-hackathon main branch to make sure test are up to date + fixing last details

Thank you @KhalidAlt ! merging now

@VictorSanh VictorSanh merged commit c7486b6 into bigscience-workshop:eval-hackathon Jun 25, 2022
@KhalidAlt KhalidAlt deleted the add_lama_dataset branch June 27, 2022 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants