Wikiann Prompts #772
Wikiann Prompts #772
Conversation
…orkshop#778) * accelerate `get_infos` by caching the `DataseInfoDict`s * quality * consistency
There was a problem hiding this comment.
Thanks Jess! NER is just hard to be adapted to a prompt form because the task itself is kind of a non-natural. Normal humans don't just carry around a vector for each word that can be fed into a classifier. Recall that we talked about this with @yongzx on Slack:
Unless we are talking about entity classification (where we already know what is the entity and we want to figure out its type). NER on the other hand is entity extraction + entity classification.
Your prompt gives the model the gold span as opposed to let models extract the span. So that's already non-original task. Further, several of your prompts ask for a binary classification, not multiple choice among location, person, organization. So those are also non-original task prompts. Right?
Lastly, are you sure the metrics are "AUC, Accuracy, COQA F1, and Other"?
| 41b109ce-7504-4263-b455-315fa7a7f3c2: !Template | ||
| answer_choices: Yes ||| No | ||
| id: 41b109ce-7504-4263-b455-315fa7a7f3c2 | ||
| jinja: 'Given, "{{'' ''.join(tokens)}}", is {{spans[0].split('':'')[1]}} a location? |
There was a problem hiding this comment.
Why is the '' ''.join(tokens) here joining using multiple spaces?
|
Also, the targets of your *_bool prompts should be, e.g., "False" as a single target sequence. Right now it's saying this example has "F", "a", "l", "s", "e" as five separate correct answers.
|
| 53bd1751-f0e6-4ca8-84d4-80c1c09a230d: !Template | ||
| answer_choices: True ||| False | ||
| id: 53bd1751-f0e6-4ca8-84d4-80c1c09a230d | ||
| jinja: 'Given the following information: |
There was a problem hiding this comment.
As opposed to "Given", or "Given the following information:", maybe it's more natural to say something like "In the sentence…"?
|
I am working on this right now, but I ran into some weird GH issue (been Googling the whole day to debug it) where there's wikiann en templates in the folder, but streamlit doesn't load it. Do you know how to resolve this?
|
No description provided.