English PTB to UD 2.0

[arademaker]

Does anyone know that is the best approach to convert a treebank in PTB format to UD 2.0? I found the page https://nlp.stanford.edu/software/stanford-dependencies.html, but it is not clear if the code supports UD 2.0. Suggestions are welcome.

[amir-zeldes]

You can use CoreNLP to convert PTB brackets for English to UD v1 (more or less, I think it represents a particular moment in time before 2.0 was released, but fairly close to v1 still), like this:

java -cp "*;" -Dfile.encoding=UTF-8 edu.stanford.nlp.trees.ud.UniversalDependenciesConverter -encoding UTF-8 -treeFile FILENAME

If you have a good conversion to Stanford Dependencies, you can also use DepEdit to convert the data to the current UD standard, more or less accurately depending on whether you have some additional entities (e.g. entities to resolve flat/compound better, etc.). This process is described and evaluated in this paper:

https://www.aclweb.org/anthology/W18-4918/

Finally you can also use a quick and dirty UD1>UD2 DepEdit script to transform the CoreNLP output from the command above to the current guidelines, but there are certain to be errors if you don't have the additional annotations from the paper. This basically just renames the labels that were changed in V2, rewires cc+conj, etc.:

pos=/VERB/;func=/nmod/	#1>#2	#2:func=obl
func=/.*/;func=/conj/;func=/cc/	#1>#2;#1>#3;#1.*#2	#2>#3
func=/dobj/	none	#1:func=obj
func=/mwe/	none	#1:func=fixed
func=/name|foreign/	none	#1:func=flat
func=/neg/	none	#1:func=advmod
func=/nsubjpass/	none	#1:func=nsubj:pass
func=/auxpass/	none	#1:func=aux:pass

If you want the code from the paper, let me know, but it is probably not 100% runnable out of the box (hardwired paths etc.)

[sebschu]

Since CoreNLP v4.0.0, the converter actually outputs UDv2!

You can run it, as suggested by Amir, using the command:

java -cp "*;" -Dfile.encoding=UTF-8 edu.stanford.nlp.trees.ud.UniversalDependenciesConverter -encoding UTF-8 -treeFile FILENAME

[arademaker]

Just to let people know... I got some errors when I run the UD validation script on the output data produced by CoreNLP 4.0 over the https://catalog.ldc.upenn.edu/LDC2013T19 dataset. the top 15 most frequent errors are:

41505  [L3 Syntax rel-upos-cop] 'cop' should be 'AUX' or 'PRON'/'DET' but it is 'VERB'
1780  [L3 Syntax rel-upos-advmod] 'advmod' should be 'ADV' but it is 'ADP'
 780  [L3 Syntax right-to-left-conj] Relation 'conj' must go left-to-right.
 568  [L3 Syntax rel-upos-aux] 'aux' should be 'AUX' but it is 'VERB'
 489  [L3 Syntax rel-upos-punct] 'punct' must be 'PUNCT' but it is 'SYM'
 320  [L3 Syntax rel-upos-nummod] 'nummod' should be 'NUM' but it is 'DET'
 304  [L3 Syntax rel-upos-nummod] 'nummod' should be 'NUM' but it is 'ADJ'
 234  [L3 Syntax rel-upos-advmod] 'advmod' should be 'ADV' but it is 'X'
 208  [L3 Syntax rel-upos-cc] 'cc' should not be 'DET'
 175  [L3 Syntax rel-upos-advmod] 'advmod' should be 'ADV' but it is 'NOUN'
 136  [L3 Syntax upos-rel-punct] 'PUNCT' must be 'punct' but it is 'conj'
  63  [L3 Syntax rel-upos-case] 'case' should not be 'ADJ'
  61  [L3 Syntax rel-upos-nummod] 'nummod' should be 'NUM' but it is 'ADV'
  48  [L3 Syntax rel-upos-mark] 'mark' should not be 'DET'
  46  [L3 Syntax right-to-left-appos] Relation 'appos' must go left-to-right.

[nschneid]

Any update on CoreNLP's PTB->UD conversion producing invalid UD? @sebschu @manning @AngledLuffa

[AngledLuffa]

that looks like a project! i will find time this year to start chipping away at that, but there's some work i simply can't put off any longer as i promised it for an upcoming industry event

[AngledLuffa]

actually, one way to speed this up would be to suggest a few command lines for doing the validation

[nschneid]

I think this should run validation for EWT:

$ cd UD_English-EWT
$ git clone https://github.com/UniversalDependencies/tools/
$ tools/validate.py --lang en en_ewt-ud-{dev,test,train}.conllu

[AngledLuffa]

Drilling down a bit into the most common error, that of a cop being AUX instead of VERB, here is a concrete example. In the EWT tree

( (S
    (NP-SBJ (DT The) (JJ actual) (NN vote))
    (VP (VBZ is)
      (ADJP-PRD
        (NP (DT a) (JJ little))
        (JJ confusing)))
    (. .)))

Our POS tag converter code has a comment:

https://github.com/stanfordnlp/CoreNLP/blob/main/src/edu/stanford/nlp/trees/UniversalPOSMapper.java
https://github.com/stanfordnlp/CoreNLP/blob/3499d27e615c35702f23948e886a7389b5695c33/data/edu/stanford/nlp/upos/ENUniversalPOS.tsurgeon#L45

% Don't do this, we are now treating these as copular constructions

and that part of the conversion being commented out results in the tag VERB instead of AUX

1       The     the     DET     DT      _       3       det     _       _
2       actual  actual  ADJ     JJ      _       3       amod    _       _
3       vote    vote    NOUN    NN      _       7       nsubj   _       _
4       is      be      VERB    VBZ     _       7       cop     _       _
5       a       a       DET     DT      _       6       det     _       _
6       little  little  ADJ     JJ      _       7       obl:npmod       _       _
7       confusing       confusing       ADJ     JJ      _       0       root    _       _
8       .       .       PUNCT   .       _       7       punct   _       _

whereas the UD version of that sentence is

# sent_id = weblog-blogspot.com_aggressivevoicedaily_20060629164800_ENG_20060629_164800-0002
# text = The actual vote is a little confusing.
1       The     the     DET     DT      Definite=Def|PronType=Art       3       det     3:det   _
2       actual  actual  ADJ     JJ      Degree=Pos      3       amod    3:amod  _
3       vote    vote    NOUN    NN      Number=Sing     7       nsubj   7:nsubj _
4       is      be      AUX     VBZ     Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin   7       cop     7:cop   _
5       a       a       DET     DT      Definite=Ind|PronType=Art       6       det     6:det   _
6       little  little  ADJ     JJ      Degree=Pos      7       obl:npmod       7:obl:npmod     _
7       confusing       confusing       ADJ     JJ      Degree=Pos      0       root    0:root  SpaceAfter=No
8       .       .       PUNCT   .       _       7       punct   7:punct _

First there's a somewhat unfortunate DRY violation here, in that the same rules are repeated in the tsurgeon file and in the constituency -> dependency converter rules:

https://github.com/stanfordnlp/CoreNLP/blob/main/src/edu/stanford/nlp/trees/UniversalEnglishGrammaticalRelations.java

So I'll need to figure out how extensive that problem is and how best to resolve it. There have been a few dependency converter fixes over the years which I assume are not reflected in any way in the POS converter. I also need to figure out how or why this particular rule about cop is being ignored and what to do to fix it.

The other errors probably have similar origins when it comes to UPOS tags being flagged by the validator. They'll each require some individual attention regarding what kind of tree is causing the error and how to fix.