Joint Parsing of Syntactic and Semantic Dependencies

The task proposes the merging of both syntactic dependencies (extracted from the Penn Treebank ) and semantic dependencies (extracted both from PropBank and NomBank) under a unique unified representation. The task has several novel objectives:

1. The first objective is to perform and evaluate Semantic Role Labeling (SRL) using a dependency-based representation for both syntactic and semantic dependencies in a quite novel way [1]. Furthermore, the SRL problem will address not only propositions centered around verbal predicates but also around nouns.
2. The syntactic dependencies to be modeled will be more complex than the ones used in the previous CoNLL evaluations: Johansson and Nugues have shown that a richer set of syntactic dependencies improves semantic processing [2].
3. The proposed evaluation offers a practical framework to perform joint learning for the two problems.

There are several issues that make this task attractive:

1. We believe that the proposed dependency-based representation is a better fit for many applications (e.g., Information Retrieval, Information Extraction) where it is often sufficient to identify the dependency between the predicate and the head of the argument constituent rather than extracting the complete argument constituent. Furthermore, it was shown that the extraction of dependencies can be performed with state-of-the-art performance in linear time [3], which can give a significant boost to the adoption of this technology in real-world applications.
2. The task will motivate several important research directions. For example, is the dependency-based representation better for SRL than the constituent-based formalism? Does joint learning improve syntactic and semantic analysis?

The participants are required to extract both types of dependencies. Information about the representation of input and output can be found in the data format page.

Similarly to the CoNLL-2005 shared task, this shared task evaluation is separated into two challenges:

• Closed Challenge. Systems have to be built strictly with information contained in the given training corpus, and tuned with the development section. In addition, the PropBank frames can also be used (see Official Resources). Note that this means that constituent-based parsers or SRL systems can not be used in this challenge because the constituent-based annotations are not provided in our training set. The aim of this challenge is to compare the performance of the participating systems in a fair environment.
• Open Challenge. Systems can be developed making use of any kind of external tools and resources. The only condition is that such tools or resources must have not been developed with the annotations of the test set, both for the input and output annotations of the data. In this challenge, we are interested in learning methods which make use of any tools or resources that might improve the performance. For example, we encourage the use of rich semantic information, by using WordNet, VerbNet or a WSD system. Also, in this challenge participants are encouraged to use constituent-based parsers and SRL systems, as long as these systems were trained only with the sections of TreeBank/PropBank/NomBank used in the shared task training corpus (the exact section numbers will be announced in due time). To encourage the participation of the groups that are only interested in SRL, the organizers will provide the output of a state-of-the-art dependency parser as input in this challenge. The comparison of different systems in this setting may not be fair, and thus ranking of systems is not necessarily important.

1. Hacioglu K. Semantic Role Labeling Using Dependency Trees. In Proceedings of COLING-2004, 2004.
2. Johansson R. and Nugues P. Extended Constituent-to-dependency Conversion for English. In Proceedings of NODALIDA 2007, 2007.
3. Nivre J., Hall J., Nilsson J. and Eryigit G. Labeled Pseudo-Projective Dependency Parsing with Support Vector Machines. In Proceedings of the CoNLL-X Shared Task, 2006.