ORKG Paper Annotator evaluation

Introduction

Thank you for participating in this Open Research Knowledge Graph (ORKG) user study. In total this evaluation will take about 30 minutes. The evaluation consists of two parts, firstly you are asked to use a new ORKG paper entry system. Secondly, you should fill out the evaluation form where you describe your experience with the system.

The deadline for participation is October 7, 2020.

Goal

We are always looking for new ways to make adding a paper to ORKG easier. The purpose of this evaluation is to let you use a new tool, called the Paper Annotator tool. The goal of this tool is to annotate the most important sentences within a paper. Artificial Intelligence (AI) techniques are used to support you during this process. In the end, a Knowledge Graph is automatically created based on your annotations.

How to participate

The evaluation consists of two parts: 1) Annotate your paper, 2) Fill out the questionnaire.

1. Annotate your paper

In this step, you have to annotate a paper. Please follow the steps listed below.

Some tips while using the tool

• Some sentences will be automatically highlighted in grey color. These sentences might be of interest, but you can ignore them if they are not. Also make sure that (apart from the automatically highlighted sentences), you are expected to also select text yourself within the paper
• Don’t be afraid to make mistakes (basically everything you do is correct)! The tools tries to guide you whenever possible. But if there is something you are not sure about, just continue. You will not be graded in anyway 😊
• A completion bar is shown at the left top of the tool. This is only to guide you during the annotation process. It is not required to reach 100% before saving. You can save whenever you feel you are ready
• Try not to annotate more than 3 times the same type (e.g., do not make 4 annotations of background information). Only annotate the most important sentences

Participating in the experiment

1. Choose a paper where you are (co)author of, or choose a paper that you recently read
2. Visit the ORKG Paper Annotator
3. Upload your paper to the annotator (license information: the file you upload is only used during the annotation task will not be published or stored on our servers)
4. A help tour will appear, explaining the most important features of the system. Please read them carefully
5. Annotate your paper, save the results and come back to this page

2. Fill out the questionnaire

After saving your paper, you can fill out the online questionnaire. The questionnaire consists of 5 pages which all take about the same amount of time. Please click here to go to the questionnaire. Tip: read the questions carefully, for some questions a high rating is positive, while for others it is negative. We did this to keep you sharp 😉

After filling out the questionnaire, you are done with the evaluation. Thank you again for participating, we appreciate it a lot!

Interface

The interface looks like this:

License information

As already mentioned, the article you upload is only used during the annotation task will not be published or stored on our servers. The text annotations you make are publicly published under to the ORKG data license.