Vol. 15 No. 1 (2024)
Articles

Quality Assessment of Research Comparisons in the Open Research Knowledge Graph: A Case Study

English PDF
  • Jennifer D'Souza,
  • Hassan Hussein,
  • Julia Evans,
  • Lars Vogt,
  • Oliver Karras,
  • Vinodh Ilangovan,
  • Anna-Lena Lorenz,
  • Sören Auer
Jennifer D'Souza
TIB Leibniz Information Centre for Science and Technology
Hassan Hussein
TIB Leibniz Information Centre for Science and Technology
Julia Evans
TIB Leibniz Information Centre for Science and Technology
Lars Vogt
TIB Leibniz Information Centre for Science and Technology
Oliver Karras
TIB Leibniz Information Centre for Science and Technology
Vinodh Ilangovan
TIB Leibniz Information Centre for Science and Technology
Anna-Lena Lorenz
TIB Leibniz Information Centre for Science and Technology
Sören Auer
TIB Leibniz Information Centre for Science and Technology
DOI: https://doi.org/10.36253/jlis.it-547
Categories

Published 2024-01-15

Keywords

  • Knowledge Graph,
  • Open Research Knowledge Graph,
  • Linked Open Data (LOD),
  • Human-Computer Interaction,
  • Survey

How to Cite

D’Souza, Jennifer, Hassan Hussein, Julia Evans, Lars Vogt, Oliver Karras, Vinodh Ilangovan, Anna-Lena Lorenz, and Sören Auer. 2024. “Quality Assessment of Research Comparisons in the Open Research Knowledge Graph: A Case Study”. JLIS.It 15 (1):126-43. https://doi.org/10.36253/jlis.it-547.

Copyright (c) 2023 Jennifer D'Souza, Hassan Hussein, Julia Evans, Lars Vogt, Oliver Karras, Vinodh Ilangovan, Anna-Lena Lorenz, Sören Auer

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Crossref
Scopus
Google Scholar
Europe PMC

Funding data

Abstract

The Open Research Knowledge Graph (ORKG) is a digital library for machine-actionable scholarly knowledge, with a focus on structured research comparisons obtained through expert crowdsourcing. While the ORKG has attracted a community of more than 1,000 users, the curated data has not been subject to an in-depth quality assessment so far. Here, proposed as a first exemplary step, within a team of domain experts, we evaluate the quality of six selected ORKG Comparisons based on three criteria, namely: 1) the quality of semantic modelling, 2) the maturity of the Comparisons in terms of their completeness, syntactic representation, identifier stability, and their linkability mechanisms ensuring the interoperability and discoverability. Finally, 3) the informative usefulness of the Comparisons to expert and lay users. We have found that each criterion addresses a unique and independent aspect of quality. Backed by the observations of our quality evaluations presented in this paper, a fitting model of knowledge graph quality appears one that is indeed multidimensional as ours.

English PDF

Metrics

Metrics Loading ...

References

  1. Auer, Sören, Allard Oelen, Muhammad Haris, Markus Stocker, Jennifer D’Souza, Kheir Eddine Farfar, Lars Vogt, Manuel Prinz, Vitalis Wiens, and Mohamad Yaser Jaradeh. 2020. “Improving Access to Scientific Literature with Knowledge Graphs.” Bibliothek Forschung Und Praxis 44 (3): 516–29. https://doi.org/10.1515/bfp-2020-2042. DOI: https://doi.org/10.1515/bfp-2020-2042
  2. Berners-Lee, Tim, James Hendler, and Ora Lassila. 2001. “The Semantic Web.” Scientific American 284 (5): 34–43. https://www.lassila.org/publications/2001/SciAm.pdf. DOI: https://doi.org/10.1038/scientificamerican0501-34
  3. Bizer, Christian, and Richard Cyganiak. 2009. “Quality-Driven Information Filtering Using the WIQA Policy Framework.” Journal of Web Semantics 7 (1): 1–10. https://doi.org/10.1016/j.websem.2008.02.005. DOI: https://doi.org/10.1016/j.websem.2008.02.005
  4. Bornmann, Lutz, and Rüdiger Mutz. 2015. “Growth Rates of Modern Science: A Bibliometric Analysis Based on the Number of Publications and Cited References: Growth Rates of Modern Science: A Bibliometric Analysis Based on the Number of Publications and Cited References.” Journal of the Association for Information Science and Technology 66 (11): 2215–22. https://doi.org/10.1002/asi.23329. DOI: https://doi.org/10.1002/asi.23329
  5. Daniel, Florian, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. “Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions.” ACM Computing Surveys 51 (1). https://doi.org/10.1145/3148148. DOI: https://doi.org/10.1145/3148148
  6. Färber, Michael, Basil Ell, Carsten Menne, Achim Rettinger, and Frederic Bartscherer. 2018. “Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO.” Semantic Web Journal 9 (2): 77–129. Accessed March 15, 2023. https://doi.org/10.3233/SW-170275. DOI: https://doi.org/10.3233/SW-170275
  7. Hussein, Hassan, Allard Oelen, Oliver Karras, and Sören Auer. 2022. “KGMM - A Maturity Model for Scholarly Knowledge Graphs Based on Intertwined Human-Machine Collaboration.” In From Born-Physical to Born-Virtual: Augmenting Intelligence in Digital Libraries: 24th International Conference on Asian Digital Libraries, ICADL 2022, 253–69. Hanoi, Vietnam: Springer-Verlag. https://doi.org/10.1007/978-3-031-21756-2_21. DOI: https://doi.org/10.1007/978-3-031-21756-2_21
  8. Johnson, Rob, Anthony Watkinson, and Michael Mabe. 2018. “The STM Report: An Overview of Scientific and Scholarly Publishing. 5th Edition.” https://www.stm-assoc.org/2018_10_04_STM_Report_2018.pdf.
  9. Oelen, Allard, Mohamad Yaser Jaradeh, Kheir Eddine Farfar, Markus Stocker, and Sören Auer. 2019. “Comparing Research Contributions in a Scholarly Knowledge Graph.” In SciKnow 2019: Third International Workshop on Capturing Scientific Knowledge, 21-26. https://doi.org/10.15488/9388.
  10. Oelen, Allard, Mohamad Yaser Jaradeh, Markus Stocker, and Sören Auer. 2020. “Generate FAIR Literature Surveys with Scholarly Knowledge Graphs.” In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, 97–106. https://doi.org/10.34657/5212. DOI: https://doi.org/10.1145/3383583.3398520
  11. Pipino, Leo L., Yang W. Lee, and Richard Y. Wang. 2002. “Data Quality Assessment.” Communications of the ACM 45 (4): 211–18. https://doi.org/10.1145/505248.506010. DOI: https://doi.org/10.1145/505248.506010
  12. Shotton, David. 2009. “Semantic Publishing: The Coming Revolution in Scientific Journal Publishing.” Learned Publishing 22 (2): 85–94. https://doi.org/10.1087/2009202. DOI: https://doi.org/10.1087/2009202
  13. Wang, Richard Y., and Diane M. Strong. 1996. “Beyond Accuracy: What Data Quality Means to Data Consumers.” Journal of Management Information Systems 12 (4): 5–33. https://doi.org/10.1080/07421222.1996.11518099. DOI: https://doi.org/10.1080/07421222.1996.11518099
  14. Wilkinson, Mark D, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J.G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A.C ’t Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Barend Mons. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3 (1): 1–9. https://doi.org/10.1038/sdata.2016.18. DOI: https://doi.org/10.1038/sdata.2016.18
  15. Zaveri, Amrapali, Anisa Rula, Andrea Maurino, Ricardo Pietrobon, Jens Lehmann, and Sören Auer. 2016. “Quality Assessment for Linked Data: A Survey.” Semantic Web 7 (1): 63–93. https://doi.org/10.3233/SW-150175. DOI: https://doi.org/10.3233/SW-150175
  16. Zhang, Jing. 2022. “Knowledge Learning With Crowdsourcing: A Brief Review and Systematic Perspective.” IEEE/CAA Journal of Automatica Sinica 9 (5): 749–62. https://doi.org/10.1109/jas.2022.105434. DOI: https://doi.org/10.1109/JAS.2022.105434