Making the knowledge contained in scientific papers machine-readable and formally computable would allow researchers to take full advantage of this information by enabling integration with other knowledge sources to support data analysis and interpretation. Here we describe Biofactoid, a web-based platform that allows scientists to specify networks of interactions between genes, their products, and chemical compounds, and then translates this information into a representation suitable for computational analysis, search and discovery. We also report the results of a pilot study to encourage the wide adoption of Biofactoid by the scientific community.
Biological pathways organize sets of molecular interactions and reactions that underlie cellular processes. This information can be used for experimental design, interpretation of genomics data (
To manage, visualize and interpret the large amount of available pathway information, researchers require computational tools such as pathway information resources, structured data representation standards and analysis software (
For reasons of efficiency and accuracy, structured knowledge of biomedical discoveries could be provided directly by the original authors of research reports, rather than post-hoc, through the curation efforts of knowledge base teams. By analogy, molecular 3D structures submitted directly by authors to the Protein DataBank (PDB) have become a key resource among structural biologists (
Here we introduce Biofactoid (
Authors can use Biofactoid to share structured information from research articles both as part of the publication process and outside of it. Personal email invitations sent to authors as part of a pilot outreach effort resulted in them contributing over 95 articles to Biofactoid from a range of journals; collectively these pathways have described a total of 426 entities (340 unique) from eight organisms (
Biofactoid enables molecular-level detail of biological processes reported in articles to be shared in a structured format accessible to humans and computers. Interactions (binding, post-translational modification and transcription/translation) involving molecules of various types (proteins, nucleic acids, genes, or chemicals such as metabolites and drug compounds) can be represented. Users begin by entering the article title, or identifier (e.g. PubMed identifier or Digital Object Identifier). Next, article metadata, including authors, abstract and journal issue, are automatically retrieved. Next, users draw a network of biological entities and interactions using the Biofactoid curation tool, which has an easy-to-use interface similar to graphical illustration software (e.g. Microsoft Powerpoint, Adobe Illustrator) which will be familiar to users, but with the added ability to generate structured data from the author-drawn biological pathway. The major features of the Biofactoid curation tool include support for:
Molecular entities. Genes, gene products and chemicals are created in the network using an “Add a gene or chemical” tool, which creates a node (circle) that users label (
Interactions. A “Draw interaction” tool lets users link two nodes by clicking one and dragging to the other (
Complexes. Molecular complexes can be created by dragging genes or chemicals close to each other, resulting in a box that encloses them, which can also be labeled (
Context. A context input box that encourages authors to “List terms that describe the context (e.g. T cell, cancer, genome stability)”. Allowing authors to provide a list in free text form has the benefit of accommodating any terms or concepts they consider important for specifying the research context.
Automatic saving and co-editing. Pathway representations in Biofactoid are automatically saved as changes are made and a live-sync capability enables multiple authors to collaboratively edit the same pathway, analogous to Google Docs.
Curation in Biofactoid involves drawing a network of relationships between genes or chemicals. (
Once complete, the pathway data is validated by pressing a "Submit" button. At this point, the user may address any potential quality issues flagged by the system (e.g. unlabelled nodes, empty document) before confirming their submission. Submitted pathways are shared publicly with the research community and users retain the ability to edit their pathway via the link they received at the start of the workflow. Authors who wish to have their submissions removed from Biofactoid can do so by contacting the support team (
When research findings are shared through Biofactoid in a structured and computable manner, curated knowledge is connected to, and becomes part of, a collective pool of computable knowledge that the community can access in different ways. Visitors to the Biofactoid website (
Users can select any entity in the pathway diagram to see more information about genes (via NCBI Gene database links) and chemicals (via ChEBI database links). To support attribution, each author of an article is automatically linked to their profile in the Open Researcher and Contributor ID database (ORCID;
To generate these recommendations, we first retrieve an article’s references, articles it is cited by and “Similar articles” from PubMed using the NCBI Entrez Programming Utilities service (
(
Beyond the “Explorer”, Biofactoid data represented in the formal BioPAX ontology is interoperable with external pathway and interaction knowledge using technologies for BioPAX processing and analysis, including Paxtools (
For example, researchers can ask the question: “Which known pathways are related to an interaction in Biofactoid?” As an example answer to this question, a search of Pathway Commons shows how two interactions made computable by authors using Biofactoid (i.e. “
We tested the Biofactoid software over multiple iterations with researchers from the authors’ institutions, journal editors and finally pathway authors, updating the software each time based on user feedback. This process supported two major goals: to improve the user experience of the software for authors, who are typically unfamiliar with curation and structured data concepts, and to develop a model for integrating Biofactoid into the publication process with journals. We performed user testing, with a user-centred design process, of a prototype of the Biofactoid curation tool (
Once Biofactoid software achieved a sufficient level of sophistication and completeness, satisfying many of these initial users, we engaged journal editors to determine the feasibility of using Biofactoid to capture information from authors. Our pilot study consisted of three phases (
Roughly 8.5% of these authors successfully shared their research in Biofactoid, proving that many authors can and will use Biofactoid (
A three-phase pilot tested the feasibility of Biofactoid and involved journal editors and authors of research articles. Phase I and II involved editors and authors whose articles were recently published. In Phase III, 2,065 published articles were screened and authors of suitable articles were invited to Biofactoid. The articles screened were from 16 journals (
ISSN | Journal | Coverage² [Vol. (Issue)] | Articles screened | Hits | % Hits |
---|---|---|---|---|---|
2211–1247 | Cell Reports | 30(1) - 32(11) | 953 | 109 | 10.3 |
1097–4164 | Molecular Cell | 73(1) - 79(6) | 725 | 85 | 10.5 |
1549–5477 | Genes & Development | 34(1-2) - 34(17-18) | 93 | 15 | 13.9 |
1476–4679 | Nature Cell Biology | 22(4) - 22(9) | 84 | 10 | 10.6 |
1083–351 X | Journal of Biological Chemistry | 295(31) - 295(37) | 210 | 21 | 9.1 |
- | - | 2065 | 240 | - | |
- | - | - | - | 10.4 |
Only journals in which at least 80 ‘hits’ were identified were included. A ‘hit’ is an article that provides direct evidence for a molecular interaction that can be captured by Biofactoid. The ability of Biofactoid to capture an interaction depends upon the type of bioentities, the relationship types and organisms described in the article. In total, articles from 16 journals were screened: EMBO; Molecular and Cellular Biology; Cell; Cancer Cell; iScience; J Biol Chem; Cell Metabolism; Science; Nature Genetics; Science Signaling; Science Advances; Immunity; Cell Reports; Molecular Cell; Genes & Development; Nature Cell Biology. ²Coverage indicates the span of journal issues that were included. Only primary research articles from each issue were screened.
Biofactoid focuses on the capture of published pathway information in computational form to augment discovery, attribution and communication of scientific knowledge. In developing our strategy, we have considered how technology can best be used to aid authors. The result is a generic approach that can be extended to capture other types of biological knowledge, at the source of knowledge creation, in computable form. To be successful, Biofactoid development has focused on providing key elements of efficiency, incentives and better technology for human-computer interaction.
The computable knowledge capture model proposed here includes formal knowledge representation using ontologies, easy to use curation support software that links molecules to corresponding database identifiers (normalization), submission to widely available knowledge resources, a connection with the publication process and author attribution. This model expands on prior work defining digital abstracts and building crowdsourcing efforts for pathway data and software (
Comparison of projects that support community curation of pathway and interaction knowledge as their primary concern.
Project | Scope | Source | Integrated curation tool | Automatic entity recognition | Single-article oriented | Ref. |
---|---|---|---|---|---|---|
Biofactoid | Pathway | Author | ✓ | ✓ | ✓ | This study. |
Structured Digital Abstract (FEBS Letters) | Protein-Protein | Author | - | - | ✓ | |
WikiPathways | Pathway | Anyone | - | - | - | |
SourceData | Figure | Author | ✓ | - | ✓ |
A key assumption underlying Biofactoid is that scalable knowledge capture can be achieved by community curation. However, crowd-sourcing of biological research findings presents unique challenges that must be considered if widespread use is to be realised. First, the biological research literature contains a wide array of biological entities and concepts, with descriptions that range from detailed to abstract. As support for more biological entity types are added, a curation system can become more complex, increasing the barrier to entry and making it more likely users will require training to use it effectively. Thus, a successful strategy should support wide coverage of biological concepts while minimising complexity.
In a related issue, biological researchers are accustomed to communicating pathway concepts using visual depictions that employ symbols and conventions, which, although comprehensible to the community, are often ad hoc
A second major challenge to achieving uptake is raising awareness and communicating the goals of the project within the research community. Pathway information is qualitatively distinct from items that researchers are accustomed to depositing (e.g. DNA sequencing data), as pathways are models that represent the outcome or interpretation of multiple experiments. Ideally, potential contributors should be able to independently determine whether their articles are suitable for Biofactoid.
A third challenge to realising wide uptake is incentivizing authors to contribute the knowledge in their article. In particular, authors will be more likely to contribute if the information they enter will be useful, either directly to themselves and/or others in their community. Scalable and sustainable growth is more likely to be achieved when researchers directly use and benefit from Biofactoid information and then decide to contribute themselves.
We designed Biofactoid to enable researchers to express details of their research quickly and effectively, with minimal instruction. To preserve ease of use, we have purposely limited both the scope of biological concepts and the granularity of detail that can be captured, though this can be expanded in the future. With regard to entities, only major model organisms and SARS-CoV(–2) are supported; we have not included more abstract concepts like cellular processes (e.g. apoptosis) or fine details like reaction stoichiometry, molecular sub-regions (e.g. promoter region) and modification (i.e. amino acid position, mutations). With regard to interactions, the system does not explicitly support biological concepts that involve spatial (e.g. cellular location) or temporal considerations (e.g. ordered recruitment of proteins).
The aforementioned details are supported by the underlying BioPAX ontology and additional concepts (e.g. global biological context, that is, the environment in which pathway components act) can be supported by referencing other controlled vocabularies or databases from the BioPAX model. While the current set of features constrains the scope of capturable research facts, it enables Biofactoid to be accessible by a wide audience. Nevertheless, future work will expand the range of biological concepts supported.
Distributing curation effort among authors has the potential to be efficient but also accurate because of their first-hand knowledge of the research. To ensure authors can use Biofactoid effectively we developed an automatic entity recognition tool that accurately matches the names of genes and chemicals provided by authors to an entity database record. We also flag potential issues with author input at submission so they can be corrected immediately (e.g. unlabelled entities, genes from multiple organisms). Finally, we evaluated our system through user testing to improve the speed and accuracy with which users can curate interactions described in text. In the future, we will develop strategies to monitor the accuracy of pathway data post hoc
To monitor data accuracy, we will seek out experts to independently rate pathway items, which is an approach that has been successfully applied to problems in computer-aided diagnostics and healthcare (
A convenient time to capture knowledge is during publication, when authors - the primary source of knowledge - are most aware of the details of their report and when they are typically asked to deposit other types of data (e.g. DNA sequencing or gene expression data). As with any innovative approach, introducing the system to editors, journals and their respective authors requires an investment of effort for all parties. It will likely be necessary to directly engage individual journals, to familiarize them with the system, and accommodate new requirements they may have. There are many ways to offer Biofactoid to authors, including, for instance: integrating it with the peer-review process, requiring it as a condition of publication or recommending it to authors post-publication-acceptance.
As a demonstration, our pilot study engaged publishers to establish requirements for using Biofactoid within their publication pipeline, ideally as a condition for acceptance, and to drive greater awareness of the system among the wider readership. The pilot study also revealed that directly inviting authors of suitable articles to contribute, even after publication, is a viable engagement strategy. Ideally, we would invite all authors who publish suitable content to use Biofactoid, however, the rapid pace of publication makes it impractical for the Biofactoid team to manually screen articles, arguing for the development of systems (e.g. using NLP) to automate article triage. If successful, this approach could be used to regularly identify relevant articles from the pool of all new entries indexed by PubMed.
Another approach to reach users is to notify those whose research articles are referenced by information curated in Biofactoid. For this purpose, we have developed a system that automatically notifies authors when their research is linked to papers curated in Biofactoid (e.g. by citations or because of related content) so that they may further explore this information and curate their own pathway knowledge. More generally, we intend to engage the research community by cultivating partnerships with knowledge database organizations including those that curate information about articles (PubMed), biomolecules (e.g. UniProt, NCBI, ChEBI), pathways and interactions (e.g. Reactome, STRING), model organisms (e.g.
To improve the efficiency and utility of Biofactoid, we are developing NLP technology to support authors in using Biofactoid, as well as to enable the representation of pathways in textual form (
We are also improving the search and exploration functions of the system, ensuring Biofactoid information is connected to other useful knowledge, and that related information is easily accessible starting from a Biofactoid entry. In the future, we envision that information entered by an author in Biofactoid will serve as a custom query that can be used to regularly notify the author of new information (e.g. from other publications) related to their interests, such as interacting molecules and phenotypes. This work provides a basis for the development of new technologies to make scientific knowledge more computable and accessible and help researchers identify information within the rapidly growing scientific corpus.
Biofactoid is written in JavaScript. The backend server uses a microservice architecture, with Node.js, Express and RethinkDB. Client-server data synchronization, supporting automatic saving and concurrent editing, uses websockets and a model similar to differential synchronization (
The network visualizations which summarize a search of known pathways and interactions related to interactions captured by Biofactoid (
Two rounds of user-centered testing were performed, once following the development of the initial prototype of the Biofactoid curation tool, and once again after further development of the tool. Test participants were drawn from volunteers who responded to individual email invitations. In each round of testing, participants were provided with a one-page handout that provided an overview of the project, brief instructions on how to use the curation tool, followed by one or more text excerpts describing a biological interaction that they were asked to input. Ten researchers participated, including faculty members (n = 2), graduate students (n = 3) and research staff (n = 5). Participants used the software on an individual, in-person basis from a provided computer, during which screen and voice were recorded. All participants provided written consent to volunteer and have their session recorded. We also elicited user feedback both in-person and by email for a version of Biofactoid able to support the entire "Data sharing workflow", including: user onboarding material presented on the
A group of two editors from Molecular Cell and Cell Systems volunteered to evaluate Biofactoid in response to a direct invitation. A pilot study was designed with three phases. Phase I introduced the software to the journal editors to determine whether it met basic requirements for capturing pathway knowledge in articles in their journals and identify any issues precluding interaction with authors. To do this, we simulated the author’s experience using the Biofactoid curation tool to input pathway information. First, an editor and a member of the Biofactoid team selected a recently published article from the Molecular Cell, "Online Now" (
Phase II validated the Biofactoid software and the workflow with authors. First, a set of suitable articles that were recently published by Molecular Cell (Online Now) were selected with journal editors. Second, email invitations addressed to the corresponding author of each article were sent from the institutional email account of a Biofactoid team member. As in Phase I, emails contained a link to the curation tool with their article information already filled in. To our knowledge, these authors had no prior contact with the journal with regard to Biofactoid. Follow-up reminders were sent after one week to authors who had not used the system. We repeated this process with two additional groups of articles (ten total) from Molecular Cell (Online Now) that were selected by the Biofactoid team.
Phase III examined the author participation rate following an invitation. First, we selected a total of sixteen journals, with a preference towards those with a mandate for molecular biology and mechanistic studies backed by biochemical-level evidence. Journals with broad and diverse readership (e.g. Nature, Science;
Biofactoid is available to biomedical researchers for data sharing and exploration at
No competing interests declared
Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Visualization, Writing – original draft, Writing – review and editing
Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Writing – review and editing
Data curation, Software
Software
Software
Conceptualization, Methodology, Project administration, Writing – review and editing
Conceptualization, Investigation, Methodology, Writing – review and editing
Software
Software
Conceptualization, Methodology, Supervision
Resources
Resources
Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review and editing
Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review and editing
Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review and editing
Human subjects: Participants of user testing provided written consent to volunteer, have their testing sessions recorded and have quotes obtained in the session published.
All Biofactoid data are available under the Creative Commons CC0 public domain license. To download the data and code, please refer to the documentation on the Biofactoid GitHub repository (
We thank Quincey Justman, Miao-Chih Tsai, and Anita DeWaard for feedback on making Biofactoid useful for editors and authors; the Reactome database team for support and feedback on the curation workflow; Alfonso Valencia, and Miguel Vazquez for early support; and the many community members in Toronto, Boston, Portland, and beyond for feedback on the design and concept of Biofactoid.
In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.
Thank you for submitting your article "Capturing scientific knowledge in computable form" to
The reviewers and editors have discussed the reviews and we have drafted this decision letter to help you prepare a revised submission.
Summary:
The manuscript of Wong et al., introduces Biofactoid, a novel and intuitive tool to collect interaction data from publications. The manuscript text is clear and provides both a general background for non-specialists as well as it contains the key technical details for experts. The figures nicely illustrate Biofactoid and its application. The authors not only developed the tool but also carried out multiple tests and pilots with users, editors and invited authors. All these phases are properly documented and presented in the manuscript. However, a number of points need to be addressed to make the article suitable for publication.
Essential revisions:
1. Please revise the title so that it more precisely describes the current capabilities of Biofactoid and the use cases demonstrated.
2. The amount of information collected for an interaction (that is, after the drawing just the type of molecular event) is not enough to really utilize the power of this approach in other projects. In particular not having an option to add some kind of biological context and detection methods is a missed opportunity. Please consider introducing an optional box for the biological context, in particular to name the tissue/cell line or the condition (healthy, cancer, etc.). This could come very useful when there are conflicting entries (A activates B; A inhibits B). With the biological context it could be clear that both are correct, otherwise it would look like the same interactors but different signs for the interaction and no further information. As for the type of detection, integrating the Molecular Interactions Controlled Vocabulary (https://www.ebi.ac.uk/ols/ontologies/mi) and facilitating data entry with autocomplete functions would provide useful extra information without overly increasing the time it takes to make a submission. Please consider updating Biofactoid to include this functionality. Please also clarify what ontology (if any) is used for relationships in Biofactoid.
3. Please highlight the fact that submissions to Biofactoid can be edited by users later on using the link sent in the original email. This may be useful if users make a mistake during the submission process or if they need to update the diagram during the publication process.
4. Please address whether there is a process for removing a submission altogether (e.g. if a submission is found to be incorrect or if a paper is being retracted).
5. The two sub-figures of Figure 2B are not easily reproducible. With https://biofactoid.org/ as an entry point, it was not possible to get any networks spanning more than the small ones extracted from a single article as shown in Figure 2A. With pathwaycommons.org as an entry point, it was not possible to find the interactions given in the figure. Concretely, on https://www.pathwaycommons.org/, trying "senp1 sirt3" and clicking on "Interactions", the result was "No interactions to display". Please provide more detailed instructions so the figures shown can be reproduced, or modify Figure 2B to reflect a network that can be obtained using Biofactoid.
6. Please include a discussion of how the facts stated in the paper, regarding the use of Biofactoid, could be qualitatively or quantitatively assessed. Currently, an author working with Biofactoid at all is taken as a successful end point, but surely it will be important to verify that the tool works as described and is, therefore, useful to the community. An assessment could be done, for example, in comparison to professional curators' work, as is often done for text mining approaches.
7. Please include the overall number of recorded facts generated over the multi-step process, including the number of papers used and the facts per paper.
8. Please temper your statements regarding the search and exploration functions of the system. Currently biofactoid.org seems to have no query interface. Since the data is not yet in PathwayCommons, interactive access is effectively only by browsing the papers shown in the landing page, but the figures and text imply much more functionality than is actually available to the user at this point.
9. Please include a comparison to related approaches, considering the citations referenced in line 235 of the manuscript.
10. Please state under what license contributors to Biofactoid are contributing their data, and make it clear both in the article and also in the website, so that users know how they can (re)use the data.
11. Please provide each of the Figures in your manuscript as a separate figure file.
Essential revisions:
1. Please revise the title so that it more precisely describes the current capabilities of Biofactoid and the use cases demonstrated.
Our title has been amended to emphasize the role of authors in pathway curation:
“Author-sourced capture of pathway knowledge in computable form using Biofactoid”
2. The amount of information collected for an interaction (that is, after the drawing just the type of molecular event) is not enough to really utilize the power of this approach in other projects. In particular not having an option to add some kind of biological context and detection methods is a missed opportunity. Please consider introducing an optional box for the biological context, in particular to name the tissue/cell line or the condition (healthy, cancer, etc.). This could come very useful when there are conflicting entries (A activates B; A inhibits B). With the biological context it could be clear that both are correct, otherwise it would look like the same interactors but different signs for the interaction and no further information.
We have updated the Biofactoid curation tool to include a context input box that encourages authors to “
As for the type of detection, integrating the Molecular Interactions Controlled Vocabulary (https://www.ebi.ac.uk/ols/ontologies/mi) and facilitating data entry with autocomplete functions would provide useful extra information without overly increasing the time it takes to make a submission. Please consider updating Biofactoid to include this functionality.
Biofactoid aims to capture biological pathway knowledge supported by evidence described in a research article. In this case, authors are free to describe an interaction that represents the findings of one experiment or a set of different experiments. Because entering this information using controlled vocabularies like MI could be complex, even for a trained database curator, we prefer to let authors work with easy to enter plain text data entry in this case (e.g. in the context field). In the future, we will consider ways to help authors annotate the experimental evidence supporting their interactions, for instance by listing the relevant MI terms or by using natural language processing, while still maintaining an easy to use curation interface.
Please also clarify what ontology (if any) is used for relationships in Biofactoid.
Relationships in Biofactoid are mapped to the standard BioPAX ontology that our team developed and that is in use by most major pathway database groups. BioPAX has controlled vocabulary terms for major interaction types (e.g. molecular interaction, catalysis), and extends this to use more specific types by reusing the standard molecular interaction (MI) controlled vocabulary “interaction type” subtree (e.g. “phosphorylation reaction”). We use these in Biofactoid.
3. Please highlight the fact that submissions to Biofactoid can be edited by users later on using the link sent in the original email. This may be useful if users make a mistake during the submission process or if they need to update the diagram during the publication process.
The concluding paragraph of the “Data sharing workflow” section has been updated accordingly:
“Submitted pathways are shared publicly with the research community and users retain the ability to edit their pathway via the link they received at the start of the workflow.”
4. Please address whether there is a process for removing a submission altogether (e.g. if a submission is found to be incorrect or if a paper is being retracted).
The final sentence of the “Data sharing workflow” section has been updated accordingly:
“Authors who wish to have their submissions removed from Biofactoid can do so by contacting the support team (support@biofactoid.org).”
Also, the Biofactoid Explorer web app has been updated so that if PubMed identifies an article as retracted, then users will see a “Retracted Publication” message for that article’s record in Biofactoid.
5. The two sub-figures of Figure 2B are not easily reproducible. With https://biofactoid.org/ as an entry point, it was not possible to get any networks spanning more than the small ones extracted from a single article as shown in Figure 2A. With pathwaycommons.org as an entry point, it was not possible to find the interactions given in the figure. Concretely, on https://www.pathwaycommons.org/, trying "senp1 sirt3" and clicking on "Interactions", the result was "No interactions to display". Please provide more detailed instructions so the figures shown can be reproduced, or modify Figure 2B to reflect a network that can be obtained using Biofactoid.
The results in Figure 2B are intended to demonstrate the utility of information added to Biofactoid. In particular, we show that interactions contributed by two different authors could be used to bridge otherwise disjoint mitochondrial pathways in Pathway Commons.
To create Figure 2B, we manually searched public data in Pathway Commons and Biofactoid with genes of interest and loaded them into the free Cytoscape software for network visualization (cytoscape.org). As a specific example, we executed the following steps to generate the first network in Figure 2B:
1. Visit pathwaycommons.org and search for genes that are part of interactions in Figure 2A: SENP1, SIRT3, UCP1, SIRT5
2. In the search results, listed under “Pathways”, select the Reactome pathways entitled “SUMO is proteolytically processed”, “Transcriptional activation of mitochondrial biogenesis” and “The fatty acid cycling model”. Download a Simple Interaction Format (SIF) file for each. This downloads all of the Reactome pathway information for these genes.
3. Also in the search results, select “Interactions” and download it in SIF. This downloads interactions from many databases involving these genes.
4. Using Cytoscape, import each SIF file as a new network then merge the individual networks using Cytoscape’s merge tool. Manually add the two Biofactoid interactions in Figure 2A to the merged network within Cytoscape.
The description of how these figures were created has been added to the section "Visualization of network data across sources" in Materials and methods.
In the future, we would like to develop additional features for Biofactoid or Cytoscape such that the steps used to create the networks in Figure 2 could be more easily reproduced using automated tools. Specifically, we are working to integrate Biofactoid data continually in Pathway Commons to make querying across all publicly available pathway data easier.
6. Please include a discussion of how the facts stated in the paper, regarding the use of Biofactoid, could be qualitatively or quantitatively assessed. Currently, an author working with Biofactoid at all is taken as a successful end point, but surely it will be important to verify that the tool works as described and is, therefore, useful to the community. An assessment could be done, for example, in comparison to professional curators' work, as is often done for text mining approaches.
In the Discussion (paragraph 6) we have described how Biofactoid currently supports authors in high-quality self-curation, followed by a proposal for efficiently monitoring and improving data quality. Briefly, we will aim to:
Monitor data accuracy using trained curators:
– Recruit at least two curators experienced in collecting pathway knowledge (e.g. Reactome curators)
– Curators independently rate a sample of records with regards to accuracy of entity grounding, interactions, as well as record completeness
– The curator’s ratings are compared to each other to estimate variability
Support community-based data accuracy monitoring:
– Encourage researchers with questions or concerns about a pathway in Biofactoid to communicate directly with the author(s) of the article to resolve issues and, if necessary, the authors can edit the pathway to fix the issues
– Enable researchers to directly report an issue to the Biofactoid support team
– Reserve the right for the Biofactoid support team to lock pathway editing, flag it as disputed or remove it from the collection
7. Please include the overall number of recorded facts generated over the multi-step process, including the number of papers used and the facts per paper.
At the time of writing, Biofactoid contains data for 73 articles in which a total of 330 entities (263 unique) from 7 organisms (
8. Please temper your statements regarding the search and exploration functions of the system. Currently biofactoid.org seems to have no query interface. Since the data is not yet in PathwayCommons, interactive access is effectively only by browsing the papers shown in the landing page, but the figures and text imply much more functionality than is actually available to the user at this point.
We have altered statements from our initial submission that more precisely reflect the existing capabilities of Biofactoid.
– Abstract: “The resulting data is interoperable with public information resources, allowing biological facts from different research reports to be unambiguously connected and making it possible for author-curated knowledge to be appreciated in the context of all existing computable knowledge.”
– Data sharing and exploration: “Beyond the “Explorer”, Biofactoid data represented in the formal BioPAX ontology is interoperable with external pathway and interaction knowledge using technologies for BioPAX processing and analysis, including Paxtools (Demir et al., 2013) and cPath2 (Cerami et al., 2006). This permits author-contributed information to be connected to existing information, and leverage tools for searching, visualizing and analyzing data across different sources.”
– Figure 2
– Legend title: “Biofactoid data is connected to information sources and establishes a bridge between related pathways”.
– Legend: “Biofactoid data establishes a bridge between related pathways described by structured biological knowledge from distinct data sources.”
– Figure 2B, subtitle: “Bridging related pathways.”
9. Please include a comparison to related approaches, considering the citations referenced in line 235 of the manuscript.
A summary and comparison of related approaches is presented in Table 2 which is referenced in the Discussion. This table includes projects that support community curation of biological pathway and interaction knowledge. Given this refined criteria, we have removed references to NDEx (Pratt et al., 2015) and GraphSpace (Bharadwaj et al., 2017), since they are primarily concerned with network information storage and dissemination; removed the reference to Wikidata (Waagmeester et al., 2020) as it only stores metadata for pathways imported from elsewhere (e.g. Reactome, WikiPathways); and removed the reference to INDRA-IPM (Todorov et al., 2019) as it does not involve data storage.
10. Please state under what license contributors to Biofactoid are contributing their data, and make it clear both in the article and also in the website, so that users know how they can (re)use the data.
We have added a "Data availability section to Materials and methods that indicates all contributed data is made freely available without restriction under a Creative Commons CC0 public domain licence. Likewise, the Biofactoid homepage now states that “All Biofactoid data is made freely available to download to the research community under a public domain-equivalent license (Creative Commons CC0 license)” (under section “By researchers, for researchers'').
11. Please provide each of the Figures in your manuscript as a separate figure file.
Each figure is provided as a JPEG file.