The #OpenTox 2011 predictive toxicology framework evaluation report is now available from the OpenTox Reports area or can be downloaded from:
OpenTox D1.4 Report on Evaluation of the Final Implementation
I provide a summary here.
The final OpenTox Evaluation report summarises the testing and evaluation carried out on OpenTox in the final year of the project. The evaluation included assessment of OpenTox from the following points of view a) framework, b) standards, c) ontology, d) data resources, e) services, and f) applications.
In 2011 an OpenTox Virtual Machine was developed. This allows the user to download and install a virtual machine on their computer where OpenTox applications can be run in a local environment. Hence although OpenTox supports a distributed set of resources, it may also be collapsed to run as a local stand-alone application. We brought the virtual machine software to SETAC Africa and successfully installed it on ca. 40 laptops of workshop participants.
Online tutorials have also been created supporting usage of the applications on an ongoing basis. International use in 2011 has been broadened and demonstrated by the coordinator who for example has worked with risk assessors at Health Canada and African scientists at SETAC Africa in the use of the applications in workshops, in addition to responding to a number of presentation invitations from industry and international conference organisers.
These developments were disseminated and evaluated through a number of means, ranging from internal testing and evaluation by non-developers within the OpenTox project, to provision of online tutorials and exercises, to a hands-on workshop involving 9 exercises carried out in the final month of the project, which was subsequently made available online for further evaluation, and to dissemination at a variety of international conferences. This report summarises the results, feedback and evaluation of these developments of the OpenTox framework and its services and applications. We also provide recommendations for next steps and future directions.
Internal testing and evaluation supported the development of a number of OpenTox resources, services and applications so that they reached the stage of prototypes that could be provided to users to carry out exercises in a workshop setting or on their own. Underlying these is the OpenTox Application Programming Interface (API), which is an important output of the FP7 research project. It provides a specification against which development may continue by both OpenTox partners and the broader development community. In addition to further development of existing resources and applications, new ones may be created compliant with the specifications, thus supporting a growing set of interoperable linked resources for the field of toxicology. As the specification is an open standard it may also continue to develop and improve. As the latest version of the API (1.2) supports Authorisation and Authentication, such infrastructure can support the integrated use of both open and commercial resources. Policies for controlled access to resources can also be implemented which supports user access to resources based on licences, data sharing agreements and legal contracts. The use of OpenTox API-compliant REST services to communicate instructions between linked resources with URI addresses supports the use of a wide variety of commands to carry out operations such as data integration, algorithm use, model building and validation. OpenTox thus provides a “lower level programming language” to the more technically-oriented user for carrying our research, executing tasks or application development.
The OpenTox ontology development made good progress in the last year of the project and included development of an ontology to support endpoints and organ effects, and the capability to convert from the existing ToxML data standard into a semantic web format. From the toxicology ontology workshop held at the EBI resulted a perspectives review and a toxicology ontology roadmap to provide guidance to the field. As OpenTox creates a semantic web for toxicology, it should be an ideal framework for incorporating such future ontology developments, thus supporting both a mechanistic framework for toxicology in addition to best practices in statistical analysis, a combination that the field will surely profit from.
A number of (Q)SAR models were built with OpenTox for a number of REACH endpoints (carcinogenicity, mutagenicity, aquatic/fish toxicity, LogP), and a representative subset is presented in the report. Several of these models have been made available with validation reports through the ToxPredict application. We propose a “challenge” should be organised in which the community could develop different models using OpenTox services applied to different REACH endpoints.
The OpenTox 2011 conference in Munich, held at the end of the FP7 project in August 2011, attracted a diverse and knowledgeable audience including many principle investigators in the field, working in different areas. Hence, the group of ca. 80 scientists was strongly cross-disciplinary, cross-sector and international. With this backdrop we also ran a final workshop in which 9 different OpenTox exercises and applications were run by participants in groups in a hands-on format. It is difficult to communicate the intensity of the day as many participants engaged in a variety of activities all of which had been made possible by the project. This report summarises those experiences extended through additional experiences that went before and were continued afterwards through testing and evaluation and through the provisions of online tutorials.
The evaluation of OpenTox by these users provided a clear response that the project has produced very valuable and concrete results. Moreover, although still at the prototyping stage, many users were able to carry out activities with applications of significant scientific interest, including activities that had not been previously possible for them. The majority of evaluation answers to our questions produced answers in the excellent and very good categories, providing a sound endorsement by the user community that we have created something really valuable and worthwhile from the project. OpenTox should therefore be seen as a special success story the European Commission has supported through its research program.
At the end of every research project there are always issues that the work has surfaced which provide input into future work. Here we summarise some of the main issues which we currently have discovered or recognised:
1. OpenTox is a brand new framework, hence developers need to learn about the new approaches before being able to implement;
2. It is difficult to maintain the performance of a distributed set of resources and to ensure the ongoing evaluation and stability of core distributed services;
3. Certain users may prefer a local application, which is an area requiring future development by application developers;
4. Additional resources are needed to ensure the quality of a resource e.g., to curate a data source and to filter and label for quality;
5. Applications requiring integration of resources from multiple partners require a significant integrated testing methodology and effort;
6. A well-defined API is insufficient on its own to support developers in their development of resources to be accessed by other applications;
7. A significant effort is required to incorporate support for a broad range of datasets, protocols and models in the expanding field of predictive toxicology. Even more important is communication between different disciplines who do not understand each other very well;
8. The “21st century” mechanistic approach to predictive toxicology requires a framework and methods that place numerous demands beyond the building of a “simple” (Q)SAR for a single endpoint;
9. A substantial amount of work is needed to organise and harmonise the set of ontologies needed for predictive toxicology. This will take a significant coordinated effort and investment;
10. Further application development, communication skills and learning methodologies are needed to provide easy-to-use fit-for-purpose tools for end users not specialised in computer science techniques.
We provide here recommendations for the future development and sustainability of OpenTox and its benefit and impact:
1) OpenTox should be developed further to support creation of a global interoperable infrastructure for Predictive Toxicology to aid achievement of the goals of the 21st century vision. This could be supported through a dedicated EC infrastructure project supporting the goals of a number of programs including European activities under REACH, FP7 Health, FP7 Environment, EC JRC, and Innovative Medicines Initiative and international activities such as the US EPA ToxCast program and Tox21. The unique benefit of OpenTox lies in its core commitment to open standards which can support in a neutral way the goals of all stakeholders;
2) A program should be put in place to implement the Toxicology Ontology Roadmap. OpenTox can play a leadership role in collaboration with other stakeholders;
3) In consultation with industry, CEFIC and ECHA, an analysis should be carried out promptly to identify steps that could be taken to optimise OpenTox for supporting the activities of industry in meeting the 2013 REACH deadlines;
4) A business ecosystem approach providing value to industry and SMEs should be developed for OpenTox in support of its economic exploitation;
5) Consideration should be given to the formation of a legal structure for OpenTox e.g., as an international organisation or foundation;
6) The potential of OpenTox to be exploited in the development of safer medicines should be pursued;
7) OpenTox has evolved a community of interested parties beyond the initial FP7 partners. This should be developed further as an active and vibrant network;
8) OpenTox partners committed at their final project meeting to a combined interest and resolve to continue to develop OpenTox beyond the end of the project. Doing so will be rewarding!
OpenTox 2011 Technical Reports
You can also access and download the OpenTox 2011 technical reports on the framework, database, algorithms, validation and confidential data. See links at:
http://barryhardy.blogs.com/theferryman/2011/12/opentox-2011-technical-reports.html
Recent Comments