...

Commits (2)
This diff is collapsed.
 \chapter{Conclusion} The DFG-funded Conquaire project has been concerned with investigating the feasibility of reproducing the analytical phase of research in experimental sciences. We have conducted eight case studies in various areas such as biology, linguistics, psychology, robotics, economics and chemistry as a basis to understand obstacles and best practices towards ensuring reproduciblity of scientific results. The DFG-funded Conquaire project has been concerned with investigating the feasibility of reproducing the analytical phase of research in experimental sciences. We have conducted eight case studies in various areas such as biology, linguistics, psychology, robotics, economics and chemistry as a basis to understand obstacles and best practices towards ensuring reproducibility of scientific results. The reproduction of analyses still involves substantial effort. Originally, we had set ourselves the goal to invest a full working week (40 hours) into the reproduction of each of these case studies. In many cases, the time needed to reproduce a result has exceeded this amount by a factor of three. The reason is that, in many cases, while data and scripts were available, the documentation was not sufficient to reproduce the analyses without step-by-step guidance of the authors of the original publication that we set out to reproduce. In addition to the effort devoted to the reproduction itself, the Conquaire project has performed a number of workshops with all the researches from the eight use cases to introduce them to the goals of the project, to introduce Git, etc. As a conclusion, we can say that the success rate for reproduction was very high. We were able to reproduce the results within all case studies. Yet, the level of reproducibility was not the same for all project. According to the taxonomy of levels of reproducibility introduction in chapter \ref{conquaire_book_intro}, we have on clear case of full analytical reproducibility and three further project that reached the category of full analytical reproducibility by the end of the project after recoding analytical workflows using open and free programming languages. Four case studies have the status of \emph{at least} limited reproducibility as the reproduction of their work (still) involves obtaining third-party commercial licenses for tool. It requires a minimal further investment to bring these cases into the level of full analytical reproducibility. This is a clear success in our view, clearly showing that analytical reproducibility is feasible. The main obstacles for analytical reproducibility found were i) the lack of documentation and thus reliance on guidance by the original authors, ii) the reliance on some manual steps in the analytical workflow (e.g. clicking on a GUI) , iii) the reliance on non-open and commercial software, and iv) lack of information about which particular version of software and/or data was used to generate a specific result. An institutional policy and infrastructure can alleviate most of the problems mentioned above. Our experience shows that using a distributed version control system is a best practice to be followed and a basic step towards reproducibility. Our experience shows that scientists in any field can quickly learn to work with Git, in particular if GUIs such as GitLab are provided. Most of the scientists involved in case studies in Conquaire had no issues in uploading their data to a Git repository. Our experience also shows that scientists are deeply motivated to make their results reproducible, even if this leads to a level of exposure that might lead to errors being discovered. In some cases we discovered minor errors in plots, scripts etc. and the involved scientists were more than happy to correct these minor issues. The exposure and independent validation brings benefits that are generally appreciated. This is indeed an important conclusion from Conquaire. While at the beginning of the project we were sceptic how open scientists would be willing to make their research artifacts available and support reproduction, we are more than convinced that there is a strong culture within science of being as open as possible to ensure external scrutiny or validation of scientific results. Our experience has been positive thus and we would like to encourage research organizations world-wide in setting up policies encouraging their researchers to make their results analytically reproducible. On the basis of the results of Conquaire, Bielefeld University is working towards the establishment of policies in this respect. We would like to end this book with a number of clear recommendations to research institutions wanting to support their scientists in making their results reproducible: \begin{itemize} ... ... @@ -20,4 +27,5 @@ We would like to end this book with a number of clear recommendations to researc \item \textbf{Open software:} We clearly recommend to set up policies that encourage researchers to rely on open, free and non-commercial software to facilitate reproduction of results on independent machines without the need to install commercial software and pay high license fees. \item Metadata: Organizations should train and support researchers in creating high-quality metadata for their data and also train them in selecting and specifying under which licenses their data can be used. Consulting on data exploitation and use while taking into account privacy aspects is crucial. Bielefeld university has created a center for research data management with the mission of consulting and training researchers on such dimensions. \end{itemize} However, the most important lesson learned is that analytical reproducibility should not be considered as an afterthought and delayed to the end of a research project. Analytical reproducibility is easy to achieve if one designs experiments and software environments from the start with the goal to make analytical workflows executable on any server by a third party. This minimizes efforts needed as workflows are not disrupted in the middle of a project and minimizes the opportunity to post-modify data and results, thus creating transparency. Applying continuous integration principles from the start and taking into account data quality and publishing data and scripts early in the research process as well as specifying tests that monitor data quality and run analytical workflows independently of the researchers carrying out the research as well as publishing results continuously and transparently in some repository is an effective way of fostering analytical reproducibility. \ No newline at end of file
 ... ... @@ -44,7 +44,7 @@ Accordingly, the main objective of that study was to relate inter-species differ The overall data workflow used in this project is summarized in the chart shown in Fig. \ref{fig:fig2-workflow} (left column). There were three processing episodes: (i) data acquisition, (ii) manual editing and annotation, and (iii) secondary processing. The coloured boxes illustrate the procedure for recording the different types of data and how it was ultimately processed to reconstruct body and leg kinematics as displayed in Fig. 3 in the paper of Theunissen et al. \cite{Theunissen_EtAl_2015}. The colours of the boxes indicate the software used for a given step in the data processing pipeline (yellow: \textit{Vicon Nexus}; green: \textit{PixeLINK Capture}; blue: \textit{MATLAB}). The boxes and connecting arrows are labelled with the data file types produced, the relative file paths to the corresponding subdirectories, and the names of custom-written MATLAB (MathWorks, Natick, MA, USA) scripts. \begin{figure}[ht] \begin{figure}[] \centering \includegraphics[width=11cm,keepaspectratio]{images/fig2-Workflow.png} \caption{\textbf{Research data acquisition and processing pipeline.} For raw data acquisition, whole body motions were recorded with a marker-based motion capture system (Vicon) and an additional digital video camera. Furthermore, the anatomy of the animal, along with the marker positions on different body segments were recorded with a microscope camera. In a first step of manual editing and annotation, marker trajectories of selected episodes were labelled and, potentially, connected in case of recording gaps. This step resulted in a \textit{.c3d}-file, a file format described in section \ref{c3dServerIO}. The body pictures were used to generate a body model containing, for example, segment lengths and information about marker position in a body-centred coordinate system. The model is stored in a MATLAB \textit{.mat}-file. Finally, the kinematic reconstruction was achieved in MATLAB by combining marker trajectories with the body documentation. The resulting processed data, i.e., joint angle time courses, gait pattern, and velocity, were saved as another MATLAB file.} ... ... @@ -85,7 +85,7 @@ Accordingly, the main objective of that study was to relate inter-species differ \begin{figure}[h] \begin{figure}[] \centering \includegraphics{./images/fig3-MotionCaptureBodyKinematics.jpg} \caption{\textbf{A marker-based motion capture and whole-body kinematics calculations.} \textbf{A:} Insects were labelled with reflective markers. \textbf{B:} For kinematic analysis, the body was modelled by a branched kinematic chain. The main body chain (left) consists of the three thorax segments (Root, T2, T1) and the head. Six side chains (right) model the legs, with the segments coxa, femur and tibia (cox, fem, tib; only right legs are shown, labelled R1 to R3). All rotation axes (DoF) are indicated (3 for the root segment, 2 for thorax/head segments, and 5 per leg). DoF are denoted according to the subsequent segment and the axis of the local coordinate system around which the rotation is executed. Leg DoF are: cox.x, cox.y, cox.z (labelled for R2 in right panel), fem.y and tib.y (labeled for R1 in right panel). [Fig. 1 A, B of \citep{Theunissen_Duerr_2013}]} ... ... @@ -165,7 +165,7 @@ As a result of our reproduction experiment we could reproduce the walking and cl Figure \ref{fig:compare_duerr} shows on the left the original panel from the paper published by Theunissen et al. \cite{Theunissen_EtAl_2015} for \textit{C. morosus}. On the right, our reproduction of the same trial is depicted. As the figure shows, asides from the rendering of the obstacle and the colouring, we could successfully reproduce the plots from the original paper. \begin{figure}[ht] \begin{figure}[] \centering \includegraphics[width=12cm]{../ch2-BiologyDuerr/images/fig5-compare.png} \caption{\textbf{Representative trial of unrestrained walking and climbing behaviour of \textit{C. morosus} as one of the three species investigated in the original paper published by Theunissen et al. \cite{Theunissen_EtAl_2015} (Figure 3).} ... ... @@ -185,12 +185,14 @@ Figure \ref{fig:compare_duerr} shows on the left the original panel from the pa We have described a reproducibility case study in the field of biology. We have in particular attempted to represent the main results of a study in whole-body movement analysis of three species of stick insects. The main objective of the study was to relate inter-species differences in kinematics to differences in overall morphology, including features such as leg-to-body-length ratio, that were not an obvious result of phylogenetic or ecological divergence. We have shown that we could successfully reproduce a main figure of the paper \emph{Comparative whole-body kinematics of closely related insect species with different body morphology''} by Theunissen et al. \cite{Theunissen_EtAl_2015}. We classify this case as one of \emph{limited analytical reproducibility}. While we could reproduce the whole-body movements for a number of experimental runs that the authors provided in a GIT repository, this has only been possible by direct guidance of the authors. Further, the reproduction relies on use of commercial software, in particular MATLAB as well as the C3Dserver running on Windows only. \FloatBarrier \section*{Acknowledgements} We would like to thank Florian Paul Schmidt for uploading the files to the \textit{biological-cybernetics} repo in the Gitlab \textit{Conquaire} group. We would like to thank Lukas Biermann and Fabian Herrmann (Student Assistants in Conquaire) for helping with the reproduction of the analyses in MATLAB. \bibliographystyle{plain} We would like to thank Florian Paul Schmidt for uploading the files to the \textit{biological-cybernetics} repo in the Gitlab \textit{Conquaire} group. We would like to thank Lukas Biermann and Fabian Herrmann (Student Assistants in Conquaire) for helping with the reproduction of the analyses in MATLAB. \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch2-BiologyDuerr} } ... ...
 ... ... @@ -338,6 +338,7 @@ In future work, the potential and benefits of using virtualization in combinatio %A detailed description of the work was presented in . \FloatBarrier %\bibliographystyle{plain} \bibliographystyle{unsrt} %\bibliographystyle{alpha} ... ...
 ... ... @@ -256,13 +256,14 @@ The data has been uploaded to the DFG FOR1525 project website (https://www.ice-n %aimed at reproducing the analytical workflow that lead to the results published in the paper \emph{BINARY: an optical freezing array for assessing temperature and time dependence of heterogeneous ice nucleation'} by Budke and Koop \cite{Budke2015}. The central diagram of this work showing the relation between the number of active sites of ice nucleation in dependence of temperature could be successfully reproduced by reimplementing the original analytical workflow in OriginPro via a Python script. As we did not exactly reproduce the original workflow, we have thus a case of limited analytical reproducibility. As a result of the project, both the derived data and the Python script described in this chapter are available for further re-use and validation of the original results. % %S-7 \FloatBarrier \section*{Acknowledgments} \label{Ack} %S-7 We thank Carsten Budke for providing the data and technical discussions during the computational reproducibility process. \bibliographystyle{plain} \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch4-ChemistryKoop} } ... ...
 ... ... @@ -321,11 +321,13 @@ Firstly, if researchers store their simulation data on an ongoing basis during a Secondly, if pre-generated simulation data is available from a published paper from the original authors, then the FLAViz toolbox could be directly applied to this dataset to reproduce the plots of the published paper. These can then be used to check the validity of the claims made by the original authors in their paper. It is in these two important ways that toolboxes such as FLAViz can be regarded as helping us to ensure the analytical reproducibility of research data. %S-*7 \FloatBarrier \section*{Acknowledgements} We would like to thank Krishna Devkota for implementing the FLAViz library and Fabian Hermann for documentation and bug fixing. \bibliographystyle{plain} \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch5-EconomicsHoog} } ... ...
 ... ... @@ -311,12 +311,13 @@ The analytical pipeline that was used to generate results for publication was de %S-7 \FloatBarrier \section*{Acknowledgments} We would like to acknowledge the support of Lukas Biermann and Fabian Herrmann for helping with implementation of the scripts and data analysis. \bibliographystyle{plain} \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch6-LinguisticsRohlfing} } ... ...
 ... ... @@ -542,12 +542,12 @@ The demonstration code for the Deep Disfluency library worked out of the box, en The research project is already very much aligned with FAIR data principles as it adopts open software practices and makes large parts of the original experiments easily accessible. Overall, this case corresponds to a case of limited reproducibility as the results could be partially reproduced for the offline settings, albeit not exactly. \bibliographystyle{plain} \FloatBarrier \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch7-LinguisticsSchlangen} } % Add Bibliography to ToC \addcontentsline{toc}{section}{Bibliography}
 ... ... @@ -316,7 +316,8 @@ Inspite of all data being available and the results being in principle reproduci %S-6 This chapter has described a Analytical Reproducibility case study in the area of neuro-cognitive psychology. In particular, we have described our effort to reproduce the main results of the article by Foerster and Schneider: \emph{Expectation violations in sensorimotor sequences: Shifting from LTM-based attentional selection to visual search'} \cite{foerster_schneider_2015b}. The main result of the article mentioned above was the finding that expectation violations in a well-learned sensorimotor sequence in humans caused a regression from LTM-based attentional selection to visual search. The authors of the original publication (also co-authors of this article) provided the Conquaire project with all primary data and all scripts and spreadsheets used to reproduce the results. While we were successful in reproducing the results, we classify this use case as one of \emph{limited analytical reproducibility}. The reason for this is that some parts of the analytical pipeline rely on proprietary and commercial tools such as Matlab or SPSS that can not easily be replaced by open and free tools. Further, the lack of documentation of the pipeline requires interaction with the original authors to reproduce the pipeline faithfully. Both limitations could be easily overcome if further efforts are invested. %S-*7 \FloatBarrier \section*{Acknowledgements} %S-*7 We thank Lukas Biermann and Cord Wiljes for assistance with the reproduction of the analyses. ... ... @@ -328,7 +329,8 @@ We thank Lukas Biermann and Cord Wiljes for assistance with the reproduction of %\end{verbatim} \bibliographystyle{plain} \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch8-PsychologySchneider} } ... ...
 ... ... @@ -306,13 +306,16 @@ This chapter has shown that it is possible to reproduce a robotic experiment at %% References with bibTeX database: \bibliographystyle{plain} \FloatBarrier \bibliographystyle{unsrt} {\raggedright % group bib left align \bibliography{ch9-TechnologyWachsmuth} } % Add Bibliography to ToC \addcontentsline{toc}{section}{Bibliography} %% Authors are advised to submit their bibtex database files. They are %% requested to list a bibtex style file in the manuscript if they do %% not want to use model1-num-names.bst. ... ...