Thank you, Alex.
So when talking about reproducibility in science, I wanted to start with clarifying some of the notions important for reproducibility.
And to start with, I want to introduce a good scientific practice.
the scientific research is committed to the good scientific practice in generating new knowledge
and this also implicates some other notions like research integrity uh clear um you change your
screen you're now seeing the other then no we're back no it's fine no then i have to
Oh, I think I have to share another screen.
Well, it doesn't matter.
I will just look at the other screen.
So this implicates other notions like research integrity, which is important to create trust and confidence in the methods and findings.
Also implicating the understandability, which is actually a quality of the comprehensible thought.
Then comprehensibility itself is the ability of stakeholders to understand development aspects.
And transparency, well, it is important to make the research process understandable to third parties.
This would integrate data documentation and availability for a good methodological word.
So, I've introduced the most important notions here.
And then we can come directly to the notion of reproducibility.
So this, just in a simplified way of expression, is using the same data, the same methods,
from this description of an experiment,
which should also result into the same output.
And next to reproducibility,
there's also the term replication or applicability,
which would integrate additional data on new data.
Again, with the same methods,
it should result to the same outputs.
So reproducible research is fundamental
for scientific integrity and reproducibility
would be the minimum standard,
particularly if replicability is not feasible.
So if you can't find with available resources,
additional data for example to have a visual overview on these terms and to extend them
again to start with reproducible we have the same data and the same methods which come to a certain
result replicable but also result in the in the same output using different data
and the same analysis methods and then the bottom two variants would then be important for innovative
application scenarios so advancing from basic research to applied research maybe
And we're talking here about robust using the same data and different methods to get to the
same conclusion. And in the term generalizable using even different data or additional data
and different methods to come to the same conclusion.
So this is about reproducibility, replicability, and all the notions introduced before.
We can advance to the criteria and the source why we are talking about it.
This is our questionable research practices.
So in the last decades, I would say, were several studies also surveying research practices.
And here we can say one summary where at least a fifth would agree to have committed
questionnaire research practices and almost half would know of cases of even broad
practices. So what are question research practices that could involve falsification
also a fabrication of data and other other things are data trimming and data cooking.
So explicitly adding or negating results to get to a wishful conclusion, let's say.
And something which is often also noted in this regard is p-hacking.
So working on this involves also trimming and cooking of data to
to get to a certain probability that would underline a hypothesis.
Yeah, and these question research practices then led to the introduction
of the term of reproducibility or replication crisis.
This term was coined over more than a decade ago now, and it stems
from the discipline of psychology but also was taken up in the field of medicine life sciences
a summary can be found in a paper from 2012 that actually comes back to to a paper from the late
90s the last century from the psychology field where the offers from the regional study finally
in 2011 confessed to a to a scientific fraud and a massive scale and after that double replication
patient studies have been published and filed and showed other issues throughout disciplines.
And one summary, one more recent summary showed that over half of the researchers faked to
reproduce experiments, either from their own or from other scientists.
And this can be, this is also nicely summarized by a recent paper, this can be due to several reasons ranging from honest errors.
So, unintentional mistakes due to complicated analysis, but also to shameful cases of data manipulation.
Now, I've introduced all the important notions and terms up to the replication crisis, and I can introduce our guest speaker.
oh sorry before i introduce our guest speaker i just wanted to mention this site of a website
of retraction watch where you can uh get up-to-date information of protective publications and what
happens out there in the scientific world so now the introduction of our guest speaker today
Today, we will talk about reproducible research and how to implement it locally.
It's a postdoc at our university for a time being, and a former student and research assistant
at the University of Göttingen.
I will stop sharing.
and Nick, you can take over.
Thank you, Claire.
Okay.
I hope you can see my screen.
Yes.
Perfect.
Okay.
Yeah.
Thank you, Claire, for the introduction.
Thank you for being here
for example locally here and give you an expression about my personal experiences with that
maybe to so that you actually know where who i am so basically my background is i'm from
the field of computational magnetic resonance imaging so these magnetic resonance imaging
devices as you can see it here on the right you probably saw them before and probably also
experienced them before i'm mostly working on reconstruction software so how to get from the
raw data to the actual images and even get or extract more information from this raw data than
and it's typically used, that's called quantitative MI.
I mostly focus on, or I mostly work
with an open source software,
which is called the Berkeley
Advanced Reconstruction Toolbox.
It's a software designed by a global community
and who or on which basically most in our institute actually work so um and i'm yeah i like open
software a lot and i'm almost exclusively work on open software um yeah so in principle i'm
personally really lucky because i'm in an institute and i have a supervisor with professor
who is really strong in reproducible science
and has a really strong focus on reproducible science.
Just to give you a rough overview.
So for example, we have a B annual reproducibility event days.
That's something I will talk to you a bit more later
or i will elaborate on this a bit later we have a policy to publish all the figure scripts with
our publications so kind of our publications are fully reproducible you can just run those scripts
and they will create you at least the reconstructions or even the full figures automatically
Some of the examples I worked on,
you can later check out in the PDF here.
And we also even have a continuous integration framework
to see if we change our software,
which is continuously developed
that this doesn't break previous papers.
we work on tutorials webinars and workshops so i'm really lucky to be in an institute which has such
a large focus on reproducibility so and from that i want to tell you a bit more about how i experienced
And my talk will be basically structured in,
I want to motivate the local influence
and especially the benefits you can get
when you implement reproducibility in your daily workflow.
Then I want to talk about briefly about how to learn
about reproducibility and use that as a bridge to tell you a bit more about a tool which i called
the reproducibility day which i developed and we are actually working with since 2021 in our
to have a lot of advantages, you will see that later.
So let's start with the local influence of reproducibility.
So we've already, or Claire already gave us an overview
about this importantness of reproducibility.
and I will maybe just add a few things to that.
So it's really about this democratization
and this democratized assess to research.
It's about enhancing this accountability
of this research integrity.
And also we have all this self correction process,
which research heavily relies on and this is also facilitated by over by reproducibility and also
open science so it's i think there's a lot of um yeah yeah there's a lot of over yeah kind of
joint areas of that so reproducibility is one but open science is also heavily added to this one
and um yeah there's also this opportunity that it can increase your productivity but
also the overall productivity like for example in this covet research what we've seen in the past
years but as i mentioned i want to mainly focus on the local perspective so what do i understand
behind this local perspective if i think about our institute and about the way of when we
implement reproducibility here we i see two basic areas where this can have an influence it's
on the on this whole area of our institutional or our institute and on my personal um on my personal
benefits so let's talk about the institute first so if you design a workflow or your workflow
reproducible this also means that you create a lot of documentation or even or at least improve
your documentation which make it much easier for colleagues to understand your work and to start
collaborations but also i will focus on the institute itself first but if you have a good
documentation also obviously institutes outside of your institute you want to collaborate with
benefit from that this simplifies this knowledge distribution especially in the institute itself
and with that can reduce the impact of personal fluctuations which is one of the largest problems
i think one of the many or one of the problems actually research has but personal fluctuations
and loss of knowledge is definitely one of that.
And if you have this knowledge spread and distributed,
that at least reduces the impact a bit.
From going from the institutional side
to the personal perspective.
So in principle, if I have reproducible science
and I publish them openly.
This protects me from accusations of research misconduct.
So it really protects me.
There are also studies which claim
that the paper citation rate is actually increased
if you have reproducible work
and you also publish this reproducible work
So others can really work with it, go through your analysis, run your scripts, run your tools
and so on.
But overall, it's also just a really strong indicator of eager trustworthiness and definitely
transparency which is also a very important point for research in my opinion this is more the general
personal perspective so from my experience how do i benefit from it directly so reproducible research
helps me to actually explain and share my work much more easily so if i want to go to a colleague and
i have a project this and this i can just say okay please have a look or can you give me a comment
on that i will send you a link to for example a wiki media page where kind of documentation is on
we will also see the benefits about this later for this reproducibility day tool
another large benefit is that if i have my own work reproducible it's also much easier to
to modify analysis steps from modify analysis steps i did before and use those then for future
work so it can really for me personally reproducibility really improves the effectiveness of how i work
and one other really important thing is that it gives me feedback about errors so for example i
participated in a challenge with some researchers in sheffield and they reproduce an abstract of mine
And they could, for example, find that the figure in this abstract does not fully match the text I described because I had a perturbation level, which was instead of 1%, 0.5%.
That's not a huge difference, but it's nice to know.
it's really nice no one else would have seen this difference because it's rather small but no no one
else could have been able to find it if they were not able to reproduce this work so it really gives
me feedback about this work so i can potentially correct it or yeah at least have it in mind for
the future and fix those scripts maybe a bit more in-depth example based on one of my publications
so if i think about this reproducibility so what we typically do in our institute and here
presented on a preprint servers,
but also we share the whole software and publish that.
We share the data, all the scripts, as I mentioned,
and in this special case, that's not always the case,
but also tutorial to kind of experience how this works.
um there are some key things i see which help you to design your it also depends obviously on the
discipline you are working in but which can help you to create a reproducible workflow and that's
the first thing is focus on building tools so the scripts or things you work on should help you
to be or should be able to be applied to other um analysis steps so the reusability and the sharing
is improved this aim of publishing the software automatically helps to regularly review the code
and also to clean it up to restructure it sometimes this is also in our institute quite nice because
my supervisor also has a eye on that and gives us feedback about this this is also a really
yeah a nice position but at least if you publish it there might be in the open source community
someone who also can give you some feedback about that or even colleagues what I also
recommend a lot is an automatization of figures that's kind of sometimes or it's hard to start
with but it makes it so much easier to modify stuff for revisions and really test reproducibility
if this figure generation is automatized if you have questions about that i'm really happy later
in the q and a to answer some questions about that and the last step i can really recommend is to
structure your workflow in a way that for example in my case with this computational focus script
publications are always in my mind so they individual scripts are documented if there are
are some hard-coded things, there are comments on that.
The keep it simple rule applies to those scripts.
So this really helps you also if you think back
and you want to reproduce your work in a couple of months,
understanding those scripts much faster.
And debugging is also really, really easy or much easier.
So that's basically from this side,
how I can benefit from reproducible research.
And yeah, in principle to some,
I don't have to do a huge summary here,
but there are many reasons to aim for reproducible research just from the local perspective.
So there are many good reasons for the institute you are working in,
but there are also many, many, many good reasons for you personally and for your work.
um in the second part of this talk i now want to briefly go over this idea about how you learn
about reproducibility so obviously you are here in this webinar so that's a really good start
that's one of the best ways first to get in contact with it if you didn't think about it before but
in my opinion or from my experience most of the people learn about reproducible in a cried
reactive phrase so from the personal side for example um if someone and i personally had this
in the past had difficulties to reproduce others or even my own work at some point that was also
the point when i then started to think about and reproducibility a bit more and for the institute
It's that the data or the documentation is actually lost if someone leaves the institute.
and one also really important fact is that the starting periods of new employees becomes much
longer if this documentation is not nice if the data is not there if all the work is reproducible
this is much more shortened um yeah so what i would recommend is doing proactive learning um
That's nice. So as I mentioned, you are here. So you are starting with proactive learning about reproducibility already. So on a personal level, it's visiting events like it's this. We had the OSA info day a couple of weeks ago in the beginning of January. This is also a really
nice event for this it's about reading examples or checking out examples or guidelines attending
courses if the university offers them we had like good scientific practices and so on
and yeah also to learn it from supervisors as i had the luck with our co-authors and reviewers and so
on discuss it with colleagues that's also really important that's on the personal side on the
institutional side there are even more thing or additional things you can do so obviously creating
internal guidelines and policies is really helpful for that so if a new employee is starting they
directly get confronted with reproducible research practices.
But a second way, and that's kind of the bridge
to the next section is we had really good experiences
with organizing regular internal training days, event days,
which is explicitly target reproducibility.
And this is the termed reproducibility day.
So what is the reproducibility day?
In simple, it's basically just meant to be a fun way to learn about reproducibility and to focus on reproducible workflows.
It's nice that if they are reproducible papers, that's also great.
It's easy to check, but it should give a comfortable environment to regularly experience reproducibility.
The basic idea, or I had the basic idea to start organizing this when I thought about live fire training drills.
fighting this fires quite regularly and it needs to be a reactive or kind of a way which works
automatically because you need to have experiences in that and i think it's the same with reproducibility
so it's really important to get in touch regularly and also to test the reproducibility
of workflows and publications and so on.
And that is basically this origin
when we started about the reproducibility day.
So how is it structured?
So the basic idea is really simple.
You have a group, for example, our Institute here,
everyone in the group works on different topics maybe some have overlaps but in principle these
groups are then clustered or this whole group is clustered in smaller individual groups of
two to three people who ideally have different perspectives so ideally some topics which are far
are off from each other.
So they don't have prior knowledge about what each,
the other person, the team member is actually doing.
And then it's really easy.
It's a full day, which is structured in an introduction,
but basically in two blocks and these blocks are kind of the same.
So in the beginning, everyone in this group tries to reproduce the work of the other group partner or of the other group member.
so they typically in our group re-exchange we have one central documentation page
and this central documentation page is then used as a starting point so the other person in the
group gets your your for example documentation page and in the first part of this block starts
to to try to reproduce your work what you actually did is going reading through your documentation
having a look if it's understandable what you're writing if the aim of your project is understandable
if links and projects objects and so is working if raw data is there if the naming looks okay
if you can find everything so there are many things which can be checked even up to if you have code
which is running already just run the whole reconstruction and will produce stuff for example
and after this reproduction block then it's kind of the groups meet again and exchange feedback
And this is really amazing because every participant can actually receive feedback about your work.
So I think before that, I think it's a really great opportunity for everyone to kind of receive direct feedback about their own workflow.
So if you do that regularly, it would be amazing to have,
it is amazing to have someone who's going through your code
and all this kind of stuff.
And you can discuss your code with,
so it's a great opportunity to receive this feedback.
and this simple structure is kind of repeated
once in the morning and once in the afternoon.
Afterwards, there's just a final closing session
where we, for example, typically discuss feedback,
how the whole day works, what were challenges which occurred
but so that we can solve it for future sessions and so on.
So that's basically a feedback route.
And afterwards social gathering
is also really appreciated typically.
Yeah, so this reproducibility just from the experience.
so we did it now five times since 2021 the goal basically was biannually it almost worked but
with these five times we actually have or got quite some experience with that and it was typically
that we had 10 to 16 participants um since professor ucker went to the um uh yeah technical
university here in grads we have two locations where we coordinated this reproducibility day with
and yeah currently we focus mainly on scientific personnel like postdocs and phds but yeah we also
started to integrate interested student master students but that's always a bit difficult because
of internal rules and assess to data and code and we don't it's just nice to have someone
have a work contract for some of those things so students is not even not always so easy to
integrate them but in principle it's definitely recommended because i in my opinion you cannot
but starts to learn about reproducibility early enough.
So basically the advantages of this reproducibility day
is that for the individual person,
you can learn more about tools
because you have a look and regularly a look
into the workflows of your other group members.
You can see what type of tools they typically use, how do they structure their workflow or what do they change compared to you into your own workflow and so on.
it's a really nice thing to give feedback and to learn how to give feedback and so for practicing
communication and also obviously to learn about other projects that's one of these huge advantages
of distributing knowledge which i mentioned in the beginning in the institute this is the perfect way
of doing it if you have to or if you work through the work of a colleague for a whole day you learn
quite a lot about their projects and from the institutional side it's very similar to reproducibility
in general so knowledge distribution there's way more documentation which is created
um the networking and collaboration within the institute is really and hard or yeah increased
so because you work with the colleagues projects you learn much more about them and maybe you have
a tool especially if the projects are maybe not as far away as it was originally intended but then
you might find an area where you can even work together
and make something better.
And overall, because this knowledge is distributed
and the documentation has improved,
this overall impact reduction for this personal fluctuations.
Yeah, so in general, this whole awareness
and the benefits of reproducibility
are really increased
through this reproducibility day event.
I think one of the most important things of it
is that it should be a comfortable atmosphere.
So it's a really nice opportunity
if you give someone the freedom.
important things is to leave the participants some type of freedom to design this day.
So we have this individual blocks that's a recommendation, but in the end, it's important
that they have a nice day with this and learn about things and want to participate in that.
so it's really about giving them a guideline and this feedback and how they kind of do this
reproduction and the feedback rounds they should take something for them and also that will help
the institute one really important thing to make this happen is to centralize documentation code and
data that's really important, which makes it much, much easier because we had it in the
beginning that it was not completely like that.
So we had difficulties with assessing private GitHub accounts and so on.
So having a central media wiki, for example, for documentation or a markdown-based GitLab or GitHub documentation page, then GitLab groups where kind of the code is centralized and centralized storage solutions really makes this much, much easier and I think is a great step towards reproducible science and everyone assessing this data
this local guidelines i also mentioned before is really not a really nice have naming conventions
and all of this so this is something the institute itself has to develop but which
the reproducibility day can also really help develop over time and improve over time
Yes, and the social gathering I already mentioned
is also really nice for networking
and team building in the group.
So overall, and that's also the last summary
of my talk here is that this reproducibility day,
i can really i would really like to encourage you to try this into in your research groups
to learn and to implement reproducibility locally because you experienced reproducibility
and learn about the benefits in your daily workflow you can learn a lot about different
and projects really this knowledge is distributed
in the Institute a lot.
And one of these really important points
is to receive feedback about your work.
Maybe someone finds an error for you
or an error in your work during one of these days.
it's really nice that it's found internally instead of yeah maybe then having to correct
it in publications or so um also yeah the strengthen or the scientific connections are
strengthened with.