So, thanks for having me. My name is Tim Fitschen. I work at Inisql GmbH in Göttingen, and I will talk about research data management with Linkahead today.
I'm very happy that you are having me at the Research Data Management in Austria event series.
And so, thanks to Alex for the kind introduction, and now I guess I will start right away.
First, I'm currently working on a project called a buttcut that's short for battery cell assembly twin.
And of course, this is not about research data management or data management in particular, but it's a part of this.
And the goal of the project is to create a digital twin for battery manufacturing towards new greener and more sustainable manufacturing processes.
To quote the battery 2030 plus roadmap.
This project is funded by the European Union as Horizon Europe Consortium.
And at buttcut, we have 18 or 19 partners.
The lead is the NMBU, Norwegian University of Life Science from Norway.
We also have one university from Austria actually, from Klagenfurt, maybe someone is around today.
And of course, at the bottom, we do data management and infrastructure for this project.
And Indiscale is using Linkahead.
So my company is using Linkahead for the data management services we provide to the project.
And we will develop Linkahead as a federated hierarchical knowledge base during this project.
Because we have several partners at different sites using different electronic lab notebooks or different processes to collect data.
And they need to share their data across national institutional boundaries and company boundaries.
But in a way that will secure sensitive data, intellectual property rights, regulatory and contractual obligations.
And then we will be able to integrate data from experiment the sites into this shared data space.
And we feed the data back into machine learning algorithms and AI training and modeling for this digital twin.
So but I'm getting ahead of myself about Linkahead already.
So maybe we should start with maybe a general observation about what at least we at Indiscale think is a research data management system.
A research data management system supports researchers in data management tasks.
So we put the focus on researchers as the users and maybe not so much the public audience in the first place or, for instance,
personnel that is mainly concerned with infrastructure.
So research data management really should be focused on researchers and their activities.
According to applicable rules and policies, these might be fair guiding principles,
research data policies from the institution, from the funding partners, standards and best practices from your community or general ones.
And of course, organizational, legal and individual rules in your institute or in your work group.
And also by taking into account the conditions.
And by that I mean capacities like storage capacities, bandwidth and so on.
Also taking into account data structure, file formats and software that is about to be used or that is used already and establish workflows
because researchers usually use the workflows that work.
OK, so we don't want to change those just to make it manageable.
We want to try to adapt software that manages the research data to the workflows, not the other way around.
And of course, taking into account legacy data resources because they can be a great treasure if you can use them.
And Linkahead is designed around the vision to ensure all that.
So we made Linkahead flexible to be adaptable to different conditions.
And we made it agile.
The ideas that you need to adapt and change step by step and approve iteratively.
I will come back to that.
And it should be unobtrusive, meaning researchers determine the workflow and the tools and the research data management should be built around that.
So our idea of Linkahead would be if there were a physical object doing Linkahead's work,
that there is some kind of little robot that sorts your stuff while you're away.
So your desk is full of research data, obviously, some papers that need to be scanned and so on.
And the researcher is away currently.
The Linkahead robot, Indiebot, we call them, is plugged into your laptop and will do the stuff.
And when you're back with fresh coffee, everything is sorted away and put into the right place and you can go on with your work.
Maybe looking back, Linkahead was developed since 2011 at MPIDS, the Max Planck Institute for Dynamics and Self-Organization in Göttingen,
under the former name ChaosDB, which is not a good name for marketing, but the community was from Complex System and Chaos Research.
So that name wasn't a soft setting for them.
It's used in production since 2016 and was published open source in 2018,
has increasing prevalence since 2020 in several projects and use cases, for example, for sample management as a literature database,
management of raw data, automation of data processing for laboratory automation, indexing and annotation of data lakes and repositories and many more use cases.
As I said, open source software since 2018. You can get the code on GitLab. I will have some links at the end.
And then there was the company founding Indiske to make the former ChaosDB and now Linkahead a sustainable solution that has a constant maintaining community
and that there are also some services to run and develop Linkahead if the community cannot do this on their own.
So Indiske does the data management or data services in general, but Linkahead is one of our flagship projects.
So let's talk about how you would use Linkahead and let's look at this data lifecycle.
And you've seen one, I guess, many times probably, and they always look kind of the same.
Of course, it's a cycle and we may start anywhere, but let's start in the lower right corner where you might collect data
with ELNs or with some in-house written software that drives your sensors or your, I don't know, your experiments.
And you collect experimental results, maybe feed results into numerical simulations.
You do processing of raw data, data analysis, and at some point you will publish some of your data and your results, of course,
and your analysis and your discussion. And this, again, is the source for new ideas, new documents, new protocols, new standards
and new research questions, which then again leads to new data being collected and so on and the cycle goes on and on.
And now the idea is that Linkahead is in the middle of that.
It's the first software after you collected your data and the last software after you put your data out there into repositories, into publications and so on.
And you can feed the data back into Linkahead after you got it out for your simulations and Linkahead also will optimize these processes.
Maybe now a word about agile data management.
So research happens usually before standardization.
And so constant change is about to happen.
And this not only is about data structure and formats, but also about persons,
also personnel, also about devices that are being used and also about what kind of metadata you want to collect because metadata is always important for searching for data,
for finding data and the metadata determines what you can search for.
And if you need very detailed search, you need very detailed metadata.
And if you only want to look for maybe the date and the person that did an experiment, you don't need that much metadata.
So everything is basically changing, not only the raw data.
And so a natural approach to navigate these ever changing environments is an agile approach.
And the idea is, of course, start small, grow iteratively and adapt.
And Linkahead supports this in many ways.
I mean, you can always add a new machine to your project.
You can always add new personnel, but the data model or your schema in that case needs to adapt to this too.
Linkahead's data model supports that.
It's not necessary to have migrations on your data every time you change your schema.
You can integrate legacy data as it is.
It can stay where it is. It can stay in the form and shape that it has.
And you can keep your data formats and you don't need to bother about the future changes right now.
I mean, you have to cross that bridge at some point, but maybe not today.
Maybe this is a good topic for discussion later.
I would be happy to show in the demo how you might want to do this.
So let me summarize some features that also render Linkahead a perfect solution for an agile approach.
That is, of course, semantic data model and searching capabilities, which are very powerful.
We have a schema that is defined by the users.
So there is no fixed, let's say, experiment simulation analysis data model that you have to use.
But you can define your own schema or use an ontology that you use in your community.
We have the web-based user interface. I'll come to that.
We have interaction with external file storages so your data files can stay where they are.
They will only get monitored by Linkahead.
We have server-side scripting and an integrated ETL system for optimization.
We have user management, of course, that you will need at your institution.
We have a role-based access control management.
And we have software libraries for many programming languages to further integrate whatever software is already running in your setups.
Here's an architectural overview.
On the left side, we see kind of the stack.
We have the server, KSDB server, maybe nowadays Linkahead server in the middle of the big block.
It has several APIs.
Of course, there's the web user interface, but also the REST API and the gRPC API for different software libraries in different languages
that are being used to integrate external software.
And then we have the backend that is the SQL backend for metadata that is stored directly in the server.
And the file backend that abstracts away the file system in the local network or in the cloud or whatever.
And on the right side, we see the architecture and illustration of the idea of the ETL system that I talked about.
And we call it the Linkahead crawler.
And the idea is you have usually files on the file system, but actually you can crawl any hierarchical data.
So you could also crawl data from an SQL database or something else.
And so this file system is only an example.
Maybe you have three folders, experimental data, simulation data, data analysis with different kind of file formats.
The crawler would identify based on the path.
So the hierarchy and based on the particular file format, what kind of data it has in front of it.
And it also will open the files if possible, if you use open formats and extract metadata from it and identify value patterns
that will tell the crawler which particular datum it is crawling right now.
So let's make an example.
You have a CSV table with several columns.
And maybe this CSV table is updated every day.
So you get a new row and the Linkahead crawler, of course, will identify the CSV file based on your criteria.
And you might specify that the CSV file is identified by the path where it lives.
And so the crawler would identify the file and then update the metadata in the database every time you changed your file.
And another example maybe is you have a video and one day you decide to move it from one data storage from one file system to another.
And now Linkahead crawler needs to identify this file because the path changes.
But for files, maybe the identifiable is rather a hash sum or some other properties of that file that makes it unique and that makes it identifiable.
So this crawler really is flexible in what he uses to identify data.
And then it can insert new data because it can identify new patterns as well or update existing data in the database.
Some examples what we do to integrate Linkahead with external software.
We have some examples where users use Linkahead together with the Jupyter Notebook.
And a typical workflow would be that users search for data in the web interface.
And they can even copy a snippet of Python code from the web interface that they would just paste into the Jupyter Notebook.
And you see that we used the ksdb library here.
New names as well Linkahead.
But it works this way as well.
And it executes a query.
It says select quality factor from record analysis with quality factor.
So apparently there's an analysis object with some quality factors.
And then you can use the data in the Jupyter Notebook to conduct your analysis.
And another integration that we are deploying is integration with E-Lab FTW.
We have good contact with the E-Lab FTW community and also with the company behind it.
And there's another YouTube video which demonstrates how you can enrich data that is stored in E-Lab FTW with external data,
maybe from your simulations or enhanced search capabilities if E-Lab FTW's searching capabilities are not sufficient for your use cases.
As I said, there's a live demo.
I won't show it right now.
But you can check it out if you want any time.
It's demo.indiescale.com.
And a few words on deployment and licensing.
The whole thing is open source.
So all components.
There's no hidden feature that you only get if you pay someone.
The documentation is freely available online.
And you could deploy it on-premise and you could deploy it in the cloud service hosted by Indiescale.
But actually all of our customers and everyone else we know who's using LinkHat right now is deploying it on-premise.
So cloud services currently not really being used.
I guess the research community likes to have the data on their own servers.
And yeah, we know why and we respect that, of course.
So a little advertisement at this point.
Indiescale is offering consulting and service for deployment and, of course, customization and development.
So you can deploy it on-premise and you can, of course, change the code yourself if you need to adapt something.
But yeah, you can also let Indiescale do it for you because we might be cheaper even if you let us do it.
So some current activities of the LinkHat project.
We are part as a interested party of the ELN consortium.
Check it out on GitHub.
And we are working on an Omero integration.
We will try to bridge Omero and Elab FTW in a project.
And the idea here is, of course, there are several workgroups at an institution using these both ELNs, Omero and Elab FTW.
And yet they want to share data with each other and use data from the other group.
And LinkHat is a nice way to link this data and also integrate legacy data, integrate data from third-party software and so on.
We are part of the Fair Digital Objects community.
Of course, there's nothing ready yet because the FDO project is still in the phase of specification, demoing and so on.
But be sure to have LinkHat as one of the first tools being able to produce Fair Digital Objects and interact with them.
And of course, what I'm working on currently, we are part of the Rising Europe Consortium, Budcat, which is starting at this month, actually, since the beginning of the year.
So we are still at the very beginning of this.
For further resources, please visit the website of LinkHat, getlinkhat.com.
See again at our demo.indiscale.com.
You can find the documentation online as well as in the repository, of course, at docs.indiscale.com.
The software repository where you can find actually all the software.
As I said, there is no open core system or everything open source.
Find it on GitLab.
And there are also some publications from myself and data from 2019 and something more recent that goes into this agile topic a little bit more in Ingrid's paper.
And yeah, it's accepted, but not yet published.
So maybe it will be 2024, I guess.
So to summarize, LinkHat focuses on researchers and their activities.
It has the flexible data model approach, approaches an agile process and tries to be unobtrusive.
It integrates into established workflows of the researchers and it's completely open source.
Oh, yeah. And that's it. Thank you very much.
Okay, yeah, thank you, Tim, for a really cool presentation.
And yeah, I was pretty sure there are some questions right now regarding the LinkHat tool.
I only mentioned before that we do not have time for a full demo, of course, because this would take a lot of time to show everything.
Would it be possible for you to show us maybe a little bit about the tools, maybe the interface so that our researchers, our participants, would see what they would do with when they use the tool?
Sure. Can you see what I'm seeing?
Yeah, we're seeing the browser. Yeah.
Great. So, yeah.
So this is already full screen.
It is, I guess.
So this is the demo at demo.indiscale.com and the link is in the slides as well.
And the demo has a tool that is, of course, usually not part of the interface, but only of the demo.
And the ideas that we try to introduce
the idea of LinkHat from a very general point.
We try to explain how we map real world objects to our data model, how in this particular example.
And the example here is from maybe analysis of musical instruments.
And one idea is that we collect digital twins of the instruments in here.
We have, for instance, John Lennon's guitar and a nice guitar.
And both of them are, of course, guitars.
And this is a very basic schema for this.
So a guitar is just a musical instrument having a bullion electric, a Euro price and a manufacturer reference property.
And this is how data of musical instruments, sorry, guitars is represented in LinkHat.
And of course, the tool demonstrates all these features around the data model in several chapters.
Here's a chapter on data model.
Here's a chapter on searching and a chapter on inserting data and using the web interface.
And of course, there are some more chapters that go into details of some of the features of the web interface, for instance, bookmarking.
Maybe interesting for you guys could be the chapter on workflows.
Because the idea here is that you might want to use LinkHat as a tool for quality management and you can attach state to your data.
And in this case, it says in the publish lifecycle, this musical analysis is already published.
And if I log in, I will also see the unpublished and under review objects, which are hidden to the general public, but which are available for everyone who's allowed to see it.
And the idea, of course, is the unpublished is being edited by anyone in the work group, then you can change the state to review and then only people with the correct permissions will be able to change it.
And only those people also will be able to finally publish it and then it becomes visible to the general public and also will be freezed.
So no one can change it anymore, except you want to unpublish it again and hide it again and you can start working on it again.
So this is a way to one hand allow to use LinkHat also as a repository shared with the public and for quality management tasks.
And of course, it highlights some things like the map used for anything that is map related data.
The map is here by the way, and it has another chapter on versioning.
So LinkHat has a versioning for all of its entities.
You can access the versioning info via the web interface and this one has actually a full history.
And of course, you might jump back in time to any point in time and any prior version of this entity.
And this is of course used for quality management tasks as well and also for having a full lineage of your and a full audit rate of your objects that are stored in LinkHat.
But this web interface, of course, is only a very, very general view on your data that is being stored in LinkHat.
And that is OK if you want to have this very general view, but usually the workflows will be when you start to search for something in the web interface.
But most of the work is being done under the hood in automated pipelines and so on.
So I guess no one will actually use the edit mode to create new data in the web interface, maybe for changing data model.
But the usual workflow is to automate this and hide it away in some ETL pipelines.
That's enough for a primer.
Yeah, so thank you, Tim, again for the presentation and also for this short overview about the tool itself.
So I would like to stop the recording now.