cross-posted from: https://slrpnk.net/post/24729225
Are there any open datasets that track who is federated / defederated with who? I have the Lemmyverse datasets but it has nothing on node relationships.
lemmy-federate.com appears to have the info I am after, but it only appears to be reachable in a GUI webpage. Any way to get the dataset?
You can easily fetch through the API. You can either check API routes from the repo or I can give you some examples if you want to?
I don’t have full time Internet access. I pop into a library periodically. So I have found it useful to fetch the lemmyverse dataset and import it. Takes just a few seconds to fetch the whole dataset. Then when I am offline I search the DB to decide where to post, and write my posts offline.
I’m trying to grasp what you are suggesting. Do you mean I write a program to reach out to each instance and harvest the data? I think there are around 150 lemmy instances. I’m not sure how quickly that can be done, or if it would be welcome. When I visit slrpnk.net using my browser, there is an anti-bot check which takes a few seconds. I don’t imagine cURL or the like would get past that, which I suspect is some javascript code that must be executed.
In your repo I see routes/community.ts and nodes.ts. But therein I do not see a list of instances or communities to visit. Is
tsfor typescript? That lang is unknown to me.Here you go:
https://lemmy-federate.com/api/community.find?input=%7B%22skip%22%3A0%2C%22take%22%3A10%7D https://lemmy-federate.com/api/instance.find?input=%7B%22search%22%3A%22%22%2C%22skip%22%3A0%2C%22take%22%3A10%2C%22enabledOnly%22%3Afalse%7DYou can change the skip and take values to paginate through list.
thanks!
The instance list is tiny. But perhaps that doesn’t matter since instances can be derived from the community list.