cross-posted from: https://slrpnk.net/post/24729225

Are there any open datasets that track who is federated / defederated with who? I have the Lemmyverse datasets but it has nothing on node relationships.

lemmy-federate.com appears to have the info I am after, but it only appears to be reachable in a GUI webpage. Any way to get the dataset?

  • activistPnk@slrpnk.netOPEnglish
    2·
    23 hours ago

    I don’t have full time Internet access. I pop into a library periodically. So I have found it useful to fetch the lemmyverse dataset and import it. Takes just a few seconds to fetch the whole dataset. Then when I am offline I search the DB to decide where to post, and write my posts offline.

    I’m trying to grasp what you are suggesting. Do you mean I write a program to reach out to each instance and harvest the data? I think there are around 150 lemmy instances. I’m not sure how quickly that can be done, or if it would be welcome. When I visit slrpnk.net using my browser, there is an anti-bot check which takes a few seconds. I don’t imagine cURL or the like would get past that, which I suspect is some javascript code that must be executed.

    In your repo I see routes/community.ts and nodes.ts. But therein I do not see a list of instances or communities to visit. Is ts for typescript? That lang is unknown to me.

    • iso@lemy.lolMEnglish
      2·
      21 hours ago

      Here you go:

      https://lemmy-federate.com/api/community.find?input=%7B%22skip%22%3A0%2C%22take%22%3A10%7D
https://lemmy-federate.com/api/instance.find?input=%7B%22search%22%3A%22%22%2C%22skip%22%3A0%2C%22take%22%3A10%2C%22enabledOnly%22%3Afalse%7D

      You can change the skip and take values to paginate through list.

      • activistPnk@slrpnk.netOPEnglish
        1·
        3 hours ago

        thanks!

        The instance list is tiny. But perhaps that doesn’t matter since instances can be derived from the community list.