cross-posted from: https://slrpnk.net/post/24729225

Are there any open datasets that track who is federated / defederated with who? I have the Lemmyverse datasets but it has nothing on node relationships.

lemmy-federate.com appears to have the info I am after, but it only appears to be reachable in a GUI webpage. Any way to get the dataset?

  • iso@lemy.lolMEnglish
    3·
    2 days ago

    It is difficult to detect the connection between instances. Because Mbin and Piefed either don’t provide federated instance list or the admins can turn it off. That’s why I completely stopped checking federation status in Lemmy Federate, I just send an API request and if it fails I assume they are defederated.

    So, I’m sorry, I do not have this data.

    • activistPnk@slrpnk.netOPEnglish
      2·
      13 hours ago

      Thanks for the feedback. Your dataset would still be useful nonetheless. I fetched the lemmyverse dataset and found it to be incomplete. E.g., it only has ⅔ of the slrpnk.net communities. So if I could combine your data it would help fill in some of the missing data. Is that available anywhere?

      • iso@lemy.lolMEnglish
        2·
        12 hours ago

        You can easily fetch through the API. You can either check API routes from the repo or I can give you some examples if you want to?

        • activistPnk@slrpnk.netOPEnglish
          2·
          12 hours ago

          I don’t have full time Internet access. I pop into a library periodically. So I have found it useful to fetch the lemmyverse dataset and import it. Takes just a few seconds to fetch the whole dataset. Then when I am offline I search the DB to decide where to post, and write my posts offline.

          I’m trying to grasp what you are suggesting. Do you mean I write a program to reach out to each instance and harvest the data? I think there are around 150 lemmy instances. I’m not sure how quickly that can be done, or if it would be welcome. When I visit slrpnk.net using my browser, there is an anti-bot check which takes a few seconds. I don’t imagine cURL or the like would get past that, which I suspect is some javascript code that must be executed.

          In your repo I see routes/community.ts and nodes.ts. But therein I do not see a list of instances or communities to visit. Is ts for typescript? That lang is unknown to me.

          • iso@lemy.lolMEnglish
            1·
            9 hours ago

            Here you go:

            https://lemmy-federate.com/api/community.find?input=%7B%22skip%22%3A0%2C%22take%22%3A10%7D
https://lemmy-federate.com/api/instance.find?input=%7B%22search%22%3A%22%22%2C%22skip%22%3A0%2C%22take%22%3A10%2C%22enabledOnly%22%3Afalse%7D

            You can change the skip and take values to paginate through list.