I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site’s servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.
I have experienced this on my personal server as well (DDOS), but at the time didn’t know why, exactly.
This isn’t the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.
E: Many people have replied comparing this to the Reddit “hug of death” or “The Slashdot effect”. This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.
There are two proposed solutions I have seen. I do not claim that this is exhaustive, just only limited to the proposals that I have stumbled across.
FASP - Fediverse Auxiliary Service Provider
The idea that mastodon has is to have a third-party service that does something like generate a link preview once and then share that to fediverse instances that need it. They have a whole spec they wrote up about these kinds of services that can be run to help out fediverse instances with different things like this, search, or discoverability. They call these FASPs - the repo with the spec is here.
I don’t yet know of any FASP that is live and running, so it is kind of a paper spec at the moment. I don’t do too much on the microblogging side of the fediverse though, so I very well could be wrong.
Federate link metadata and the post at the same time
This comes from an issue on PieFed’s repo. It is basically just an idea at this point, but it is a pretty basic one; have the originating instance fetch the link preview information and simply federate it out alongside the post itself.
Both Lemmy and Mbin already attach the preview image to the post via the
imageproperty, which is of typeImagewith a URL pointed at the hosting instance. Receiving instances can then fetch that URL instead of the article.Is there a reason why Piefed doesn’t use this already established approach?
Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.
I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It’s no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.
Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don’t honour robots.txt (or aren’t yet listed).
So we need people to just mass post links to bad websites, is what you’re saying.
i’m feeling less ambivalent about those reddit repost bots for a moment
Sounds like every instance is doing the preview generation. Maybe it could grab the preview data it needs on the original instance, and pass that around with the post…
It’s almost like the software that powers the internet was designed for a different time and is only still used because there’s monied interest in keeping it around. We need GNUnet.
is there something that makes this different to 17 other proposals for a new internet or is it just a case of
Oh wow GNUNet I havent heard of that in a while. How is it nowadays?
GNUnet helps building a new Internet
GNUnet is a network protocol stack for building secure, distributed, and privacy-preserving applications.
With strong roots in academic research, our goal is to replace the old insecure Internet protocol stack.
GNUnet provides privacy-by design
It provides [improving addressing, routing, naming and content distribution](https://www.w3.org/2014/strint/papers/65.pdf) in a technically robust manner - as opposed to ad-hoc designs in place today.
GNUnet provides privacy-by design
Money doesn’t want us to have privacy.
Lemmy does this to itself, too - when you create a post it federates out and then every Lemmy instance that receives it makes a GET request to retrieve the post from the original server. Even though it just received the exact same data. Some other fedi software is even worse - Akkoma does a GET of the id of every activity it receives, not just for posts but for Likes too.
It’s insane.
Anyway, proper caching at the reverse proxy / Cloudflare level solves it. The same kind of caching setup would do the same for sites serving HTML, like NPR too.
That’s weird. Is that how it’s supposed to work, or did they just implement it in a bad way?
I mean, just how many instances are gonna ping the URL? Several hundred, low thousands at worst?
Unless it happens to be some very large media content or a dynamic page with no caching, I’d think the automated federation hits shouldn’t be too bad.
The influx of users that might come after a viral post though, that’s rough…
Yeah but we can’t say we want to encourage people to have/build/share independent smaller websites then beat them up when they are linked. That’s a huge mismatch of morality. Either we want to encourage people off the big platforms and should make it easier for the little guys or we are blowing magic smoke.
It’s just bad design. What happens when/if the fediverse scales up to many more instances?
I liked writing efficient code but every place I worked didn’t care at all about it. All they wanted was a finished feature and a kanban note moved to finished.
Thing is, Fediverse is not a product
I remember reading the same thing about reddit, slashdot, fark, and other sites. But I like the FASP idea as a way to relieve this.
Why is that? Those are centralized platforms, they should only need to retrieve the preview once each?
https://en.wikipedia.org/wiki/Slashdot_effect but people were going in, looking at the link, sometimes overwhelming the server
That’s not at all the same thing. That’s just the inability to handle large amounts of unexpected normal user traffic.
This is essentially bot traffic driven by servers all independently but simultaneously fetching the same image, without even a single human necessarily looking at it.
From server’s perspective the only difference between those is the user-agent. Traffic is the same
But what OP is talking about is a technical detail that can be fixed.
The origin and purpose of the traffic is not the same. That’s why they block one and not the other.
A slight delay would be perfectly sufficient.
If they’re all delayed the same it would have the same effect. In order to prevent that it would require some form of coordination amongst them. And even then would have the same complaint about missing link previews. Better than nothing but not really a “solution”, per se.
Random back off is an ancient and very well understood algorithm. This is not exactly rocket science.
Random numbers exist.
If they’re all delayed the same it would have the same effect. In order to prevent that it would require some form of coordination amongst them.
A delay randomized within a certain range could accomplish this without needing coordination.
Same thing used to happen with reddit giving sites the hug of death lol
It’s unfortunate but kind of expected.
Its not at all the same thing
You’re right, it’s not like… thousands of network connections are happening to the same site in both cases resulting in resource exhaustion.
Oh, wait…
Having things in common does not make them the same thing. I explained this in the OP.
You’re right, neither of them are known for providing thousands and thousands of distributed network connections to the same destination resulting in that same destination server running out of resources. They’re clearly not the same thing at all, no, definitely not at all.
Is this the same thing as the “Reddit hug of death”?
Similar, except those are caused by users visiting a link, whereas the Fediverse issue is federated servers gathering info before users even see a link.
How many servers are we talking about? 10,000 ?
Why not have the users fetch the previews themselves. The user’s browser, that is.
For sure there is more users than instances. So the amount of requests would be grater but at least they wouldn’t happen at the same time.
But if fediverse ever grow to be as big as other social media, then a single post/link can be viewed by thousands of users at the same time causing the same effect.
Randomly stagger the times of each instance’s request.
+1-300 seconds
300 seconds is a lot.
That’s basically what’s now implemented, iirc. At least in masto
they could include the link preview data as part of the federation
Perhaps a modification to the ActivityPub specification?
