Sidebar
Lemmy Project Priorities Observations
Lemmy instance Beehaw staff on Monday, August 21 2023.... https://beehaw.org/comment/1018508 "From where I’m standing, I can’t really much has changed unfortunately… which really sucks… Lemmy.world has grown substantially meanwhile the moderation tools have not improved at all. All I can say about the moderation tools is that we now know that the tools suck more than they used to. Here’s a list of ***moderation problems*** that we have discovered since then: * If a Berson is reported on another instance, we never get the report. * If a mod is banned from the community they mod, they can still take mod actions * If you get site-banned from Beehaw while you are from another instance, you can still post on the community and people from that instance and kbin can see your posts * People from other instances can’t know who if someone is an admin on the instance they’re interacting with * People from other instances can’t see when we use the shield function to signal we’re talking “officially / as a mod” * The modlog is not chronological * The modlog breaks if you ban someone for more than 4 digit days. A banned user’s description is still visible so if they link to a scat image in their description, it is still visible to moderators. Despite these newly known problems, there have been exactly no improvement whatsoever to the moderation tools. It is honestly unsettling and terrifying." Context: Lemmyy has been on GitHub and in production at Lemmy.ml for over 4 years for the purposes of running and moderating a message forum / link aggregator. Beehaw has been online for over a year before the May 2023 Reddit influx.
These issues have been obvious for months, lemmy.ml wasn't sharing the server logs Now at least there are multiple sites with a modest amount of data who see these issues: https://github.com/LemmyNet/lemmy/issues/4017
Is there something special about the 28th day of the month and precisely 90 days? A very obvious server-crashing / denial of service problem was called-out in Lemmy code two days before the Reddit deadline. https://github.com/LemmyNet/lemmy/issues/3394 Observations: 1. Why would anyone think 5 is a good design for production in the first place. It puts into question the developers for over 4 years of experience - they clearly understand the technical issue - it is the same coding / parameter issue for any programming language. What is the motivation / priority here? 2. lemmy.ml developer-run server (then the Lemmy server with the most data) was crashing from PostgreSQL overloads May and June 2023 every day... 3. there were active countdowns to the July 1 Reddit API change, This was June 28. 4. The change takes about 30 seconds to code, by no means is it difficult to understand. But it must be approved by the core developers of over 4 years on the project... and even notify live sites to urgently edit the Rust source code and re-compile. (And why not move this value to an environment variable that can be set without recompiling Rust code?) June 28 issue opened / code created July 1 Reddit API deadline September 28 code published 90 days to change what has contributed to lemmy.ml, beehaw, lemmy.world - and the entire network of Lemmy servers crashing constantly from Lemmy overload. Almost as bad as GitHub Issue 2910 being ignored all month of June 2023! https://github.com/LemmyNet/lemmy/issues/3394
A change in direction for the project this week? Maybe the reputation of stability on lemmy.world and people realizing that the amount of activity really wasn't that high - and lemm.ee shutting out images. Most of all, Beehaw's criticism maybe finally resonated. Beehaw was online a full year before Reddit - and saw just how long-term issues were not being addressed... maybe that is what it took. It is worth keeping a positive eye on things.
It seems api_tests is unstable, failing around half the time.... been that way for days it seems.
Cambridge Analytica was well underway in 2013, now 10 years ago. People like to think that just because researchers find x on Twitter and y on Facebook - that that is the clearly documented cases - that the tactics and general psychology didn't copy everywhere. Cambridge Analytica is mostly famous for Facebook... but I don't view their direct targeting of individuals to be the long-term damage. The long-term damage is that they legitimized psycological manipulation, falsehoods, as a form of winning audiences. The were Psychology/Psychiatry professionals who applied human history and experience towards making people believe false things. Like a rebirth of Dr. AA Brill from 1929 on a new scale. The legitimization of it without any ethical uprising...
The only instance with significant creation activity that isn't all bot content... had to resort to cloudflare due to the data performance... and now the problems with that solution have started to be taken on... https://lemmy.world/post/4366376
yesterday .world had to turn off sign-up and even shut down shitposting community. This is basically the front-door of Lemmy. And as others are starting to notice, 60K users after all the people seeking better and trying out things isn't that many.. And there are seemingly a lot of people who use multiple servers given the technical instability of Lemmy's code... I am one of those people who spends 10 minutes a day on 4 servers, but I'm cutting back because content just isn't there and it's now often gaming topics and news making the rounds as duplicate stories over a 2 to 4 day period (besides memes) Social media in general... there was so much Facebook hate for the past 8 years... but not much betterment came of it. Today looking over Reddit, it hasn't really changed that much in the past 4 months... there was a group of dissatisfied people who don't seem to want to actually build something better - just want to protest Reddit. YouTube and TV advertising - that's a huge topic. YouTube does have a lot of small-time original creators, but is the money the reason why? You can''t make money on Reddit or Lemmy unless you have a business shop or something related to specialty topics (such as auto repair in a discussion community about same).
It isn't just the constant database crashes in Lemmy, GitHub issue 2910... https://github.com/LemmyNet/lemmy/pull/3708 On July 24, sanitation of HTML was added to the code. But the testing was not called for and it broken titles of postings, link parameter ampersands, discussion of programming code in code blocks It's data... and now it's very difficult to undo all the damaged data that has been put into the database for weeks now. Lemmy is a Link aggregator, and it damages Links now... the ampersand parameter deliminator in URL links is now broken because of this code not being tested. Why wasn't there a call for testing to something that was going to alter every new post and comment from both federation and Lemmy itself? how did such obvious things such as a ? parameter list in URL get overlooked... and then new bugfix release comes out after this was known as an issue - and still not fixed. Database crashing that results in lost data from unsaved post and comments, failure to deliver Federation data without any way client or server operators are notified, and damaged data as fundamental as URL website links... I'm all for code changes gong in fast, but the lack of actually testing things and spot-checking on Lemmy instead of just changing Rust code without really realizing that a link aggregator uses ampersand in URL links... and not asking people to help think of side-effects... Development process could even ask just a couple sites with more attentive operators to try out the code for a few days and ask people to report any problems before advising all sites to upgrade and break their URL links.
June 4, 2023 is when I felt I had to get involved. And June 8, 2023 is when I crated my own testing-focused experimental Lemmy install that compiled Main from GitHub source and ready to test the changes I thought were surely going to come to the SQL because of GitHub issue 2910. I wanted to test the code that that developers who had been working and running lemmy.ml for over 4 full years - would surely address. It was June 4 with Issue 2910, the June 30 Reddit API cutoff deadline countdown was well under way. Lemmy.ml put in major hardware upgrades on June 13, 2023 - and I was puzzled why such an easy 2 or 3 hour fix for Issue 2910 wasn't put in... but I still had hope that everyone would see the pending countdown to June 30 API deadline and a fix could go in within 10 days - by June 23... for some slack time before June 30.. Watching Beehaw, up and running for 17 months on Lemmy - crashing constantly... I thought, surely the developers were seeing Issue 2910 happening over there... but June 28 came, June 29 came, June 30... nothing. With 4 years of experience on the Rust code base and such, they were the ones to fix Issue 2910.... but from June 4 to June 30, it just didn't happen.
ORM and SQL .. I mean it's lived up to the bad side of the reputation. It hides it. As someone said here: "Osiris on June 18, 2021 | unvote | parent | context | favorite | on: DenoDB I hate ORMs with the fury of a thousand suns. The problem is that I know SQL but now I have to spend a bunch of time trying to figure out how to convert SQL into ORM X just so it can convert it back to inefficient SQL. SQL mostly translates between various databases but ORMs are unique and you have to learn a new API for each one. I'm on a project using TypeORM and it has been fantastic at helping developers on my team make really bad schemas due to not understanding how to use TypeORM to make the right relationships. Currently I'm looking at pg-types because you just write SQL and it just helps by making some TypeScript types for you. (I have used ORMs in C#, PHP, and JavaScript and I hate all of them)." https://news.ycombinator.com/item?id=27547163
In April and May, Lemmy had a great brand... but server crashes and then lemmy.ml closed sign-up.... sort of confusing people and then Lemmy.world became the brand, then it too has had to close home page many times and constant crashes.... The SQL performance issues that were in lemmy for years really - became trouble once data was introduced. I don't even know how many times people have come to sing-up and heard about Lemmy to find it slow, crashing, errroring. Social hazing, rock star glamour attitude among the audience who keeps using Lemmy... but these odd trends in 2023 with social media aren't unique to lemmy, are they? Threads, Twitter to X, Reddit API change, Lemmy crashing and not really caring about the crashes - and Redis or Memcache to mitigate it.... it's all such an odd year. It's as if everyone stirred the pot with social media but no place to really land upon that isn't crashing or in some form of Elon Musk kind of chaos. I remember the rise of Reddit, the rise of Gmail - and it just felt like the growth was being dealt with. In 2023, Twitter seems to be leading the entire audience into accepting a kind of crash bad experience and 'stick around, no matter how bad it is' that Reddit users seem to have accepted. I don't want to be a Lemmy developer... I want the people who know Rust Diesel and such to actually make it work - or choose something else that does work. I want Lemmy to actually not crash all the time and at least be where Reddit was in 2008 when there were just a few programmers doing it. The SQL statements and server crashes in Lemmy speak for them self, just like the chaos of Reddit and Elon Musk Twitter speaks for itself... but a lot of people accept it. The audience is not asking for stability. Strange times in 2023!
Unbelievable today the push-back on filtering by what is in-essence, the entire key to lemmy - published date
https://stackoverflow.com/questions/74295698/how-to-make-a-select-where-exists-query-with-diesel
Ok, taking a total fresh approach based on all the months I've been reading people's suggestions and [!lemmywishlist@lemmy.ml](https://lemmy.ml/c/lemmywishlist) kind of things.... # All / All Remote / Local / Remote Specific-Isntance I think server operators should at minimum want the ability to view Remote-only, and even per-instance. Further, I think proxy of API to a community should be something to head towards... where a Lemmy API call can be forwarded as an API call on another Lemmy server for a specific community. # Small, Medium, Large, Trending, Featured Community Some stock multi-community lists, some of which are dynamically generated like Small / Medium / Large based on number of active posts, users, or other tunable parameters. Encourage people to engage in topics that they normally would not see... # Multi-community sharing I think foundation is that it should use words and not numbers. Right now, the entire Lemmy system is built upon using localized index numbers for community. Even if it just becomes a JSON blob to throw into PostgreSQL and recall by name... Maybe have them like communities. And people can subscribe/unsubscribe to a specific list. And the list can have moderators who regulate it (editors). And an option to clone a list to new name. /mc/ multi-community, name. And no ID numbers. A Trigger or something would have to build the ID numbers in the background. And a browser of these, much like communities are browsed... and maybe even voting on them. Lemmy doesn't have voting on communities - subscribe alone - but sometimes you don't want to subscribe because they have too much content - but you would still vote for it or recommend it.
Weeks ago I had my moment of facing the attitude of keeping all this secret. Just casually mention join_collapse_limit was tried behind-the scenes a month ago, then why are there zero post or comments in the entire Lemmy search for join_collapse_limit? I searched the entire GitHub project - no mention of join_collapse_limit. But Ready on the Spot to reveal the secret private communications tried join_collapse_limit log ago. You know what join_collapse_limit is telling yo8u? Too many JOIN is a performance problem! The entire ORM Rust code and reliance on new JOIN is going to lead to other unpredictable performance problems that varies when there are 10,000 posts vs 2 million posts! And that's exactly the history of 2023... watching the code performance wildly swing based on the size of communities being queried, etc. What I see is that pull request for ideas get created only after noise is made on a subject. There is a lack of openness to make mistakes in public. For me,** the server crashes are what annoys me**, not human beings working on solutions. But for most of the people on the project, what seems to anthem is needing to have proper tabs vs. spaces on source code and even adding layers of SQL formatting tools in the middle of what clearly can be described as an SQL performance crisis. Things keep getting broken: the HTML sanitation took a few hours to add to the project but now weeks of broken titles, HTML code blocks, even URL parameters are now broken on everyday links. The changes to delete behavior have orphaned comments and that has gone on for weeks now.
PostgreSQL data insertion
chaos about big servers crashing from PostgreSQL overload continues... 
"Freshness" can also mean frozen fresh ;) Canned fresh. The key concept behind Freshness is to limit the number of posts per community, but also consider that older content if "fresh frozen" is fine for small communities.
This kind of attitude is why I got so upset over GitHub Issue 2910 being ignored for months - and not given specific project call-out/squeaky wheel needs fixed spotlight...  Anyway, PostgreSQL 15.3.1 upgrade to 15.4 now released today, waiting on .deb files.
Lemmy.world shared a slow query, in detail! Yey! SELECT post . id, post . name, post . url, post . body, post . creator_id, post . community_id, post . removed, post . locked, post . published, post . updated, post . deleted, post . nsfw, post . embed_title, post . embed_description, post . thumbnail_url, post . ap_id, post . local, post . embed_video_url, post . language_id, post . featured_community, post . featured_local, person . id, person . name, person . display_name, person . avatar, person . banned, person . published, person . updated, person . actor_id, person . bio, person . local, person . private_key, person . public_key, person . last_refreshed_at, person . banner, person . deleted, person . inbox_url, person . shared_inbox_url, person . matrix_user_id, person . admin, person . bot_account, person . ban_expires, person . instance_id, community . id, community . name, community . title, community . description, community . removed, community . published, community . updated, community . deleted, community . nsfw, community . actor_id, community . local, community . private_key, community . public_key, community . last_refreshed_at, community . icon, community . banner, community . followers_url, community . inbox_url, community . shared_inbox_url, community . hidden, community . posting_restricted_to_mods, community . instance_id, community . moderators_url, community . featured_url, community_person_ban . id, community_person_ban . community_id, community_person_ban . person_id, community_person_ban . published, community_person_ban . expires, post_aggregates . id, post_aggregates . post_id, post_aggregates . comments, post_aggregates . score, post_aggregates . upvotes, post_aggregates . downvotes, post_aggregates . published, post_aggregates . newest_comment_time_necro, post_aggregates . newest_comment_time, post_aggregates . featured_community, post_aggregates . featured_local, post_aggregates . hot_rank, post_aggregates . hot_rank_active, post_aggregates . community_id, post_aggregates . creator_id, community_follower . id, community_follower . community_id, community_follower . person_id, community_follower . published, community_follower . pending, post_saved . id, post_saved . post_id, post_saved . person_id, post_saved . published, post_read . id, post_read . post_id, post_read . person_id, post_read . published, person_block . id, person_block . person_id, person_block . target_id, person_block . published, post_like . score, coalesce ( ( post_aggregates . comments - person_post_aggregates . read_comments ), post_aggregates . comments ) FROM ( ( ( ( ( ( ( ( ( ( ( ( post_aggregates INNER JOIN person ON ( post_aggregates . creator_id = ? . id ) ) INNER JOIN post ON ( post_aggregates . post_id = ? . id ) ) INNER JOIN community ON ( post_aggregates . community_id = ? . id ) ) LEFT OUTER JOIN community_person_ban ON ( ( post_aggregates . community_id = ? . community_id ) AND ( community_person_ban . person_id = ? . creator_id ) ) ) LEFT OUTER JOIN community_follower ON ( ( post_aggregates . community_id = ? . community_id ) AND ( community_follower . person_id = ? ) ) ) LEFT OUTER JOIN post_saved ON ( ( post_aggregates . post_id = ? . post_id ) AND ( post_saved . person_id = ? ) ) ) LEFT OUTER JOIN post_read ON ( ( post_aggregates . post_id = ? . post_id ) AND ( post_read . person_id = ? ) ) ) LEFT OUTER JOIN person_block ON ( ( post_aggregates . creator_id = ? . target_id ) AND ( person_block . person_id = ? ) ) ) LEFT OUTER JOIN community_block ON ( ( post_aggregates . community_id = ? . community_id ) AND ( community_block . person_id = ? ) ) ) LEFT OUTER JOIN post_like ON ( ( post_aggregates . post_id = ? . post_id ) AND ( post_like . person_id = ? ) ) ) LEFT OUTER JOIN person_post_aggregates ON ( ( post_aggregates . post_id = ? . post_id ) AND ( person_post_aggregates . person_id = ? ) ) ) LEFT OUTER JOIN local_user_language ON ( ( post . language_id = ? . language_id ) AND ( local_user_language . local_user_id = ? ) ) ) WHERE ( ( ( ( ( ( ( ( community . removed = ? ) AND ( community . deleted = ? ) ) AND ( post . removed = ? ) ) AND ( post . deleted = ? ) ) AND ( community . local = ? ) ) AND ( ( community . hidden = ? ) OR ( community_follower . person_id = ? ) ) ) AND ( post . nsfw = ? ) ) AND ( community . nsfw = ? ) ) ORDER BY post_aggregates . featured_local DESC, post_aggregates . hot_rank DESC, post_aggregates . published DESC LIMIT ? OFFSET ?
did a git pull request for new index on post searches by community
It's been a week since I've had to do a major attitude adjustment. I've created tons of local comments on this server to try and figure out what is going on with performance that lemmy.world is still blocking all comment links: https://lemmy.world/post/2697806 I really wish developer would have rushed in a fix for the ampersand munging in URL and other content.... a lot of links now on Lemmy are broken because of the damage to the URL.
late May... what I saw was lemmy.ml turned off new user signup, community creation, and load-shed by letting the server error out. now lemmy.world is bigger in content and activity but the drastic blocking of incoming links to comments is in place. I think the number of active users has less to do with the performance problems than the amount of data in the database. The heavy use of JOIN and custom account-specific filters. An account with a large block list and a huge list of subscribed communities likely performs wildly different when listings posts than a non-logged in user.
 Resorting to having nginx block comment links regardless of response time or rate-limits. No concurrency self-awareness in this approach.
self-awareness in the API https://github.com/LemmyNet/lemmy/pull/3805
I've adjusted my attitude. Github issue 2910 has spelled it out to me how priorities work. lemmy.world seems to have not sent any data to lemmy.ml for days it seems: https://lemmy.ml/post/2650814
Lemmy.world had to shut down the front page and put up a message about the load and a graph. They seem to chalk it down to the nature of social media sites to attract attacks. I'd hack up the Rust code to have self-awareness of concurrency with PostgreSQL and return a new busy error. Federation connections, RSS feed, API - and any other method that is hitting the database needs to have a concurrency count in the Rust code and an error message system for busy. I'd probably build a a class to help with this and once concurrency for an API is over 5 mark the high water with a timestamp and start doing logic based on elapsed time. If > 5 and elapsed time exceeds a threshold (say 1 minute), then return the busy error. is Prometheus the right way to expose these numbers for operators wanting to know about the thresholds.? I'd probably add a dedicated log file to track concurrency thresholds and busy errors. the front-end apps also need to be caching "Trending communities", I think lemmy-ui is still pulling that live from PostgreSQL for every refresh of the page. I need to check if anyone has added that.
It was last Sunday, July 23 that the site_aggregates update of 1500 rows issue was discovered. Lemmy.ca made a clone of their database. The UPDATE of 1500 rows in site_aggregates is an example of what keeps slipping through the cracks. lemmy.ml as a long-running server may have data patterns in it that a fresh-install server does not easily match. Lemmy.ca cloning a live database to run EXPLAIN was exactly the kind of thing I wanted to see from lemmy.ml 2 months ago.
the open source cross-application platforms like Kbin or Lemmy, it's been really hard for me to adjust to the chaos of it all and the server crashes causing so many people problems.
I clearly have failed to understand how integral to the social identity of the Lemmy community server crashes have become. The "little guy taking on Reddit and Twitter" for 4 years. I have personal DB2 and PostgreSQL experience in production "mission critical" applications. I should have seen that for 4 years the developers had all the social experience of social -- like Hollywood films or Music Industry labels. My brain damaged mind was WORRIED about the June 4, 2023 GitHub Issue 2910 so much that I started organizing [!lemmyperformance@lemmy.ml](https://lemmy.ml/c/lemmyperformance) on June 13 after lemmy.ml upgraded hardware and I saw more crashes. It's cultural. I should not have looked into the kitchen and let my personal DB2 and PostgreSQL memories haunt me that this was "production" system crashing. I fucked up, and I've wasted 2 months in WORRY about the Reddit API deadline the first month, and now the Elon Musk X rename, and I admit - I didn't get the social experience the developers have for 4 years working "social communities". I study religions, I should have interpreted it using that hat instead of "production PostgreSQL or DB2". It's been a bad year for me, and I've made several big mistakes. I had to drink to clear my mind for the first time this year since New Years. Those memories of running "mission critical" servers had to be shut off, stop applying them to socially-driven Lemmy culture. The developers have been running this site for 4 years, they know modern audiences and what works - I've just been pissing into the wind with the wrong interpretation :(
Perhaps I've had the wrong attitude about the server needing to be reliable, data loss importance, and crashes causing all the front-ends to look bad, etc When I step back and study the history of Microsoft vs. Apple. the "open hardware" approach of Microsoft proved to be abandoned. Apple always had the best loyalty by NOT offering hardware choice to their customers. and Linux, which offered the most hardware choices of all, was even less popular. Apple knew people didn't like to pay money for software, so they were an operating system company who packaged it with hardware. I know project developers like Lemmy do not get good money. Maybe I have been far too harsh on them ignoring PostgreSQL crashes and should just face up that the Lemmy community, the users of Lemmy, think it is all perfectly fine. 32 cores of CPU to run a server without all that much content, it bothered me a lot to see the money being spent like that and to witness server crashing. But there are people here who seem to actually enjoy all of it. It makes the fight against Big Reddit and Big Twitter seem more dramatic. If you look at how Hollywood productions are run in terms of expenses for sets, clothes, shooting locations - it can be millions of dollars for 90 seconds of a movie. Because it's in the sphere of social media, films are social. Perhaps I've been the wrong kind of fool and their approach to just allow crashes and not treat it like an important production server - is actually intuitive to how social projects of "taking on the big guys" works. If it was closed-source paid-for software, crashes would be an issue to challenge. But with open source, they seem to have taken the attitude that anyone can fix it - so not their problem. They really emphasized to me what they mean by "supported". And if lemmy.ml is their idea of a "supported" server, I should take it as reality and give up any idea I have that easily-fixed crashes are important. It's product defining. It was hard for me to come to terms with. I should have stuck to topics like ChatGPT, Neil Postman, Marshall McLuhan.
Hate of Twitter, hate of Reddit seems to have been the primary motivation for many people to join Lemmy. In the past 2 months, I have seen many people requesting feature in apps to block entire instances of Lemmy or enhance other blocking features. Bashing on Facebook was often a thing you saw on Reddit, bashing end-users. Are people largely attracted to this? I'm mostly just saying it out loud. These trends like what Elon Musk has done to Twitter, or ChatGPT becoming mainstream in 2023 wit all it's misinformation creation abilities... are happening. Regardless of the anything going on with Lemmy. ChatGPT is probably the bigger real change. The news/world news on Lemmy isn't really that different than Reddit from my 60+ days of reading. Same general style of comments, reactions, and flow of postings.