There were a few minor hiccups requiring some minor downtime, but we're now running `0.19.5`! 🎉 You can read the full release notes [here](https://join-lemmy.org/news/2024-06-19_-_Lemmy_Release_v0.19.5_-_A_Few_Bugfixes). The primary downtime was due to upgrading our Postgres instance from `15` to `16` and the total duration was ~15 minutes. While this wasn't required, it was good to get it out of the way now for any future changes that may require it. Additionally, we were able to finally migrate `pict-rs` from its internal `sled` DB to using `postgres` instead! This requirement was preventing the service from properly scaling as well as causing issues when the service became unhealthy. With these updates taken care of, I've gone ahead and reserved some capacity in efforts to reduce our monthly costs. While they haven't risen much since we started, I figured it'd be good to lock in some savings now. As always, let me know if you encounter any issues. Thanks!

Aside from some minimal downtime, we were able to update to `0.19.1` without much hassle! 🎉 This is the latest version that was released recently and contains some great, new features. You can view the release notes [here](https://join-lemmy.org/news/2023-12-15_-_Lemmy_Release_v0.19.0_-_Instance_blocking,_Scaled_sort,_and_Federation_Queue). Unfortunately, due to the nature of the upgrade this means that *everyone* will be required to log back in. Feel free to reach out if you happen to experience any issues with this or anything else related to the upgrade. Additionally, I've gone ahead and disabled community creation for the time being. This is in part an effort to try to curate quality communities that provide value to the broader Fediverse. If you have an idea, feel free to forward your proposal and we can discuss. Beyond that, there's some general improvements I'd like to tackle in my free time that will hopefully ensure we remain on good terms with the other instances. Happy Holidays! 😀

As some of you may know, a vulnerability was discovered and exploited last night against a couple Lemmy servers. The admins were quick to respond, crowd-source solutions, and mitigate the damage. Patches have been deployed and the malicious content has been removed. For more details on the attack itself, see: [Ruud's post on Lemmy.world](https://lemmy.world/comment/1064402) Unfortunately, due to the extent of the attack this required rotating our secret which forcibly logs out all users. This is for your protection even if we weren't directly attacked. The reason for this is because as part of the attack the exploit was harvesting any user's cookies that had viewed the content. If you are a user on any other servers and have not been logged out, it's possible those accounts are vulnerable. I'd suggest reaching out to those admins to ensure they're taking the proper precautions. We will continue to monitor the situation and proceed as necessary.

First off, apologies if you noticed any issues during the following upgrade. I tried to minimize the impact as much as possible but there were a few snags along the way. 😅 I finally got around to rebuilding everything using `ARM` instead of `x86_64`. This provides a nice performance boost as well as cost savings. Enough so that I was able to double our capacity for a nominal increase in price. Our DB is already over-provisioned so this should provide a nice buffer for the time being. Additionally, while I was in there I took the time to split the API so that the batch jobs could be run separately. This allows me to run more than one API at a time for increased scalability. Overall, it went well with little disruption and improves our stability for the future. See below for the technical details, otherwise enjoy what's left of the day. 🎉 ::: spoiler Technical details The two issues that came up were related to the CDN caching an error response and the ongoing battle with `pict-rs`. The CDN issue should be resolved and auto fix itself if it ever happens again. It will now override the cache TTL for error responses with a much shorter lifespan. And while everything is configured for zero downtime deploys, the embedded DB used by `pict-rs` locks the underlying file preventing another service from starting. This means it has to be taken completely offline until the lock clears (~10 minutes). I will continue looking into better ways to mitigate this in the future. :::

This is a general purpose instance for all to use. I decided to contribute to the Fediverse as a way of giving back after decades of using FOSS. And for anyone curious, the name lemmus comes from the genus of [Lemmings](https://en.wikipedia.org/wiki/True_lemming). Since it is a new instance, please bear with any potential growing pains. I'm going to monitor the performance and scale accordingly, but let me know if you notice any issues. I'm hoping to balance cost/performance as much as possible. #### Updates * [2024-08-10 - Upgrade](https://lemmus.org/post/6887383) * [2023-12-22 - Upgrade](https://lemmus.org/post/2190260) * [2023-07-10 - Security](https://lemmus.org/post/68881) * [2023-07-04 - Instance](https://lemmus.org/post/32521)