← Feed

Posts

• View post

  Thinking about becoming hard like a godforsaken mountain—immovable, unknowable, yet undeniably there.

• View post

  :bc: Attention Beige Party-goers! :bc: I need to make a couple more tweaks to the web server, which will require taking it offline briefly. Any downtime should be less than a minute. Beige-bless :bb:

• View post

  Sorry folks, was updating docker in preparation for the 4.6 tomorrow and it ended up taking the database offline for several minutes. We should be good now.

• View post

  Going offline for Mastodon upgrade. See you in a couple hours...

• View post

  :bc: Attention Beige Party-goers :bc: I have completed the upgrade and we are now on version 4.6.2. There should be no additional downtime tonight. Beige-bless :bb:

• View post

  @OohOkayKay @BaronessWinter We have been having issues at least since the 11th. I think what happened was I was running too many threads for the web server so it was eating up a lot of memory, then I tried running the reindex for elasticsearch and the combined processes were occasionally using 100% of available memory and grinding everything to a halt. That should hopefully have been fixed last Sunday when I did the 4.6 upgrade. The only time it should have gone down this week was tonight for th...

• View post

  :bc: Attention Beige Party-goers! :bc: Welp, looks like the days of me being able to update docker and have the site go down for just a few seconds are over. Since I've already started, I'm going to go ahead and update the other servers. You may see the site go down for a couple of minutes. I will post another announcement when I'm done. Beige-bless :bb:

• View post

  :bc: Attention Beige Party-goers! :bc: Sorry for that unplanned outage. We were getting overwhelmed with Sidekiq jobs again, the result being the Beige Bubble effect where posts on the timeline are delayed by several hours. In trying to add some more DB connections for SideKiq to use, I found out that pgbouncer likes to switch its IP around when you restart its docker container. Postgres was not configured to accept that new IP address, so the whole site went down. Oops. Anyways, we'...

• View post

  :bc: Attention Beige Party-goers! :bc: Looks like we had another unplanned outage. This time the culprit was the hot standby db server. At some point it stopped responding to the production server and when that happens, the prod server starts caching all changes locally until the standby comes back up. If it doesn't come back up, that cache grows until it fills the entire hard drive is filled up, which is what happened this morning. We are back up and running now. I need to check out...

• View post

  :bc: Attention Beige Party-goers! :bc: Looks like the issue was that the Sidekiq Ingress queue was getting overwhelmed. That meant that we were experiencing a delay of a few hours for fresh toots to arrive from outside the instance. In response I added additional Sidekiq worker threads to process the ingress queue. It took about an hour, but we have cleared out the backlog and everything should be up to date now. I have also upgraded us to the latest patch version of Mastodon, which should f...

• View post

  :bc: Attention Beige Party-goers! :bc: We're having some issues with the instance at the moment. I am looking into it now. There is also a new patch version of Mastodon that should fix some bugs that have cropped up since v4.6. Since we are having site issues anyway, I'm going to go ahead and update to the latest version now. You may see the site go down briefly as I restart the docker containers. Beige-bless :bb:

• View post

  :bc: Attention Beige Party-goers! :bc: Mastodon v4.6.2 has been released and it contains a security patch for a critical vulnerability involving Docker, so I am going to go ahead and do the upgrade tonight. I'm not going to set a specific maintenance window because the site should only go down briefly when I restart the docker containers, though it might go down several times while I'm performing the upgrade. I will post another announcement once the upgrade is complete. Beige-bless :...

• View post

  :bc: Attention Beige Party-goers! :bc: Well, that wasn't so bad. The Mastodon upgrade was successful and we are now on version 4.6.0. Like with the 4.5 upgrade, I'm having to reindex elasticsearch again, so test search will probably be down for a day or so while the indexes get rebuilt. Everything else seems to be working properly, but if you notice anything that looks broken, please let me know. Thanks, and Beige-bless :bb:

• View post

  :bc: Attention Beige Party-goers! :bc: When it rains it snows! Now that I've finally gotten us on version 4.5, they've gone and released 4.6. The good news is that, after reviewing the release notes, I don't foresee any major issues with this upgrade, so I am going to schedule it for this Sunday night, in our regular time slot. This is a minor release, which is actually kind of major when it comes to the upgrade process so I am going to schedule a four hour maintenance window again...

• View post

  :bc: Attention Beige Party-goers :bc: I've gotten this question from several people and just ran into the same issue myself, so I thought I'd address it here: Yes, I am aware that the site has been going down randomly since the last upgrade. In most cases it seems to resolve itself within a couple of minutes, but just now I logged in and found that the hard drive was completely full. I also found the reindexing process for elasticsearch was still running, so that might have something t...

• View post

  I know a lot of people were waiting for quote toots but one thing that immediately irritated me about 4.5 is that boosting now takes 2 clicks by default. That's a 50% reduction in boost efficiency! If you are like me and find that kind efficiency hit unacceptable, you can enable quick boosting in user preferences. Give that baby a click and keep on boosting like a champ!

• View post

  :bc: Attention Beige Party-goers! :bc: I have completed the 4.5 upgrade and bumped us up to 4.5.11. Text search broke again so I am reindexing. That could take a day to complete. Other than that, everything seems to be working properly. If you notice anything that's broken, please let me know. Beige-bless :bb:

• View post

  :bc: Attention Beige Party-goers and prospective party-goers! :bc: Due to a surge in fake account sign ups, I am closing new registrations for now. If you would like to sign up for an account, please contact me directly or have a party-goer recommend you. Thanks, and Beige-bless :bb:

• View post

  :bc: Attention Beige Party-goers! :bc: Phase 2 of server maintenance was successful and we are now ready to proceed with the 4.5 upgrade this Sunday night, June 14 - 15. The maintenance window for the upgrade will be in the regular time slot, but I'm extending it to 4 hours in case I need extra time. The maintenance window will begin at 3AM UTC on June 15th and continue until 7AM. In the Western Hemisphere that's 11PM EDT on the 14th until 3AM on the 15th. Expect the site to go down se...

• View post

  :bc: Attention Beige Party-goers! :bc: I had been planning to do some more server maintenance tonight to prepare for upgrading to Mastodon 4.5 next Sunday, the 14th. I need a little more time before I perform the maintenance I was planning for tonight so I will be moving that to Tuesday, June 9th. The Mastodon upgrade on Sunday, June 14th will continue as scheduled, and will not be affected by this delay. The next maintenance window will begin at 3AM on June 10th and continue until 5AM. If you...

• View post

  :bc: Attention Beige Party-goers! :bc: GREAT NEWS! After much struggle, I have finally settled on a Postgres migration strategy that I'm comfortable with. It's going to mean 2-3 hours of downtime but it has the advantage of leaving the old database intact so I can roll back in case something goes horribly wrong, which is nice. I did a dry run last night and everything went smoothly, so I feel good about moving forward with the actual database migration now. So, what this means...

• View post

  @woof@aria.dog @christymarx This issue has been resolved. The images in question have been taken down. Stop spamming this thread or I will be defederating you for harassment.

• View post

  Ok, going offline for few hours for server maintenance. See you on the other side.

• View post

  :bc: Attention Beige Party-goers! :bc: We are currently having issues with the full text search. This is because migrating the database requires reindexing elasticsearch. We've got a lot of toots in this baby so a full reindex can take a day or more. I just checked and the process is still running and the progress bar still has a ways to go. I'll post another update once the reindex process is complete. Beige-bless :bb:

• View post

  :bc: Attention Beige Party-goers! :bc: It took a little longer than expected but the database server migration was successful. We are now running Postgres 17, which means we are one step closer to the Mastodon 4.5 upgrade. The next maintenance window is scheduled for next Sunday around the same time. I will send out more details in a separate announcement later this week. Beige-bless :bb:

• View post

  ...and we're back! I think. Sidekiq queues are backed up so it might take a little bit for our toots to work their way out into the world. I also need to reindex elasticsearch so full text search might not work for a few hours. Otherwise, everything appears to be working. We have successfully migrated to the new database server, and are on track for our 4.5 upgrade.

• View post

  This is my candidate for best headline from the US presidential race so far:

• View post

  Such a relief to learn that all of my problems are in my head. The thing that I am trapped in and can never escape.

• View post

  Someone's got a case of the Thursdays

• View post

  :bc: Attention Beige Party-goers! :bc: I'm still getting reports of media loading slowly. It seems to happen more around the evenings (in the US) and on weekends, which is when traffic is tends to be higher. The weird thing is that I have not been able to catch the problem in the wild. When I check a sample of images both local and remote, everything seems to be loading fine. What's been reported is images loading fine on the TL but when clicking to enlarge, the images load at d...