Hello,

Three days ago, we suffered a temporary outage that lasted around 30 hours. This was due to a hardware failure at the provider level and mostly out of our control. We apologize for any inconvenience caused by this.

At around 4:00 PM, March 17th, our primary storage device was disconnected. This resulted in the 404'ing of worlio.com, the Worlio Git server, and Worlio Mail. During this time, our XMPP was still online and I received messages from users over XMPP notifying me that something was wrong by 4:20 PM. Our other sysadmin was asleep during this time.

At 6:00 PM, I logged into the remote shell and rebooted Worlio to clear any temporary hiccups. When the server was brought back up, the drive did not reattach and we lost all other services. Checking with other BuyVM customers confirmed that this was an issue affecting multiple BuyVM customers in Las Vegas, and was not exclusive to our site like the previous instance of this happening.

At 4:30 PM, March 18th, we received a reply from BuyVM explaining that the entire storage cluster Worlio lives on failed and our data was in the process of being migrated to a new one. No data was lost in the incident. Due to the large volume of customer data that needed to be transferred, we would not complete migration until 10 AM the next morning.

One time is an accident. Two times is a blunder. We hope this will not happen a third time.

We want to acquire fallback hardware to maintain uptime during outages such as these. However, it is not currently within our budget to do so, and may not be for a while. If you would like to donate to us, this would be a great reason for doing so.

We are also in the process of moving over some less closely related services like Worlio Git upstream to Canithesis servers to take some load off and allow Worlio to run more smoothly. When we finish this, we are going to test open registrations on Worlio Git. If things go well, we'll leave registrations open for anyone to use our Git server for the foreseeable future.

Thank you for using Worlio.

- Bonk

---

EDIT 03/23: git.worlio.com is now git.canithesis.org. All HTTP links and Git remotes should redirect automatically. SSH remotes will need to be updated to point to the new domain manually.

EDIT 03/24: Git emails are working.

not true i turned off the server to mine crypto

It's unfortunate that you give us status reports of your servers, but that you are unaware of the constant server outages that have been taking place throughout March and April.

Right now, your XMPP server is down, but the Worlio BB server is up.

The XMPP server has been down for over half of March.

Dirk

Our XMPP servers are online and operational and have been this time, as are the rest of our services.
http://status.worlio.com/

Well, I suppose that is both good news and bad news.

It's good news for you that you are keeping your servers running.

But what this implies for me is that in my part of the world, in the Canadian province of Quebec, the ISPs have removed some of your sub-domains from their DNS servers. Because, my XMPP client keeps telling me "Server not found."

Dirk

I've also been noticing problems and have received a few reports directly, we're looking into it but we don't have a lot of information right now. I would suggest using a proxy to connect to Worlio for the time being and/or editing your host file with our static IP.