Master Server Maintenance Thread

[Anyder]

Upcoming reboot on 9/8
Sep 06 2016 11:31:29 PM PT   Between 12:30am and 4am local time for your service on Thursday, September 8, we plan to gracefully shut down your VDS, upgrade the machine hosting it to an updated version of Xen/Linux, reboot the machine, and bring your VDS back online. We expect for this process to cause approximately 30-60 minutes of downtime for your service.

Because this maintenance will require a full reboot, we recommend that you configure any services which need to always be running to start automatically when the VDS is booted.

We do not anticipate problems due to this update, but to be safe, we recommend that you externally back up any critical (irreplaceable) files before the maintenance event and before any other service adjustment.

This Xen upgrade is required because of an important Xen vulnerability. We are rebooting all of our VDS-hosting machines that are running anything but the latest Xen release the same morning.

[Led]

MS is back up now after the NFO maintenance   

[strohhut]

There are problems in swbf2 after the maintenance.

From around 8 servers I guess, now there are only shown 2 servers now.
Approximately after we will have played one map now on a server that still runs, there will be only one left.
Because the last one just crashed after one map that was still running.

I am in the spy gaming team with server access. I restarted all servers, servers still do not show up.


I tried hosting a server from my computer, the server wont start.

So something is wrong there right now.
Just to let you know!

[William]

They aren't loading at all for me. Hopefully it will fix itself

[oldsnake]

the ps2 servers are working. here is a wireshark capture of the stuck pc swbf2 server 
https://mega.nz/#!bcxjxIja!F-xGyO3VijwF0OJjFuOxd1jjWKJqNabEKxFUm6DwwbQ

[Led]

I will reboot the MS and restart everything.

[strohhut]

Yo nice man.

Servers show up + running againg.

Thank you.

[William]

Yo nice man.

Servers show up + running againg.

Thank you.

Yah

[Anyder]

Brief problems due to several DDoS attacks
Sep 18 2016 07:12:52 PM PT   

Extremely large attacks against multiple servers run by a single customer in Chicago led to brief packet loss events shortly after 9pm CDT. These attacks led to immediate null-routes, but the impact was still significant because of the sheer amount of traffic that this botnet generated (well over 100 Gbps).

We are continuing to watch for further attacks and will continue to process any that we see.

Additionally, we are in the late stages of adding another 10 Gbps of upstream capacity in Chicago. We expect it to be online within the next few days, following additional testing.

[Anyder]

Problem with an upstream in Chicago

Sep 23 2016 12:41:37 PM PT   One of our upstreams in Chicago (GTT) appears to be experiencing a partial outage in Chicago. We are currently investigating. Clients reaching us, or being reached through, GTT may be experiencing packet loss or a loss of connectivity entirely right now.

[Anyder]

Upcoming upstream bandwidth maintenance early 10/18

Oct 12 2016 10:58:36 PM PT   
Between 1am and 8am CDT on Tuesday, October 18, one of our upstream providers (Telia) has informed us that they will be doing maintenance on our connections. This maintenance may cause us to be unreachable by some clients for approximately 15-30 minutes at two times during the morning, they tell us.

[Led]

Upcoming Xen upgrade with reboot @ 1:30am CST on Nov 21
Nov 18 2016 06:05:10 PM PT    We are planning to reboot the machine hosting your VDS at approximately 1:30am CST on November 21, 2016, in order to upgrade it to the most recent version of Xen. This is necessary because the new version includes a fix for a newly-discovered, critical Xen vulnerability.

For the reboot, your VDS will need to be shut down, then booted back up. We estimate that downtime will last 15-60 minutes.

Windows Server 2012 R2 still has a bug that causes VDSes running that OS to sometimes experience boot failures (but no problems at any other time), particularly when there is a high I/O load for the machine. If you have a Win2012 R2 VDS and it does not come back online after the maintenance on its own, please wait a bit and then try rebooting it again through the Server Control page."

Problems due to system reboots are exceedingly rare, but to be safe, we recommend that you remotely back up irreplaceable files before the maintenance event.

[Led]

I will restart the master server when I wake up.   

[Led]

bump.  only SWBF1 servers are running at the moment.  After the maintenance, I will start the SWBF2 and PS2 servers.

[Led]

Update @ 1:26am PST on 11/21: The start of this maintenance had to be delayed by 10-30 minutes for many machines while we worked out a few kinks in the process and did some last-minute testing. Most machines were back online within 30 minutes of the trigger, falling within the 15-60 minute estimate given in the post. The next time that an emergency maintenance is required, though, we will be more specific about what we mean by 'approximately' in reference to when it is planned to start, to avoid any confusion, and we will include significant padding to help compensate for unforseen events.

We changed our procedure so that VDSes were saved to disk, then restored, instead of fully shutting them down, after we determined that a bug with Xen which previously prevented this had been fixed. If the save operation worked properly, this means that your VDS should have only seen network connectivity loss, instead of a full reboot. You may need to check the clock in your VDS to make sure that it is synced properly, as this could have caused it to be knocked off. (If the save/restore didn't work properly,