Master Server Maintenance Thread

[Led]

12:30 PM central time 6/26/16

edit:  maintenance complete, let me know if there are any issues

[William]

Were there any issues before? I found that the servers loaded a lot faster myself

[Led]

Were they faster after this maintenance?

Kalle has suggested that they MS get restarted every 20-30 days.  This is the first restart that I have done in about 30 days.

[TheGangstarTY]

Were they faster after this maintenance?

Kalle has suggested that they MS get restarted every 20-30 days.  This is the first restart that I have done in about 30 days.

Is all I noticed was the TPS went up from 30 to 40. Although it did seem like better connection as I managed to rack up a lot more kills than usual

[Led]

tps is set at 40 for the ICW6, not related to MS restart

[William]

Were they faster after this maintenance?

Kalle has suggested that they MS get restarted every 20-30 days.  This is the first restart that I have done in about 30 days.

Doesn't feel any faster than it was before, so idk

[Anyder]

Performing some firewall tasks to see if we can somehow block DDoS attacks. A friend of mine and I are testing this right now, if anyone has any problem in the meantime, post here --

EDIT: Finished it!

[Anyder]

Switch reboot
Jul 15 2016 04:02:32 PM PT

At approximately 5:49pm CDT, our primary aggregation switch in Chicago rebooted itself, causing a disconnection event for all customers at that location.

This is the second time this has happened. In both cases, we have been unable to determine a cause of the switch failure.

In response to this, to prevent further service interruptions, we will be working to replace this switch in the near future. We will post a maintenance event when we have final information on when the maintenance will be performed.

[TheGangstarTY]

Is there any Maintenance going on with SWBFSpy lately? There's next to no servers across both games.

[Anyder]

Is there any Maintenance going on with SWBFSpy lately? There's next to no servers across both games.

Unless we post something, there are no maintenance events going on.

[spoiler]
Subject and date   Description
(No maintenance events recorded)


Subject and date   Description
(No events recorded)


Subject and date   Description
(No bandwidth events recorded)

[/spoiler]

[Led]

Upcoming router maintenance early 8/5
Aug 04 2016 12:17:19 PM PT    Between 3am and 5am CDT on Friday, August 5, we plan to update the OS on one of our routers in Chicago, which will require a reboot. We expect for this to cause approximately 10 minutes of connectivity loss for your service at this location.

We are performing this update because the new software release should offer higher performance than the old one, which will make the router able to better-handle particularly high-packet-rate DDoS attacks. We have seen several extremely large DDoS attacks against a customer in Chicago today that have saturated all of our upstream links, and would have needed to be null-routed no matter what, but they caused some additional impact because this router could not process all of the traffic internally.

If there is a problem with the new software release, we will roll back to the older version.

[Led]

Location down
Aug 08 2016 02:33:53 PM PT    Our primary aggregation switch in Chicago appears to have crashed, causing the location to go offline. We are currently investigating.

We already have a new switch on order to replace this one, since it has crashed twice in the past.

Update @4:43pm CDT: We are waiting for an on-site technician to arrive at the cabinet and physically survey it to determine what has happened. The main possibilities are that the switch has failed or that the PDU that it is plugged into has failed. If it is the latter, it should be a quick fix once the technician arrives; if the former, it may take a little longer for us to rearrange cables.

The replacement switch that we ordered is higher-end, with additional redundancy, including redundant PSUs.

[Anyder]

Intermittent packet loss over Telia


We have been noticing intermittent packet loss over one of our upstreams in Chicago (Telia) tonight, on the inbound from some endpoints (from New York/Ashburn, at minimum). We have contacted Telia to ask them about this. Clients whose ISP reaches us via Telia may notice skipping or other performance problems with services in Chicago.

[Led]

Master Server Maintenance Has commenced.  and is now over, seems to work 

[Anyder]

Charter packet loss

Aug 25 2016 01:29:49 PM PT   We have received reports from Charter customers that they are seeing intermittent packet loss to our Chicago location. The data we have seen so far suggest that the problem is occurring between Charter and Voxel (Internap's peering network). We have emailed Internap to ask them to investigate, and we have also emailed Charter.