LV1 on X4B Network Status

[Resolved] Software Upgrades

Mon, 10 Nov 2025 22:30:00 +0000

Update NY done. All Done. If you have any issues please do let us know.

Update Vegas done. Sorry that one was a bit less smooth.

Update Switzerland done.

We will be performing system software updates on Tuesday (AEDT).

It is expected that maintenance will take a couple hours per server. No interruption is expected however risk does exist.

We will update this announcment when maintenance is complete. Thank you for your patience.

[Resolved] Budget Line Disrupted (US-EAST)

Tue, 29 Jul 2025 10:14:47 +0000

Update No upstream updates at this time.

Upstream network for all three PoPs has a reported interruption in New York. New York has been routed around sucessfully according to our observations.

Eastern US may experience higher than expected latency until the situation is resolved.

[Resolved] Software Updates (Budget Line)

Thu, 17 Jul 2025 23:56:30 +0000

Update Complete

Update Started

We will be performing system upgrades on Friday. No interruption is expected.

Systems will be done in parallel with an estimated total risk of interruption period per server of 30 minutes.

If any interruption or unexpected event occurs we will update this incident.

[Resolved] Vegas and Luxembourg

Fri, 06 Jan 2023 00:00:23 +0000

We are scheduling maintenance (system restart) tomorrow for a quick patch. Interuption of less than 10 minutes is expected.

Complete

[Resolved] Vegas: Offline

Fri, 14 Oct 2022 18:25:00 +0000

Upstream Incident Report -

Short version:

Networks' been crapping itself for the past day. After 2 - 3 hard failures today we did an emergency JunOS downgrade and are cautiously optimistic that it’s resolved.

Longer/technical version:

For the past month or so we’ve been suffering a weird issue where random IP addresses would stop responding. The issue is that our router would lose/expire out the ARP entry for these IP addresses and wouldn’t attempt to learn them again unless forced to.

No amount of traffic getting sent to the IP externally (pings, random connection attempts, etc) would make the router issue an ARP “who-has” message. The only fix was to either ping the IP on the LAN (through our speedtest page for instance), forcing the router to see the ARP message on the network, or to login to the router and ping from there.

No amount of limits, tweaks, process restarts, etc, has helped with this. The only way to date has been the above methods.

This morning we had an issue where all users on 209.141.x.x dropped offline. These subnets had been moved down to our new Cisco core switch to help resolve the issue. Our Juniper router dropped the ARP entry for the Cisco switch, killing BGP. After some time it re-learned then entry reestablished connectivity, bringing those subnets back online.

This happened three times in the past 24 hours, each outage lasting anywhere from 5 minutes to 30 minutes.

Las Vegas is the only location suffering this issue. The key difference between all locations being that Vegas was running JunOS 20.4, where as everywhere else is 18.2~4. In JunOS 20.x they made changes to ARP to allow policing. While we did change this policy, it didn’t help us any.

As of an hour ago we performed a downgrade to JunOS 18.4. It obviously went terribly. After some research we found the configuration entry causing issues (our 100G ports kept channelizing) and fixed it, bringing things back online.

As of now, ARP’s are resolving near instantaneously, even for new IP addresses. We took this as a good time to change the rest of our ports to 100G, giving us a 400Gbit interlink between our router and core switch.

We’re cautiously optimistic that this issue is now resolved.

We thank you for your patience.

Upstream reports that it is conducting emergency maintenance to downgrade the router firmware in an attempt to resolve issues.

-–

Offline again, ticket has been sent.

-–

Upstream restored services.

The underlying issue (router bug) is still under investigation.

-–

Vegas upstream is offline. We have ticketed them and contacted staff members. We await response.

[Resolved] Budget PoPs offline (upstream)

Tue, 15 Feb 2022 00:07:29 +0000

Investigating - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.

Update - Upstream resolved.

[Resolved] LV1: Outage

Sun, 05 Dec 2021 23:37:00 +0000

Investigating - We are aware of an outage in Vegas, we reported to datacenter and are awaiting resolution.

Update - Local engineer on his way to datacenter to replace server PSU

Update Fixed

[Resolved] Budget: Issue continued.

Sat, 02 Jan 2021 03:49:26 +0000

Upstream - Confirmed issue with them.

[Resolved] Budget: Incident

Wed, 30 Dec 2020 04:20:00 +0000

Upstream response - Confirms issue on their end.

Upstream response - Upstream is experiencing DDoS attacks on unprotected resources they operate (extortion).

Attack looks to have ended for some hours. Marked as resolved.