X4B Network Status

[Resolved] Outage CH

Wed, 04 Mar 2026 09:20:00 +0000

Server in Switzerland appears offline. Upstream is investigating.

[Resolved] Dallas migration

Mon, 23 Feb 2026 15:55:00 +0000

Update Routes have been restored

Update Routes have been pulled. Maintenance starting.

Upstream provider has requested that we move our servers to a different cabinet. Server will be moved via offline imaging by their technicians.

We do not know exactly how long it will take but we expect a couple hours. We will be around to check services before restoring routes.

[Resolved] Brazil Maintenance

Mon, 09 Feb 2026 03:50:00 +0000

Update We have confirmed the upstream techs work and restored routes.

Upstream provider has requested that we move our servers to a different cabinet. Server will be moved via offline imaging by their technicians.

We do not know exactly how long it will take but we expect a couple hours. We will be around to check services before restoring routes.

[Resolved] Hardware Replacement

Wed, 04 Feb 2026 01:00:00 +0000

Update Routes have been restored. If you encounter any issues please do let us know

Update We are still working on this. Currently running behind schedule. Hoping to have this done tonight.

Update Currently waiting on upstream technicians to do their side. We are hoping for everything to be done tonight. Once upstream side is done we will complete testing & QA and proceed accordingly.

Update We are still working on this. Currently running behind schedule. Hoping to have this done by 7pm tonight.

Upstream has requested that we move our hardware to a new area of the datacenter. To facilitate that we are going to experience an interruption. We have been requested to do this via a system migration.

To facilitate this we will be taking Singapore routes down from midday AEST on Wednesday. We estimate that it will take 4-5 hours to complete.

[Resolved] London GTT packet loss

Tue, 09 Dec 2025 02:28:00 +0000

Update Looks to be clearing. Routes are being restored.

Packet loss has been detected on routes ingressing via GTT. We are temporarily removing london from the mix while we await GTT resolution.

[Resolved] Routing issues

Thu, 04 Dec 2025 01:36:36 +0000

Update All servers functional. Appologies for the resolution time, this was an incredibly difficult issue to resolve that required IPMI access to each server.

We are aware some locations are having issues currnetly. We have disabled the identified problematic locations.

We are working on it.

[Resolved] AMS issue

Thu, 20 Nov 2025 03:17:00 +0000

Update issue resolved

We have pulled routes from AMS while an indicident affecting the region is investigated

[Resolved] Software Upgrades: Anycast

Mon, 17 Nov 2025 22:30:00 +0000

Update as of 11:39AM all done.

Update as of 11:15AM

All servers other than LON are complete pending final review.

Hopefully will be done before lunch.

Update as of 10:15AM

SYD, SG, LAX are complete pending final review
JP, MIA, DAL are currently in progress

Update As a reminder this is starting tomorrow at 9:30 AM AEDT. We expect it will take approximately 1 hour per server (for all 11 servers)

It is expected that maintenance will take approximately 1 hour per server. No interruption is expected however risk does exist. We will be updating a couple servers at a time.

There is the chance that we will need to do the update of some servers the following day. However the plan is to get all the servers done in one day if possible.

This is the same update that has already been performed on the budget line.

We will update this announcment when maintenance for each region is complete. Thank you for your patience.

[Resolved] HTTP Log information

Wed, 12 Nov 2025 05:07:17 +0000

Update system is fixed. Appears to be a bug in the version of vector deployed at the edge. We have resolved the situation for now and will look into a software update and/or fix tomorrow.

We are investigating an information collecting HTTP information (requests, error logs).

[Resolved] Software Upgrades

Mon, 10 Nov 2025 22:30:00 +0000

Update NY done. All Done. If you have any issues please do let us know.

Update Vegas done. Sorry that one was a bit less smooth.

Update Switzerland done.

We will be performing system software updates on Tuesday (AEDT).

It is expected that maintenance will take a couple hours per server. No interruption is expected however risk does exist.

We will update this announcment when maintenance is complete. Thank you for your patience.

[Resolved] Maintenance

Fri, 07 Nov 2025 00:30:00 +0000

Update complete

We will be performing upgrades of our BGP software. During this upgrade we will restart all sessions. No interruption is expected however a short routing blip is possible.

We will do PoPs in sequence

[Resolved] NTT Brazil

Wed, 05 Nov 2025 00:10:00 +0000

Update Resolved. Issue was related to outgoing routes via Algar.

Update Issue appears to be on NTT’s side. Ticket is open with NTT.

Investigating - We are investigating a potential issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.

[Resolved] Brazil outage

Thu, 09 Oct 2025 22:29:45 +0000

Update - Upstream have confirmed the issue as a power issue affecting core router. It is being worked on.

[Resolved] Ashburn Emergency Maintenance

Tue, 02 Sep 2025 13:00:29 +0000

Upstream has just notified us

Please be advised that emergency network maintenance is scheduled at our Ashburn location to perform an Core router hardware upgrade aimed at improving network redundancy and stability.
Maintenance Window:
Date: Tuesday, Sept 2, 2025
Time: 6:00 AM – 10:00 AM PDT
Duration : - 10-20 Minutes

Expected Impact:
A brief 10–15 minute service interruption may occur during the upgrade process. We will work to minimize downtime and restore services as quickly as possible.
We appreciate your understanding and patience as we implement these enhancements to strengthen our network infrastructure.

We will route around the maintenance for anycast product lines.

[Resolved] Reduced quality of EU routes

Wed, 27 Aug 2025 02:02:17 +0000

Update issue was found upstream and resolved. Upstream will update their internal documentation to prevent recurrance.

Update Appears to be a GTT fault. Fault ticket has been opened.

Update Investigation continues upstream. This issue only appears to affect Premium Anycast customers. Standard Anycast routes seem unaffected.

Investigating - We are investigating a potential issue that might affect the uptime of one our of services. It appears to be a bug in the upstream providers routing configuration. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.

[Resolved] Recaptcha Depreciation

Tue, 12 Aug 2025 04:29:29 +0000

We are depreciating Recaptcha & recaptcha fallback for validation pages. Currently recaptcha is in use for less than 0.01% of all verification page requests so far in 2025 and with the exception of an attack utilizing a captcha solver (a growing risk that will be mitigated by removal) was similarly represented in 2024.

If any customer requires a no-javascript verification page they may contact us and we will see what we can do, however its our current belief given the analytics gathered from user agents and usage of the default PoW verification system that this is below a significant number.

We will of course be adding an appropriate message for clients without javascipt enabled when Recaptcha fallback & support is removed before the end of 2025.

[Resolved] Budget Line Disrupted (US-EAST)

Tue, 29 Jul 2025 10:14:47 +0000

Update No upstream updates at this time.

Upstream network for all three PoPs has a reported interruption in New York. New York has been routed around sucessfully according to our observations.

Eastern US may experience higher than expected latency until the situation is resolved.

[Resolved] Cloud Service outage

Sun, 27 Jul 2025 11:08:06 +0000

Update No update from Akamai yet but we are seeing Kubernetes nodes that were offline return to function slowly starting around 5 minutes ago.

Update We currently await Akamai to fix their internal networking. 4/12 of our cluster nodes are currently online. The panel is up but slow. Some non essential services that rely on inter-node communication are degraded.

Update Panel is currently very slow due to 70% of cloud nodes being offline currently. We await Akamai’s resolution to their datacenter outage.

We apologise for the inconvenience and are doing everything we can to return to 100%. Even including moving our services to another vendor.

If you would like to follow the upstream issue you may.

Update We will be keeping this incident open for now as we are still seeing substantially lowered performance in the region and half of our cluster nodes are still unavailable. The panel is however online.

We continue to monitor the upstream issue and await resolution.

Update We will be keeping this incident open for now as we are still seeing substantially lowered performance in the region and some of our cluster nodes are still unavailable. The panel is however online.

Update Holy Smokes Akamai that was a long one. Services are restored. We are going through and checking everything in line with our DRP.

Upstream Update - Our SMEs continue to work bringing our services back online, and we will provide an update as progress is made.

Upstream Update - Identified - Our team has identified the issue affecting connectivity in our US-EAST (Newark) data center. The issue is related to heating/cooling complications in the data center due to a power outage . The power outage has been fixed and we are working quickly to bring our services back online, and we will provide an update as soon as the solution is in place.

Upstream currently has an outage affecting their US-EAST PoP where our core cloud (web panel, data collection, etc) is homed. This does not affect customer services are all essential services (BGP, route validation, mitigation analysis) is multi-homed. Email delivery (i.e on DDoS attack) will not be available until service restored.

“Investigating - Our team is investigating an emerging service issue affecting All Services in US-EAST (Newark). We will share additional updates as we have more information.”

[Resolved] LUX: Migrating to Switzerland

Sat, 26 Jul 2025 14:06:45 +0000

Luxembourg services are moving to Switzerland. Our upstream in the region is co-ordinating the move.

No substantial interruption is expected for the move as we will do a live migration.

Appologies for the short notice. While we knew this was planned we were not informed until just now of the exact details.

[Resolved] LUX interruption

Sat, 26 Jul 2025 13:18:31 +0000

Update Looks like theres an interuption currently. No information yet provided from upstream on current or previous interruptions.

Update No interruptions today. Upstream really messed up communication on this one. They will be emailing everyone over the next couple of days at which time we will know more. The next couple of days may have some network noise but is generally expected to be fine.

Update. No information currently from upstream. Will leave open as more interruptions are posssible during the night.

Update Appears down again.

Update Upstream reports resolved through early moving our hardware to Switzerland.

Luxembourg network issues affecting Budget Line Luxembourg services currently.

No information currently on the cause or status.

Services will return online when upstream resolves.

[Resolved] Software Updates (Anycast)

Tue, 22 Jul 2025 23:00:00 +0000

Update Complete

We will be performing system upgrades on Wednesday. No interruption is expected.

Systems will be done in parallel in groups with an estimated total risk of interruption period per server of 30 minutes.

This is the same upgrade that was carried out on the budget line last week without interruption.

If any interruption or unexpected event occurs we will update this incident.

[Resolved] Software Updates (Budget Line)

Thu, 17 Jul 2025 23:56:30 +0000

Update Complete

Update Started

We will be performing system upgrades on Friday. No interruption is expected.

Systems will be done in parallel with an estimated total risk of interruption period per server of 30 minutes.

If any interruption or unexpected event occurs we will update this incident.

[Resolved] Autossl issues

Fri, 11 Jul 2025 09:39:54 +0000

Update - Looks resolved. We will continue to monitor.

Investigating - Issue seems to have been limited to 13/37k certificates we generate. We beleive this was caused by an increase in errors due to a large number of bad certificate generations being attempted.

We have made some changes and are monitoring. Please let us know if you have any issues.

[Resolved] AMS: Scheduled Maintenance

Thu, 05 Jun 2025 08:00:00 +0000

Our upstream datacenter has notified us today of upcoming maintenance in Amsterdam.

Network engineers will be upgrading top-of-rack switches and reworking some of the network topologies for a more robust and stable overall network in the server cabinet.

Though the outage is not expected to last more than 5 to 10 minutes, the work window will be longer as our team will test and ensure connectivity is stable before completing the maintenance.

Maintenance Details

Day/Date: Thursday, June 05th, 2025
Start Time: 6:00 AM (PDT)
End Time: 8:00 AM (PDT)
Outage Estimated Duration:  5-10 mins
Effect: Short Outage
Facilities: AM11
Device Description: 0818.16.Black

[Resolved] AMS Interruption

Fri, 30 May 2025 13:58:47 +0000

Update Hardware has been replaced upstream and we have restored routes. Sorry for any interruption over the past 5 minutes we had a limited window for restoration while under the eye of upstream NOC today.

Update Upstream confirms that the issue is taking longer than expected to resolve. We continue to route around the issue.

Upstream has confirmed an interruption in AMS affecting firewalled TCP traffic. We have pulled routes from the region until incident confirmed over.

[Resolved] Dallas: Network Maintenance

Mon, 19 May 2025 09:00:00 +0000

Update Maintenance complete. Routes restored.

Update Pulling network routes.

Upstream network is performing maintenance on our network slice. We will take this time to do some long delayed network configuration changes brining this location into sync with other locations configuration.

We expect 15 minutes of interruption in Dallas during this time. We will pull routes from the region in advance of the maintenance. We will update on status here.

Should there be any issues we will keep routes pulled until resolved however interruption is expected to be minimal.

[Resolved] Issues with Akamai

Sun, 30 Mar 2025 11:29:49 +0000

Update Although we are yet to receive a response the issue has been resolved now for nearly 6 hours. As such we will mark as resolved.

Some services are affected by a service interruption with Linode (likely related to https://status.linode.com/incidents/x14zys0wddxk). We opened a ticket 5 hours ago when it appeared as a minor issue on our monitoring however are yet to receive a response. Certain dashboard features are interrupted during this time.

[Resolved] Dallas: Potential issue with Cogent

Mon, 10 Mar 2025 09:40:07 +0000

Update Upstream has installed a workaround and is looking to drop cogent in the near future. We have been able to restore routes to Dallas PoP. If you encounter any Dallas issues please let us know.

Update We still await upstream fix

[Resolved] Block Storage outage

Thu, 05 Dec 2024 07:05:41 +0000

Update Panel restored.

An outage at our cloud service provide is affecting the web panel.

No customer services are affected. Panel access will be restored when upstream resolves issue.

[Resolved] LA: Fault

Thu, 24 Oct 2024 09:42:25 +0000

Update Routes have been restored as Cogent have resolved the issue.

Update Upstream response: They are working with Cogent and hope to have a solution in a couple days. We will keep the routes removed from LAX until fixed.

Note We are aware of a fault upstream in LA. While we wait on resulution we have pulled all routes from the region.

[Resolved] Paypal payments delayed

Tue, 15 Oct 2024 00:53:48 +0000

Update - Still yet to receive any solution from Paypal. Only an acknowledgement of suspected fault on their end.

We are processing payments through our workaround as per normal. However there is still some element of unknown until paypal resolve the actual issue (which appears to be issues specifically with our x4b.net domain name)

As this issue does not have any known customer impact we are going to bring it to a close now.

Should you experience any delayed payments (either immeidately, or after 3 hours when the new correlation script can recover any missed payments) via paypal let us know and we will sort it out.

Update - Still yet to receive any solution from Paypal.

Our Transaction log check job remains in place. Having discovered a suspected workaround to the Paypal IPN system bug have implemented a workaround that appears to enable IPNs to be sent again. We will continue to run both.

Update - Still yet to receive any solution from Paypal.

We have implemented a payment check job which is able to import payments made once they are available via Paypal REST api (up to 3 hours). We understand this is inconvenient however at this time it is what we have available to us.

Paypal’s IPN system has abruptly stopped communicating with us and we await a serious response from Paypal on this matter.

Stripe (Credit Card) and Crypto payments are unaffected. If you need to make an urgent payment we strongly recommend using Stripe until this matter can be resolved.

Investigating - Still yet to receive any solution from Paypal.

Transactions are being manually processed in a priority queue. Open a support ticket if you need manual processing urgently.

We are working on a temporary solution to reduce the delay to ~5-10 minutes while we wait on a fix from Paypal.

Investigating - We are aware of delays processing paypal payments. We have contacted Paypal merchant support for a fix and are awaiting a response.

If you need a payment manually processed open a support ticket including a screenshot of your payment and we will get to it ASAP.

[Resolved] Lux: offline event

Tue, 08 Oct 2024 14:37:17 +0000

Update - Upstream router had a major issue interrupting services to all in Luxembourg.

[Resolved] Ashburn offline

Fri, 23 Aug 2024 02:10:00 +0000

Update We have made adjustments to the VLAN routing and have restored service in Ashburn.

Zayo maintenance on the dark fiber within the network is interrupting service. We are working on a temporary fix to re-route currently.

Routes have been pulled during the issue.

[Resolved] Sydney: routes pulled

Thu, 11 Jul 2024 13:25:03 +0000

Update - We have been informed that the replacement parts for the failed upstream router have arrived. They will be getting installed on Friday.

[Resolved] UK Migration 04/06 - 05/06

Tue, 04 Jun 2024 10:10:22 +0000

Update Routes are live. Migration is complete.

Update Server has been installed in the new DC. Just waiting on some network updates to be made before routes can be made live.

Update Server de-racking has begun. Due to the risk of reboots as fibers are moved we have decided that we will be removing routes until the hazard is cleared. Routes have already been removed in advance of our sever being de-racked.

Our server will be getting physically moved to a new datacenter between these dates. This will help us combat issues observed over the past few months including:

- Increased temperature within the cage our server is in

- Replacement of a failing switch (this has been responsible for a few 5-10s per week interruptions over the past month)

Expected interuption is likely just a couple hours, however depending on when we get scheduled we may restrict routes before / after depending on the availability of our technicians to verify / prepare. We are relying on remote hands and a significant number of servers are being moved during this time so we do not at this stage know our exact time.

We will aim to revise the scheduled time however to reduce the range closer to the event.

[Resolved] Dallas storm

Sun, 26 May 2024 12:15:00 +0000

[Resolved] UK / LON removal (temporary)

Sat, 13 Apr 2024 12:28:14 +0000

Update** LON1 is being restored to the mix. Upstream techs have been able to repair the existing switch (replaced fan). We have monitored the switch for the past 48-hours after fan replacement and has been reliable.

“We observed that the onsite fans were ineffective, but the replacement fan we dispatched has proven successful. At present, the switch seems to be operating reliably, and we’ve been actively monitoring its performance. "

-—

We are temporarily removing LON1 from the anycast mix due to recurring issues with a failing switch.

Upstream techs have assured us the replacement switch is on its way however to prevent further short disruptions we are removing the location temporarily.

Routes to LON will be restored once the new switch is installed.

[Resolved] Scheduled Maintenance on Datastore

Wed, 07 Feb 2024 23:00:00 +0000

We will be performing maintenance on the Datastore powering access logs, error logs, ssl logs and the filtering report tomorrow.

No significant interruption is expected however reduced performance and minor disruptions are likely.

Slightly reduced performance may occur tonight as we prepare for the update.

It is possible that data collected during this time may be under representative of actual traffic and patterns.

[Resolved] London Core Router Software Update

Fri, 02 Feb 2024 14:00:05 +0000

Maintenance Details:

Activity: Core Router Software Update
Impact: Temporary Service Interruptions
Duration: Up to 30 to 60 minutes
Date: Feb 03 2024
Maintenance Window: 1:00 PM GMT to 9:00 PM GMT

Details: During the specified maintenance window, our team will be performing a critical update to the Juniper code on our core router. This update is essential to enhance network performance, security, and overall stability.

Expected Impact: While we strive to minimize disruptions, it is anticipated that your services will experience interruptions during this maintenance window. The expected downtime is estimated to be within the range of 30 to 60 minutes. We understand the importance of your services and will work diligently to complete the update as quickly as possible.

Action Required: To minimize any inconvenience, we recommend scheduling critical tasks outside the maintenance window or notifying relevant parties within your organization.

[Resolved] Network Maintenance

Thu, 01 Feb 2024 08:08:16 +0000

There will be network maintenance carried out affecting US & EU PoPs of the Standard and Premium anycast networks within the next 24-hours.

Our upstream is making changes to their transit mix. We will be adjusting our network profile accordingly.

Some degree of interruption is inevatable. Please feel free to report any problematic routes if we fall behind on resolving them during the maintenance period.

[Resolved] Singapore: Emergency Maintenance

Wed, 24 Jan 2024 20:16:20 +0000

Update - Problem appears resolved. We will continue to observe for any regression.

[Resolved] Dashboard & Website offline

Wed, 17 Jan 2024 09:01:18 +0000

Update - Vendor informs us that service is restored and is continuing to monitor (https://status.linode.com/incidents/cv0y0409nl7f)

We confirm our services website and assosiated dashboard services are back online.

Update - Vendor has informed us that its an issue affecting all Block Storage in the Newyork Region and that we should expect an update soon.

Investigating - Block Storage failure affecting our database server currently has the panel offline.

We have made the vendor aware and are awaiting a fix.

[Resolved] Chicago: Offline

Mon, 15 Jan 2024 11:39:00 +0000

Equinix IBX Site Staff reports that multiple chillers are down at the moment and the vendor has been contacted. The vendor is expected to be onsite at approximately 10:30 Site Local Time. Temperatures at the IBX are rising and are above SLA. IBX Critical Facility Engineers continue to monitor.

[Resolved] LAX: Temporarily removed

Fri, 01 Sep 2023 12:11:32 +0000

Update - At long last upstream has resolved the issue.

“One of the edge routers encountered issues due to a firmware bug. detecting the issue was challenging as no logs were generated, due to the router running outdated firmware. the issue was resolved by updating the RE code and rebooting the router.”

Routes to LA have been restored.

Investigating - We have been investigating some issues this month in LA upstream. While we wait / work on resolution we have pulled routes from the region.

We will return routes once the situation is resolved.

[Resolved] LAX: Issues

Sun, 20 Aug 2023 08:37:56 +0000

Update issue resolved. Still awaiting more information.

We will restore LAX routes.

There is the possibility of routes being pulled in the near future for some maintenance. But for now routes are being restored.

We are aware of some issues affecting LAX. As such we have removed LAX from the Anycast mix.

We will update and restore routes when resolved.

Traffic will now be redirected to nearby PoPs.

[Resolved] Hazard Notice

Wed, 26 Jul 2023 03:00:00 +0000

We will be performing maintenance tomorrow. No interruption is expected however risk exists for iterruptions over this time window.

[Resolved] Hazard Notice: Amsterdam, Ashburn, Chicago

Thu, 29 Jun 2023 01:00:03 +0000

Update** Maintenance completed (smoothly). If you hve any issues please do let us know

We will be be performing some maintenance in these three locations in sequence tomorrow. No interruption is expected however some periods of packet loss are possible.

Certain internal metrics such as filter graphs may be affected for periods of the maintenance.

[Resolved] Dallas & Japan issue detected

Mon, 26 Jun 2023 12:47:18 +0000

Update Dalas issue resolved. We are waiting on a fix for Tokyo.

We have removed routes to both regions.

[Resolved] Hazard Notice

Fri, 23 Jun 2023 04:00:00 +0000

Update** Maintenance completed (smoothly). If you hve any issues please do let us know

We will be be performing some maintenance in these three locations in sequence tomorrow. No interruption is expected however some periods of packet loss are possible.

Certain internal metrics such as filter graphs may be affected for periods of the maintenance.

[Resolved] SYD1: Hazard Notice

Mon, 19 Jun 2023 09:30:00 +0000

Update Maintenance complete for tonight. One action has been delayed to a further maintenance period which we will perform in the coming days.

We will be be performing some maintenance in Sydney tonight. No interruption is expected however some periods of packet loss are possible.

Certain internal metrics such as filter graphs may be affected for periods of the maintenance.

[Resolved] Configuration update delay

Fri, 12 May 2023 12:11:54 +0000

Update service appears to have been restored for about half an hour. We havent yet got an update on our ticket. We will re-open this incident if necessary.

Due to an upstream service interruption affecting our Kubernetes cluster configuration updates are currently unavailable.

We have spoken to our upstream (Linode) and they have assured us the matter is being worked on.

When their issue is resolved any backlogged updates will be processed.

[Resolved] SYD1: Maintenance

Tue, 28 Mar 2023 03:56:28 +0000

Investigating - We are taking SYD1 offline temporarily for some maintenance. All routes will be redirected to nearby PoPs.

Complete Maintenance complete

[Resolved] Vegas and Luxembourg

Fri, 06 Jan 2023 00:00:23 +0000

We are scheduling maintenance (system restart) tomorrow for a quick patch. Interuption of less than 10 minutes is expected.

Complete

[Resolved] NY1

Wed, 04 Jan 2023 04:30:10 +0000

We will be conducting some emergency maintenance shortly to resolve a discovered issue. We expect only a few minutes of interruption while the server restarts to apply the fix.

Update Completed without issue (<1m interruption)

[Resolved] Vegas: Offline

Fri, 14 Oct 2022 18:25:00 +0000

Upstream Incident Report -

Short version:

Networks' been crapping itself for the past day. After 2 - 3 hard failures today we did an emergency JunOS downgrade and are cautiously optimistic that it’s resolved.

Longer/technical version:

For the past month or so we’ve been suffering a weird issue where random IP addresses would stop responding. The issue is that our router would lose/expire out the ARP entry for these IP addresses and wouldn’t attempt to learn them again unless forced to.

No amount of traffic getting sent to the IP externally (pings, random connection attempts, etc) would make the router issue an ARP “who-has” message. The only fix was to either ping the IP on the LAN (through our speedtest page for instance), forcing the router to see the ARP message on the network, or to login to the router and ping from there.

No amount of limits, tweaks, process restarts, etc, has helped with this. The only way to date has been the above methods.

This morning we had an issue where all users on 209.141.x.x dropped offline. These subnets had been moved down to our new Cisco core switch to help resolve the issue. Our Juniper router dropped the ARP entry for the Cisco switch, killing BGP. After some time it re-learned then entry reestablished connectivity, bringing those subnets back online.

This happened three times in the past 24 hours, each outage lasting anywhere from 5 minutes to 30 minutes.

Las Vegas is the only location suffering this issue. The key difference between all locations being that Vegas was running JunOS 20.4, where as everywhere else is 18.2~4. In JunOS 20.x they made changes to ARP to allow policing. While we did change this policy, it didn’t help us any.

As of an hour ago we performed a downgrade to JunOS 18.4. It obviously went terribly. After some research we found the configuration entry causing issues (our 100G ports kept channelizing) and fixed it, bringing things back online.

As of now, ARP’s are resolving near instantaneously, even for new IP addresses. We took this as a good time to change the rest of our ports to 100G, giving us a 400Gbit interlink between our router and core switch.

We’re cautiously optimistic that this issue is now resolved.

We thank you for your patience.

Upstream reports that it is conducting emergency maintenance to downgrade the router firmware in an attempt to resolve issues.

-–

Offline again, ticket has been sent.

-–

Upstream restored services.

The underlying issue (router bug) is still under investigation.

-–

Vegas upstream is offline. We have ticketed them and contacted staff members. We await response.

[Resolved] Premium Anycast: Lumen Transit

Tue, 04 Oct 2022 01:55:08 +0000

Update It finally happened, it was a saga, but london Lumen has finally been lit.

We are currently in the process of onboarding our new (replacement) transit provider Lumen (formerly Level3). Originally lit only for APAC and South America Lumen transit has now been lit for US and EU PoPs with one exception.

We are currently waiting on transit availability in London (Globalswitch). Due to the close pysical location and significant advantage in lighting Lumen accross EU and US at this time we have opted to proceed before London is ready.

The current ETA for London’s lumen transit is “a few weeks”.

[Resolved] Miami: Instability

Tue, 05 Jul 2022 04:54:31 +0000

We will attempt to keep interruption to a minimum.

Update - We have rebooted the system into a new kernel with a proposed fix. The underlying cause (potentially hardware) continues to be investigated. We beleive however that the action taken will prevent the issue from impacting reliability further.

We will keep this incident open and update it as matters progress.

Update - The workaround applied yesterday has proven itself effective. We will keep the workaround applied at this time until the underlying [assumed] hardware issue can be resolved. We do not anticipate any further instability and will schedule maintenance as appropriate for the next step seperately.

[Resolved] AMS: Removed

Sat, 25 Jun 2022 09:11:42 +0000

After observing intermitant interuptions throught the day we have made the decision to temporarily remove the AMS PoP from both Anycast networks while we await upstream resolution.

Routes via AMS will be restored once we get a suitable response or stability has appeared to return.

[Resolved] Miami: Network Unavailable

Mon, 16 May 2022 01:22:11 +0000

**Update** Upstream restored connectivity

We are aware the Miami PoP is unavailable currently. Issue looks upstream and we await response from their operations / techs.

[Resolved] Equnix IX (Singapore)

Tue, 05 Apr 2022 02:42:37 +0000

Update We have restored Equinix SG IX. Issue was an incorrect route policy upstream. Upstream has issued a fix. We will be continuing to monitor the situation for any regression.

Update We have made adjustments temporarily reducing the traffic going to SG but preventing route over-reach. We await the IX fix.

Update we are awaiting upstream response on this matter

We are aware of some (HE) traffic on the Premium Line from around the world arriving in Signapore. This appears to be due to a BGP bug in Equinix IX (Singapore). We are working with our upstreams on it.

[Resolved] Client Panel & Website Upgrade

Tue, 22 Mar 2022 02:00:00 +0000

Update Deployment of all queued customer configuration updates has been performed. If you are encountering any issues please do let us know.

Update Customer deployments and all customer facing services are up. If you are encountering any issues please do let us know. We will process the backlog of request for all services after a short break.

Update Most site components are are up. Customer service deployments will be available shortly. Appologies that this has taken much longer than expected.

Update Site is up (in dry-run mode). Changes to customer services will be deployed once we have completed full system tests. We will be taking this slowly and will update when an ETA is available.

Update We are currently running a bit behind schedule on the first stage of the maintenance. We now hope to complete the first stage within the next 45 minutes.

Update Maintenance has begun

Update Maintenance will begin on time in less than half an hour.

On Tuesday afternoon we will be moving our web panel and database to new upgraded servers. This will consist of approximately 1 hour of panel and site outage (maintenance page) and up to 1 hour of partial ouage (dry run mode) while we complete manual tests on all components and services. During this time any customer service changes requested will be queued for execution on completion.

We appologies for any inconvenience this may cause.

We will update this status announcement as appropriate with updates.

[Resolved] Budget PoPs offline (upstream)

Tue, 15 Feb 2022 00:07:29 +0000

Update - Upstream resolved.

[Resolved] Luxembourg Network Outage

Mon, 24 Jan 2022 01:31:04 +0000

Update No official all clear yet as work related to the move continues. However all XCs are now up and services online for over half an hour.

Update Some routes are back. Some XCs are not yet spliced.

Upstream network provider has announced a maintenenace affecting Lux customers.

–

We apologize about the short notice (again).

In the next 12 to 24 hours we will be migrating our router and cross connects in Luxembourg. Due to LuxConnect using a 3rd party to complete the fiber splicing, we can’t set an exact start time.

To complete this work a full network outage must occur. The actual moving of the router upstairs should go quick since it’s a single unit, but given the delays we’ve had already I won’t promise any ‘completion’ time.

What we can say is this:

Luxconnect claims each XC will take around 10 minutes to splice.
We have 4 cross connects in this location so their side will take about an hour to complete.
We don’t require all 4 cross connects to be completed for service to be restored. Although there will be high latency and poor speeds until the work is fully completed.
Root really shouldn’t take more than 15 - 20 at the most to move that single unit as it’s quite light and doesn’t require 2 people. Famous last words.
Once we see one of or links go down we’ll press on Root to start the router move ASAP.

Since we’re not on site to complete this work ourselves we’re at the mercy of our vendors. We’re sorry we can’t give concrete completion times.

[Resolved] LV1: Outage

Sun, 05 Dec 2021 23:37:00 +0000

Investigating - We are aware of an outage in Vegas, we reported to datacenter and are awaiting resolution.

Update - Local engineer on his way to datacenter to replace server PSU

Update Fixed

[Resolved] Observed issues with GTT routes

Thu, 14 Oct 2021 03:23:50 +0000

We are aware of issues with GTT routes at the transit level. These occur primarily from EU<->US however some issues have been observed with other routes. We have contacted them for a comment however have not (as of yet) received one.

Observed Resolved We are not longer observing GTT related packet loss.

[Resolved] Chicago Rack Maintenance

Tue, 12 Oct 2021 22:00:00 +0000

Scheduled Maintenance Upstream datacenter will be performing maintenance in the rack our server is located in. To keep downtime to the mininmum they have offered to move our server to an adjacent rack. We expect have been told to expect an interruption of between 10-20 minutes between 5:00pm and 7:00pm on Tuesday the 12th of October (Centeral Time)

Update Maintenance complete

[Resolved] LA: Hardware Upgrade

Tue, 28 Sep 2021 04:50:32 +0000

Update Routes are up. We will observe closely for any issues. Please do open a support ticket if you notice anything we don’t.

Update Maintenance is proceeding roughly on schedule. We expect to be complete in 15-30 minutes.

Update Maintenance is begining now

We will be performing maintenance in L.A on Monday and Tuesday to upgrade the hardware we have deployed in the region.

We will endeavour to keep interruption to a minimum by removing the PoP before and during major works. We expect to be able to perform this maintenance within 1-2 hours.

We will update this announcement with further information when available.

[Resolved] Outage

Fri, 24 Sep 2021 16:47:43 +0000

Investigating - We are investigating a potential upstream issue that might affect the uptime of one our of services. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.

Update - Upstream beleives it’s a breaker or power related. We await feedback from dc techs.

Update power restored

[Resolved] Chicago Launch

Fri, 27 Aug 2021 07:22:55 +0000

Due to unforseen circumstances the launch of the Chicago PoP has been delayed from the 27/08/2021 to 28/08/2021. We appologise for the inconvenience.

Routes are up for all ranges in Chicago. Please do report any issues via ticket.

[Resolved] Amsterdam: Instability

Wed, 30 Jun 2021 14:58:53 +0000

Update Resolved. Will continue to observe.

[Resolved] Ashburn Maintenance

Mon, 28 Jun 2021 04:48:39 +0000

As of this latest update we are marking this maintenance window as complete. If you are experiencing any issues please don’t hesitate to open a support ticket.

Update Regarding the network interruption (~30 second) that occurred during the Ashburn maintenance we do appologise for any inconvenience, particularly those in other regions adversely affected that should not have been involved in this maintenance. Clearly we did not intend for this response and will be investigating a fix for the faulty module ASAP which will be deployed before any other hardware upgrades are completed.

Update Update not going as smoothly as planned. However there should be no further interruptions. Further information to follow once upgrade is complete.

Update We will be starting maintenance shortly. We expect minimal interuption. It is however possible some connections will be interrupted near the start and end of the maintenance. As always we will do our best to limit interruptions.

We are planning to conduct a hardware upgrade during off-peak in Ashburn. Exact time is currently being scheduled with local technicians. We will update this entry with more information when we get scheduling confirmation.

[Resolved] Brazil: Packet Loss

Sat, 26 Jun 2021 03:07:48 +0000

Investigating - We have detected packet loss international and possibly domestic links. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.

[Resolved] Amsterdam: Issues

Thu, 03 Jun 2021 00:46:45 +0000

Investigating - We are seeing an issue with either network port stability or ixgbe driver stability in Amsterdam. We are investigating.

Monitoring - We continue to watch for issues

Monitoring - We continue to watch for issues. Nothing has been observed as of yet.

Investigating - We continue to search for the root cause. Another incident occurred around 07:35 GMT.

Changes - Some changes have been made to test a theory. We continue to monitor.

Update - We beleive the changes to have mostly resolved the issue. Further changes will be made to complete this. With this the issue can be closed.

[Resolved] Packet Loss in London

Sun, 23 May 2021 21:17:00 +0000

Investigating - We are investigating some observed uptream packet loss in London.

Update - London network vendor has confirmed the issue

Update - We have not observed packet loss in half an hour. We await network vendor update.

Update - Confirmed fixed upstream

[Resolved] Cogent interruption

Wed, 12 May 2021 17:00:00 +0000

Investigating - We have narrowed the issue down to outgoing routes going over Cogent in two locations (AMS & LAX)

Identified - Cogent activated uRPF (without notice) on our upstream ports (2 PoPs) leading to an interruption as we don’t use Cogent for Ingress but they are used (if they are the best route) for Egress selectively.

To resolve this we are now exporting ingress routes to Cogent with Cogents own no-export community.

[Resolved] Dallas: Issue

Wed, 14 Apr 2021 11:56:26 +0000

Update from upstream “I apologize for the trouble, one of our virtual chassis switches was having problems routing packets, so all packets were being split causing 50/50 delivery, so to clear this problem along with applying changes I rebooted the problematic switch. The issue should be fixed and as well redundant routing is now active.”

[Resolved] Dallas: Switch issues

Wed, 14 Apr 2021 00:01:59 +0000

We are aware of some instability in the region. When contacted our upstream confirmed it and that it was resolved. It doesnt seem to be resolved.

We have hence pulled our routes from the region.

Update upstream has reported that everything has normalized and we have obersved the same. We are restoring Dallas routes.

[Resolved] UK & Amsterdam Maintenance

Sat, 03 Apr 2021 02:00:00 +0000

We will be conducting maintenance during regional off-peak in Europe tomorrow. No interruption is expected however a low risk of packet loss or latency does exist. This is the same maintenance conducted without interuption in Miami and L.A today.

Update Maintenance is complete in both regions. Unfortunately not as smooth as yesterday with both regions experiencing short interruptions. Appologies for the inconvenience.

Everything should now be at 100%. Please feel free to contact us if you are experiencing any issues.

[Resolved] LA: maintenance

Wed, 03 Mar 2021 10:00:00 +0000

Maintenance will be be performed in LA during regional off-peak time tonight. No interruption is expected, a risk of small interruption is possible.

Update complete. No interruption observed.

[Resolved] Miami: Maintenance

Wed, 03 Mar 2021 08:00:00 +0000

We are performing maintenance tonight during the regional off-peak. No interruption is expected however a risk does exist.

Update Completed without interruption.

[Resolved] London: Hardware Crash

Sat, 27 Feb 2021 17:40:50 +0000

The filtering server in London experienced hard failure resulting in server crash. The server has rebooted and is now online.

We are investigating the cause.

[Resolved] Amsterdam: Minor Maintenance

Mon, 15 Feb 2021 20:00:00 +0000

We will be conducting minor maintenance in Amsterdam. There will be a risk of a minutes of interruption as sessions are updated. We will attempt to keep the interruption as quick as possible if required (which we hope it will not be).

[Resolved] Amsterdam: Hazzard Notice

Mon, 15 Feb 2021 02:00:00 +0000

Today we will be performing preperation for tomorrows scheduled maintenance. No interruption is expected.

[Resolved] London: Minor Maintenance

Fri, 12 Feb 2021 19:00:00 +0000

At some point between the 7AM and 9:30 AM AEDT tomorrow we will be performing minor maintenance in London. Expected interruption is 1-5 minutes as we apply changes from todays preperation session.

Update Rescheduled due to upstream scheduling conflict.

Update complete

[Resolved] UK: Hazard Warning

Wed, 10 Feb 2021 03:00:00 +0000

We will be conducting some minor maintenance in London this afternoon. No interruption is expected however as always when work is being carried out there is a small chance of issue. Due to this we are conducting this maintenance outside of UK peak hours.

Update Maintenance is proceeding well. End time has been revised to reflect some expected delays.

Update Maintenance complete. Minor interruption did occur towards the end. Procedure has been updated to prevent this in future executions of this plan.

[Resolved] Dallas: Partial Interruption

Mon, 08 Feb 2021 18:51:06 +0000

Investigating - A partial outage has been detected in Dallas. We have pulled routes from region while we investigate / resolve.

Resolved - Issue related to some minor maintenance being performed behind the scenes not going as planned. Caused has been fixed and maintenance is complete.

[Resolved] Hazard Notice: Dallas

Thu, 14 Jan 2021 01:50:45 +0000

We will be performing some preperation for upcoming maintenance in Dallas at a time after 6PM AEST tonight. We do not expect any interruption in region however the possibility of minor packet loss or short interruption does exist. We will endevour to perform preperation without any interruption.

[Resolved] Amsterdam: Maintenance

Sat, 09 Jan 2021 01:40:16 +0000

Investigating - We are investigating an increase in latency observed last night. In order to perform this investigating without risk to customer performance we have removed AMS from the network with UK taking over to provide capacity while we investigate.

Complete - AMS routes restored. Issue should be resolved.

[Resolved] London: Investigating

Tue, 05 Jan 2021 04:31:15 +0000

Investigating - We are investigating a potential issue that might affect the uptime of one our of services. As such we have pulled routes from the London region while we investigate.

Update - Attempted to restore routes however we are seeing a problem with that. Please hold while the issue is identified.

[Resolved] Dallas: Fault Detected

Mon, 04 Jan 2021 00:32:13 +0000

Investigating - We have detected a fault in our network cable / switch port. We will be scheduling replacement later today.

Update - Replacement occurring now

Update - While we confirm the issue is resolved we will keep routes disabled. If the lack of Dallas routes is causing you issue please open a support ticket and we will see what we can do.

Update - Confirmed resolved, routes re-enabled.

[Resolved] Budget: Issue continued.

Sat, 02 Jan 2021 03:49:26 +0000

Upstream - Confirmed issue with them.

[Resolved] Budget: Incident

Wed, 30 Dec 2020 04:20:00 +0000

Upstream response - Confirms issue on their end.

Upstream response - Upstream is experiencing DDoS attacks on unprotected resources they operate (extortion).

Attack looks to have ended for some hours. Marked as resolved.

[Resolved] Lux: Packet Loss

Sun, 27 Dec 2020 09:39:32 +0000

Upstream Response - Confirms the issue and will have it fixed ASAP.

Under Observation - Upstream has the issue resolved for now and will continue observation.

[Resolved] Luxembourg: Unplanned Hardware Migration

Wed, 23 Dec 2020 09:50:55 +0000

We have been made aware of an emergency migration taking place in Luxembourg. We have been given an ETA of 1hr to move our data. Once moved we will endevour to get the node booted ASAP.

Appologies for the inconvenience expecially at this time of year.

Fixed Maintenance complete. We are currently conducting tests to ensure everything is A-OK. Please let us know if something is currently not.

[Resolved] Amsterdam: Connectivity Issues (2)

Sat, 19 Dec 2020 14:42:11 +0000

Investigating - It appears that the Emergency maintenance performed by upstream to resolve Amsterdam: Connectivity Issues has not been successful in resolving the ToR stability issue. We have seen what appears to be a similar interuption. We have pulled routes again and await upstream response.

Investigating - CPU spikes on upstream ToR switch. They have made some adjustments and continue to monitor. We have restored routes.

Fixed - Upstream reports fixed. We will continue to monitor closely over the next few days however.

[Resolved] Amsterdam: Connectivity Issues

Fri, 18 Dec 2020 16:02:44 +0000

Identified - We hae confirmed a partial interruption in network traffic in region. This is similar to a smaller incident observed yesterday which was reported upstream. We have collected data and reported this incident upstream and are awaiting resolution. Routes have been pulled from Amsterdam at this time.

Identified - Upstream beleives they have identified the issue on their end and are taking action to confirm. Routes remain removed from region.

Resolution (1) - An emergency top of rack switch update has been scheduled to resolve a software bug in the managment software. Routes restored until we have a date/time.

Resolved - ToR switch updated

[Resolved] Amsterdam: Reverse Proxy issue

Wed, 16 Dec 2020 11:37:16 +0000

An issue affecting Layer 4 reverse proxies in Amsterdam after todays maintenance has been reported and resolved. Appologies for the inconvenience.

[Resolved] Maintenance: Amsterdam Network Upgrade

Tue, 15 Dec 2020 19:50:52 +0000

We will be upgrading our network capacity and filtering hardware in Amsterdam. Any customers who have configured a HA type setup will be largely unaffected as routes will be removed from the region at-least 10 minutes before start of maintenance and traffic redirected to London and Ashburn PoPs.

We expect the total maintenance to take approx 1-hour.

Update - Maintenance taking a little longer than expected. Routes will be restored to Amsterdam as soon as possible.

Update - Routes have been restored, final finishing of tasks continues in the background with no expected interruption.

Update - Final checks complete. If you are experiencing any issues please don’t hesitate to contact us.

[Resolved] Packet Loss (multiple locations)

Sun, 13 Dec 2020 05:53:02 +0000

Cause - We are seeing intermittent issues caused by an upstream system not working as it should or being delayed in operation. We have tickets waiting response upstream.

Update 1 - We have some mitigations in place and await full diagnosis and fix upstream.

Update 2 - We continue to await confirmation of permanent fix. Tempory fix means no issues have been observed for >24-hours.

Update 3 - We continue to await confirmation of permanent fix. However as there has been no impact on customer services for days we will close this issue.