IONERD - #ION-61394 - Core server down due to Hurricane – Incident details

All systems operational

#ION-61394 - Core server down due to Hurricane

Resolved
Operational
Started 25 days agoLasted 2 days

Affected

core.ionerd.com

Major outage from 1:10 PM to 10:03 AM

Updates
  • Resolved
    Resolved

    All services have now been restored. If you are still experiencing any issues, don't hesitate to get in touch with [email protected]

  • Identified
    Update

    Our data center has remained operational and continues to have power. Our high-efficiency UPS systems are also functioning to provide stability for any power fluctuations that may occur.

    At present our carrier nodes and hubs are still down in the area and all our primary and backup link carriers are working on restoring power and connectivity.

    We sincerely appreciate your patience and understanding

  • Identified
    Update

    Unfortunately, we have not received any significant updates from our carrier partners as the entire region has been severely affected by the storm. We are unable to provide an estimated time for resolution as all repairs are being handled by carriers and power companies across the region. We share in the frustration and understand the desire to expedite the process, but we are unable to do so.

    I want to clarify that our facility did not lose power during the storm. The facility is on the same grid as Emergency Services, and as of yesterday, it was back on utility power, with the generators no longer in use.

    We appreciate everyone's concerns for our well-being. We have staff on site and have been in touch with all staff in the area. While some are currently blocked in by storm damage, we can confirm that all staff are safe and sound.

    Thank you for your prayers and well wishes.

  • Identified
    Update

    We are actively collaborating with our partners to resolve the ongoing situation as swiftly as possible.

    Our carrier partners are in communication with the power companies, performing repairs and will provide updates at the earliest opportunity.

    We are closely monitoring the situation and will continue to share periodic updates.

    We sincerely appreciate your patience and understanding during this challenging time.

  • Identified
    Update

    We are still feeling the effects from the recent hurricane, our carrier partners have been working on restoring services and we are standing by when they restore connectivity to bring our core websites back online.

  • Identified
    Update

    We are working closely with our partners to resolve the current situation as quickly as possible. Our carrier partners are in contact with the power companies, carrying out repairs and will provide updates as soon as possible. We are closely monitoring the situation and will continue to share regular updates. We sincerely appreciate your patience and understanding during this challenging time.

  • Identified
    Update

    Our data center has remained operational and has had power, although it has been intermittent. Our backup diesel generators have been running as needed. The high-efficiency UPS systems are also functioning as expected to provide stability during any power fluctuations.

    However, the external infrastructure surrounding the data center has been significantly impacted by the hurricane. This includes widespread power outages and severe damage to carrier networks, which has affected connectivity. Despite our multiple entry points, the external carriers’ infrastructure has not been fully restored.

    We are working closely with our partners to resolve this as quickly as possible, but their repair efforts are being delayed due to the extensive damage in the area. We are monitoring the situation closely and will continue to provide updates periodically.

    We greatly appreciate your patience and understanding during this challenging time.

  • Identified
    Update

    We regret to inform you that due to a severe storm impacting the region, our Lenoir data center has experienced a loss of connectivity. Our team is working closely with our partners and our carrier providers to restore services as quickly as possible.

    We understand the inconvenience this may cause, and we deeply appreciate your patience and understanding during this time. Please be assured that we are doing everything in our power to minimize the disruption, and we will continue to provide updates as the situation develops.

  • Identified
    Identified

    Update, Vlad is aware of the issue and after investigation it appears to be another datacenter issue.

    We are working with the data center to resolve this as soon as possible however we have no exact time on when this will be resolved.

  • Investigating
    Investigating

    We are currently experiencing a networking failure affecting the functions regarding the operations of https://bladenode.com/ and https://ionerd.com/.