1. Have you installed the new JSF Mobile app? Check out all the details here.
    Dismiss Notice
  2. One account & one avatar for all of JSF. Unified login and profile. Forum alerts on the main site, and more. Check out the details here: Forum & main site unified account feature is live!
    Dismiss Notice

Please Read - Data Center Problems!

Discussion in 'Technical' started by John Stone, Oct 2, 2006.

  1. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    The datacenter in downtown Orlando where the JSF servers are located has been evacuated due to some boneheaded construction worker who broke a massive water pipe on the 15th floor. The pipe break caused the sprinkler system to go off in the entire building. Between the broken pipe and the sprinklers, a massive amount of water was released. There is presently a lot of water coming down the elevator shaft and all the equipment closets. Thankfully my equipment and most of the critical equipment in the datacenter was not damaged by the sprinklers....

    ...however, the water is slowly building up under the floor where all the wiring and electrical outlets are. Further, the authorities have cut the power to the entire building and ordered everyone out. At this point we can't do anything except hope that the water doesn't rise high enough to short out the equipment, and keep fuel in the two massive diesel-powered generators that are keeping all the equipment running. It may be days before power is restored, and the possibility still exists that the authorities will make us turn off the generators.

    All JSF data is backed up and safely off-site, and I have fully redundant hardware off-site. In an effort to further minimize any possible data loss, I will be doing backups twice daily instead of once daily until this problem has been resolved.

    If the site suddenly goes down, you'll know why. I'm doing everything in my power to make sure it doesn't, but right now I'm pretty helpless to do much more.

    I'll keep you all updated.
     
  2. guava

    guava Elite Member
    Lifetime Platinum Member

    Joined:
    Feb 15, 2004
    Messages:
    12,607
    Likes Received:
    19
    I got an interesting message something like this:

    Thank-you for your post. You will now be redirected to the thread you replied in.

    I don't remember that happening before. Is it related?
     
  3. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    It's looking very likely at this point that the fire department is going to make us turn off the generators due to the large amount of water in the building. This could happen within the next 10 minutes.

    JSF may be down for a day or two, it all depends on how quickly they can clean up the mess and allow us to turn the power back on.
     
  4. wh0rume

    wh0rume Senior Member

    Joined:
    Jul 22, 2004
    Messages:
    10,928
    Likes Received:
    10
    that was quick.
     
  5. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    I got a voicemail while I was working out that said the fire dept. has given tentative permission for the generators to remain on. There is still the possibility that things may need to be shutdown again with very little notice. For now, however, we are back up.
     
  6. 1FastGTX

    1FastGTX Elite Member
    Lifetime Platinum Member

    Joined:
    May 17, 2004
    Messages:
    9,306
    Likes Received:
    2
    Fingers crossed! :)

    Good luck!
     
  7. gravityhomer

    gravityhomer Elite Member
    Lifetime Platinum Member

    Joined:
    Jan 23, 2004
    Messages:
    3,627
    Likes Received:
    2
    Yes, good luck with everything. I'm amazed at the amount of precaution and backups that you do with the forum. I can't believe the site is even up now after what you described.

    1-2 days!!! :eek: :eek:
     
  8. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    Thanks guys, I've got my fingers crossed!
     
  9. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    Datacenter flood/evacuation: Updated October 3, 7:45 AM EDT.

    Yesterday afternoon the datacenter in downtown Orlando where the JSF servers are located was evacuated due to some boneheaded construction worker who broke a massive water pipe on the 15th floor. The pipe break caused the sprinkler system to go off in the entire building. Between the broken pipe and the sprinklers, a massive amount of water was released. There was a lot of water coming down the elevator shaft and all the equipment closets. Thankfully my equipment and most of the critical equipment in the datacenter was not damaged by the sprinklers.

    The authorities have cut the power to the entire building and ordered everyone out. It may be days before power is restored. Yesterday afternoon (October 2) the fire department forced the datacenter to turn off the emergency generators, resulting in a complete outage starting about 5:30 PM EDT. Thankfully around 7:00 PM last night permission was given to turn the generators back on. JSF has been up and running on generator power since then. The possibility still exists that the authorities will force the datacenter to turn off the generators again today as cleanup and repair work continues.

    All JSF data is backed up and safely off-site, and I have fully redundant hardware off-site. In an effort to further minimize any possible data loss, I will be doing backups twice daily instead of once daily until this problem has been resolved.

    Please be aware that the I might have to shut everything down at a moment's notice, and there could be extended downtime. I'm pleased that so far we've only been down for about an hour and a half--fairly impressive considering what happened!

    I hope by the end of the day things stabilize and power is restored. I will post updates to this thread. Thanks for your patience.
     
  10. eleonardo

    eleonardo Well-Known Member

    Joined:
    Dec 1, 2004
    Messages:
    2,469
    Likes Received:
    0
    exciting stuff :)
     
  11. gravityhomer

    gravityhomer Elite Member
    Lifetime Platinum Member

    Joined:
    Jan 23, 2004
    Messages:
    3,627
    Likes Received:
    2
    I get this message everytime I use the multiquote feature, but otherwise I don't get it. I get a shock everytime I see it, because I think something happened to my post.
     
  12. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    Wow, this is the second day in a row where we went down just before my workout, and by the time I finished my workout we were back up! :)

    The BellSouth and Time Warner equipment that provides the transport for the DS3 data circuits at the datacenter finally ran out of backup power at 3:14 PM EDT. That equipment is located in the basement of the building, and is not on the datacenter generators (nor is it maintained by the datacenter). A gasoline-powered portable generator was used to restore power to the Time Warner and Bell South equipment. It's a good thing, too, because now we are being told that power to the building will not be restored until Thursday at the earliest!

    At this point there is plenty of fuel in both generators to last several more days, and more fuel can be added while the generators are running if needed. The possibility still exists that the power company or fire department may force the generators to be turned off for safety reasons, but that possibility is diminishing. Just be aware that we are not out of the woods yet, and there may be further outages.

    What a week this has been so far!
     
  13. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    The datacenter is still running on generator power as of this morning. I'm now told that power to the building will probably not be restored until Friday at the earliest. It all depends on how much damage the water did to the electrical system. The cleanup crews have been working night and day, and a full damage assessment is scheduled to take place sometime today. Any needed repair work will begin late today or tomorrow.

    There were no outages yesterday, but there is still the possibility that the generators may need to be turned off so the repair crew can safely perform their duties. It seems there was no damage to the datacenter's electrical system, so hopefully that won't be necessary.
     
  14. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    Yesterday was another day with no outages, yet we are still running on generators.

    I'm told the cleanup & repair work is going well, and we hope to be back on permanent power sometime today. I'll be glad when this ordeal is over!
     
  15. John Stone

    John Stone Every day is Leg Day
    Staff Member Owner

    Joined:
    Jan 20, 2004
    Messages:
    20,926
    Likes Received:
    75
    Everything is back on perminant power down at the datacenter. :)
     

Share This Page