Any estimate of when the outage is over?
TD/Affiliate Support Area Login
The TD/Affiliate Support Area is temporarily unavailable due to a planned outage.
We apologize for any inconvenience
I think we brought it back online around 3:30 CT. The membership webstore is still down, and we don’t have an ETA on it. To assist members with membership transactions over the weekend, the office will have someone answering the phone to process memberships from 8AM to 3PM central time on Saturday and Sunday.
Update: We got the webstore repaired Saturday afternoon, I do not know if the office will be staffed on Sunday.
Down again (4:27 pm EST, 9/1/2019).
Seems to be working for me.
Down again (9:45 pm EST, 9/22/2019)
There appears to be a problem with the DB server in Crossville. Epsilon is aware of the problem, I don’t know what the issue is nor do I have an ETA for its return.
Followup: The epsilon tech is estimating it’ll be back online in an hour or two.
Not including delay.
Estimate is a more genteel word than ‘guess’. It’ll be up when it’s up, and I don’t know when that will be.
It looks like Gambit came back online around 4 AM Crossville time.
I don’t know what caused the outage, I do know Epsilon is digging into it to try to prevent it from happening again.
The next generation of ‘internal’ servers is likely to be implemented in the cloud as opposed to on dedicated hardware at the office. That’s no guarantee it won’t find a way to crash, as we have experienced several outages on the servers (uschess and secure2.uschess) at AWS.
Thanks for helping to get this fixed, Mike.
Are there contingency options which we could consider in the meantime, so we don’t have to contact you in case the next outage occurs at 3:00 am?
Epsilon was aware of the problem long before I saw your note, but their estimate of how long it would take to get back online was off. I think they were aware of the previous outage before I was, too, so I think there may be sufficient procedures in place to detect and initiate work on repairing outages.
Should there be a process by which notifications of unplanned outages are posted? I think that’s a question for the ED/EB. I’m not even sure where an appropriate place for those notifications would be, because the underlying cause may affect what’s available to post something about it.
US Chess already runs Facebook and Twitter sites and it appears that they are regularly updated. These vehicles could be considered to post information about website outages.
I don’t use either Facebook or Twitter, I don’t know how many TDs do.
I do not.
My wife started on Facebook so that she could see what our nieces and nephews are up to.
That said, I can be patient when there is a problem and I would anticipate that most of the impatient TDs would also be on Facebook and/or Twitter.
I posted updates on a couple of Facebook groups for TD’s…we all appreciate the hasty response here! It got resolved quickly!
Danny
There was a power failure in Crossville earlier today and Gambit (the main database server) is still in recovery mode. As long as it is down, TD/A and the membership webstore can’t function.
I don’t know when it will be back online, possibly not until after midnight.
Thanks for update. Still down at 11:50 CST.
Gambit came back online a little after 2 AM Central time on Sunday. This outage was due to a power failure in Crossville. I’m not sure if the plan is to host the new CiviCRM server (for membership processing) on the virtual stack in Crossville or in the cloud. There are advantages and disadvantages to both methods.
Long term, the cloud is probably the most logical place for all of US Chess’s servers, but that is not without issues (notably security) and possibly higher costs.
Data will not remain hosted in Crossville long term. Cloud storage becomes more and more economical but we will see.
I doubt the fire that burned the famous Crossville tree house had any connection. But it was the biggest story in town in a while.
Maybe the Crossville power company decided on a blackout for safety reasons a la PG&E in California. 