Two ConnectWise customers say Hosted Server — recently rebranded as ConnectWise .net Cloud Server – is down, and ConnectWise has sent emergency notification emails to both customers. Here’s some quick analysis but I’m being cautious here: I don’t know if this is a really small, isolated outage or something larger. Updated Dec. 22, 7:40 p.m. eastern: ConnectWise has restored the service. Here are the details.
ConnectWise sent an automated email to at least two Cloud Server customers at 6:31 a.m. pacific on Dec. 22. The email included an emergency notification stating:
“You are currently unable to access the ConnectWise .net Cloud Server, formerly known as the Hosted Server.
We are working to resolve the issue as quickly as possible and you will receive notification as soon as the problem has been resolved.
Thank you for your patience and we apologize for any inconvenience this may cause you.
For assistance, please contact the Partner Support Services Team.”
One ConnectWise customer says he has received roughly 5 email updates from ConnectWise today, stating the service is still down. A second ConnectWise customer says the Cloud Server system failed around 9:00 p.m. on Dec. 21 and he started receiving ConnectWise alerts the morning of Dec. 22.
I’ve got voicemails into multiple ConnectWise representatives seeking comment.
ConnectWise’s Response
Updated 7:40 p.m. eastern, Dec. 22: I just got off the phone with ConnectWise COO Santo Cannone. He confirmed the outage and said it was an operational issue and service has been restored. Cannone says there was no data loss and incoming emails to partners were queued and are now being processed into service tickets.
Also, ConnectWise has sent the following email to affected customers:
Dear [Name removed by MSPmentor],
Your access to the ConnectWise cloud was affected by an outage in our data center today. The outage was caused by an operational error by one of our staff. We are truly sorry for the inconvenience this has caused you.
We understand that any downtime is unacceptable, so we are offering a full month’s credit on your PSA service. We will implement new procedures to assure that this type of outage does not occur again.
We recognize that ConnectWise is central to your operations and we are committed to continually making investments that increase the value of our products and services to you.
Santo Cannone
Chief Operating Officer
Key Questions
I’ll strive to keep things in perspective until an official ConnectWise statement arrives.
Back in October 27, 2009, I jumped the gun (and the shark?) a bit when I wrote a blog headline stating “ConnectWise Cloud: Dark Until Oct. 28.” The outage was real, but in reality it involved a small portion of ConnectWise customers. No customer outage is good but I overstated the issue.
Fast forward to the present, and there’s little chatter about the current ConnectWise outage on Twitter. But there is chatter. Contacted by phone, multiple sources confirm they can’t access their ConnectWise .net Cloud Server. In short: Something’s up but I’m not sure if it’s a small or large outage.
Updated 7:40 p.m. eastern, Dec. 22: It was, indeed, a significant outage but there was no data loss.
Growing Pains
Multiple cloud providers have taken lumps in recent days. Big cloud providers like Amazon.com and RackSpace both have suffered outages. Ironically, MSPmentor’s own web site has hit bumps lately and we’ve drawn up some migration plans that will unfold soon.
Back in the MSP market, rivals ConnectWise and Autotask continue to compete fiercely; while Autotask positions itself as a “cloud-based IT services management pioneer,” ConnectWise has been pushing beyond its on-premise heritage to introduce cloud efforts.
ConnectWise’s march into the cloud hit a bump in October 2009. Did another issue occur today?
As soon as we have a status update regarding the ConnectWise .net Cloud Server issue we’ll share it with readers.
Updated Dec. 22, 7:40 p.m. Yes, it was a significant business issue since the outage was widespread but there was no data loss.
ConnectWise’s Cannone says service has now been restored and the company is now reviewing its operational business processes to avoid a repeat of this outage.
Read More About This Topic
Share This Post
Posted In: Professional Services Automation (PSA) | Software as a Service and Hardware as a Service
Tags: ConnectWise .net Cloud Server | ConnectWise Hosted Server | Managed Services Cloud | MSP Cloud
Interact: Add a Comment | Trackback Link | Permalink
Subscribe: RSS Feed
9+ hours of downtime is not something your clients appreciate when your suppose to be delivering uptime. ConnectWise has not provided any details or an ETA for a fix!
-Jacob
ConnectWise hosts their “ConnectWise.net Cloud Server” at http://www.myconnectwise.net. In October, this was the server that was down for a few days (for the 200-some-odd customers). Today, when you try to access http://www.myconnectwise.net, it’s not responding at all.
Seems to be a bit of a big deal, in my opinion.
Hi Jacob, Travis: Sorry to hear you were impacted by the outage. I’m told by ConnectWise COO Santo Cannone that service has been restored. We’ve also updated our blog entry with ConnectWise’s description of the issue.
-jp
Kudos to ConnectWise for being up-front and honest about the service interruption. Knowing Arnie and Santo personally, I’m certain they are working around the clock to address these issues.
Cloud services and the internet in general has moved so fast as an industry that it doesn’t surprise me that we are all experiencing these growing pains.
“Not ready for prime time” as my friend Ed would say.
Then again it can be, IF you expect growing pains. If your company can handle an outage and it’s not the end of the world, then the cloud already makes sense.
We evaluate the risk before we’d even present a cloud option to a client. When we do present it, we point out the pitfalls.
Recent audit of a client shows they lose $5,000 per hour if their network is down!! This does NOT include possible deals that could be lost etc.
Can’t put them in the cloud, not yet. Too much money to be losing hoping someone you never meant in the cloud is working on it to get it back up.
The cloud just isn’t for everyone yet… bring your umbrellas.
-Chris Chase
P.S. Happy Holidays everyone!
P.S.S. CW posted the issue has been resolved.
Yep; it’s back up and running.
I understand the sinking sensation that is felt when an “oops” is made; I feel for the person(s) that made the mistake(s) in this case!
The fact that it took almost 16 hours to correct is huge; I’m sure Connectwise lost a *lot* of money because of that.The fact that they are stepping up to the plate with honesty and an offer of compensation is stellar. Thanks, Connectwise!
I hope for those using the hosted version in production that their business hasn’t been affected too much.
Merry Christmas, everybody!
Bill
<
Do you ever get the feeling that you’re just beating a dead horse?
-Vlad
@Vlad… Who? And what dead horse?
I dont mind being diplomatic and I do not have intentions to bad mouth our PSA provider but this issue is completely unacceptable when you have clients using the downstream IT component of Connectwise and they can not access their ticketing system powered by Connectwise. This makes us lose credibility as an MSP and affects our repuation as a provider when this happens. We have come to rely heavily on this for all components of our business and there are no more excuses for this. Credits dont help, excuses are poor and the answers we receive are not acceptable. We have experienced two outages of this magnitude in the last 60 days that heavily affect our fast growing operations. I am sure Arnie and Santo are working on a plan to prevent this, but this is twice in the last 60 days. They really need to look at the impact that this causes across the industry if they are going to be number 1!
What would your clients do if you were down twice in the past quarter? It happened to Rogers today with Blackberry. I am not sure what ConnectWise’s SLA is since we are not on the hosted offering, can anyone share this and are they in violation of their SLA?
@tfox,
I think Chris said it the best. The pace at which we are moving is bound to create some issues for all providers in this segment. The question, and perhaps the big picture, is not just the reputation of reliability of a single PSA’s offering but the level of change and pace of development that is needed for this industry to survive.
I won’t speak for Autotask or ConnectWise, but I will speak for Own Web Now and ExchangeDefender: We are moving at 500 mph because if we should choose to take a break, go easy and be ultimate perfectionists our partners will be dead in the water.
The IT Solution Provider industry is moving at a snail pace compared to the speed of Microsoft, Google, Amazon and Dell. Our only option is to put 110% effort into making sure we do not wake up one morning with an afternoon project of moving our mailboxes to Gmail. That’s going to put some bumps in the road. But…
How did the alternative work out Duke Nukem Forever?
-Vlad
The partial ConnectWise Cloud outage has been resolved and all affected Partners are back online as of 7:30pm Tuesday. This was a partial outage and did not affect all Partners in the Cloud.
The outage was caused by an operational error, an “Opps I should not have run that routine, Undo, Undo” by one of our staff while performing maintenance routines early Tuesday morning 1:32am. It was a Human error by one of our NOC technicians. We are truly sorry for the inconvenience and hassle this has caused you. A credit does not make it up to you because we understand that any downtime is unacceptable. We are crediting all effected partners a full month’s of ConnectWise PSA service. We are implementing new procedures for our NOC technicians to assure that this type of outage does not occur again. We make no excuses. We will get better. You deserve the best Cloud service and reliability.
We are very sorry and ask for your trust and faith that when we have an issue we are making investments in products, people and processes to improve the services you rely upon us to deliver.
There’s a reason I bought ConnectWise and have it in house, that I don’t rely exclusively on VOIP, and my PBX is sitting in a closet in the back of my building – the Internet breaks! I’m an IT guy, and I *know* this stuff breaks, and on a pretty regular basis.
These things are too critical to my business to have them out of service, even for a few minutes! So, I opt to build my own infrastructure, bear the expense of that, and own the responsibility when it does (eventually) break for some reason.
I think ConnectWise handled this as well it possibly could have. The company has considerably more resources than I, which means they will be able to build a cloud that is considerably better than what I can build. They’ll fix it, get it right, and take good care of their partners. They always have, they always will.
The overall risk of this type of an outage is pretty small… If you assume each of the two outages in the last year lasted 24 hours (which is a high estimate) that still gives CW’s cloud 99.452% uptime. Certainly not five nines, but definitely not shabby, either.
The bottom line is this: If you can’t stomach the inherent risks of a cloud based solution, write a check and build your own cloud. Then you’re in complete control, and solely responsible.
Unless, of course, the power goes out, your generator fails, the Internet goes down, you have a flood or a tornado, or a guy on your payroll clicks “Yes” one too many times to the question, “Are you sure you want to do that?”
It’s great they emailed everyone about the issue after it happened – but Joe do you know if they warned clients about the upcoming maintenance that was being performed? If they perform maintenance on a predictable schedule and inform their customers as such I think customers would be much more forgiving.
We do this with our clients even if no downtime is expected during the maintenance, just in case things go south we still appear in control. It’s all about expectations, if you set them accordingly ahead of time even when things do go wrong your client will not be so quick to tear you a new one.
Having equipment on premise allows you to perform sometimes risky maintenance around your precise schedule – to many of our clients that is a huge benefit of not going to the cloud.
Gerson: I don’t know if there was a note about upcoming maintenance.
But in this case I don’t think this is about setting customer expectations. I freak out when MSPmentor’s service provider has 5 minutes of downtime. I can’t imagine hours of downtime.
It’s a tricky situation for all involved: Even the biggest names (RackSpace, Microsoft, RIM, Amazon) are suffering outages. As a whole, the giants are spending billions — with a B — of dollars on those cloud/SaaS efforts.
Meanwhile, it’s difficult for those who offer five-nines reliability to brag. It’s like the airline industry: You don’t dare brag about your safety record because the day you go down your messaging, company and customers take a hit.
I wonder: Will anyone in the cloud space ever be the Volvo of safety/reliability? And will they have the nerve to state it?
-jp
The current outage (that has been resolved) is causing us to re-evaluate our dependency on a single PSA offering. It seems this was a classic issue of having a single fault point and no redundancy.
If you have a service offering that is BUSINESS CRITICAL and you are promoting it as being production ready you should have procedures already in place to support it. For example if it looked like their was going to be more than 15 minutes of downtime why didn’t an additional data center with replicated information pick up the slack until the primary system was up and running?
Why is it only after a failure that a Master Managed Service company is evaluating processes that should have been part of core proactive procedures?
-Jacob
P.S. ConnectWise above all should get some major respect for being honest and upfront.
We have had problems with Google’s email system, so as far as I am concerned no one is immune.
I agree with Jacob’s latest comments. having a single point of failure is bad business practice for a critical function, ConnectWise’s business procedures need to be improved, but they get my respect for being upfront about the issue.
Like Tom Fox said, build your own infrastructure and bring it in house. In house might, of course, be a local colo facility, but at least if that is down you can drive over to see what’s happening. And you can decide what level of backup/failover protection you need and can afford. I bought before hosted was an option, and I am quite comfortable being the responsible party.
Linda,
I think that is where we are going to be heading. After the events over the last couple of weeks and watching what happens with other providers such as RIM, Amazon S3, UltraDNS and Rackspace all experiencing Cloud outages.
-Jacob
I believe a lot of the issues of CW being down wouldn’t be soo much of a huge issue if everyone had a local cached copy of their data. This would allow users to log in and see old data vs no data.
All I can say is that Connectwise’s outage cost us business and now I am trying to still recover after the holidays to make up for the extra work caused by the outages. I realize I shouldn’t have “all my eggs in one basket” but isn’t that the reason why CW gets paid by me and all their customers? I also realize that there are going to be outages at some point and not every system is perfect, but I am not comforted by the fact that Arnie came on here and said:
“The outage was caused by an operational error, an “Opps I should not have run that routine, Undo, Undo” by one of our staff while performing maintenance routines early Tuesday morning 1:32am.”
Shouldn’t I feel worse sitting here today that one Connectwise system routine can cause an outage and take down the whole thing???
Hi there Joelle – my name’s Jeannine Edwards, I work on Arnie’s team. I can understand your frustration, and can spend some time walking you through the steps we’re taking to ensure complete reliability and take your comfort level up if you’d like. I was unable to track down your direct contact info, so I thought I would leave mine for you here – feel free to reach out to Jeannine@ConnectWise.com and we can grab some time to connect.
Jeannine Edwards
Director, ConnectWise Community
Joelle: You bring up a good point. My intent was to explain the reason for the outage and to be transparently honest. We have done a full scale review of maintenance procedures and have implemented new measures to assure that this kind of human error won’t happen again.
Gerson: Your point about a maintenance windows is spot on. We will be instituting a weekly maintenance window. We will only perform maintenance in this window. There will also be a division of duties between the NOC Technicians and the ConnectWise application to assure a full system test and acceptance by the application team after the NOC work is accomplished.
We have also done a full review of this outage and have taken steps to make our cloud service even more bullet proof. We do have a high level of redundancy in our cloud offering and this outage was not caused by a lack of redundancy; it was caused because we did not have a separation of duties in one area that would have caused a set of checks and balances. That has been corrected. We are leaning from these mistakes and improving. We don’t take uptime lightly, we think it is the imperative of a cloud offering. ConnectWise is constantly improving and learning about cloud services. We are constantly improving our cloud performance and technology.
Vlad: I agree with you that every cloud provider is in a learning and perfecting mode. As a new industry, cloud services are still in their infancy and improving. ConnectWise is no exception. We are committed to the process of constant improvement in everything we do and are excited by the challenge of perfecting and bullet proofing our cloud offering.
If anyone would like to email me directly, feel free at Arnie@ConnectWise.com
Arnie Bellini
CEO, CPA, MBA
ConnectWise
Looks like their processes failed again.
Hosted ConnectWise is down.
I hope potential customers are reading this. I want to stress that these are *not* isolated issues and I would consider myself lucky if they were the only regular problems encountered.
I had non-stop server errors on the hosted offering causing my client to close every 5-10 minutes for several *months* before it was resolved by some updates. Discovering new bugs or strange functionality is a regular occurrence.
This is just par for the course with ConnectWise.
Feel free to contact me directly at dan at unetworks dot com.
Just to clarify part of Dan’s post: We have informed all of our cloud partners that we will have a standing maintenance window starting at Midnight Eastern time each Wednesday evening. The feedback we have received from our partners is that they appreciate knowing that and can plan for it. Last night, we used that time to implement new software features for our cloud partners.
We’ll make sure Dan gets the information. Other than that, we’ll be contacting Dan directly to review his issues.
Thanks.
Santo Cannone
ConnectWise
They were down again this last weekend for an extended period.
Fellow IT professionals were impacted signicantly by this outage with no explanation given to this point.
Philip
Clarification – We did not have an outage this weekend. We took advantage of the Easter weekend to complete some maintainance for our ConnectWise Cloud partners. All Cloud partners received notification before, during and upon completion of the maintainance.
Thanks very much!
Jeannine Edwards, Director, ConnectWise Community
jeannine@ConnectWise.com
I realize this is a very old bog but wanted to inquire as to whether anyone else is experiencing Connect+Wise issues as of late? We’ve had a wide array of problems over the last couple of weeks ranging from cloud outages to loss of custom settings in our software to unpredicatable software behavior and display. This has resulted in us submitting repeated tickets to Connect+Wise and all we’re really getting back is the occaisional “cloud is down” e-mail or responses to our tickets throwing it into our court as though we’ve done something to cause the issues. Anyone else having the same experiences? I’m sure we can’t be alone in this.
-Terri
Terri,
Updated, Dec. 6, 7:53 a.m. Thanks for your note. I checked in with ConnectWise and I believe the issues you experienced are isolated to your business rather than a true cloud outage or broad service disruption. I believe ConnectWise is addressing the issue.
-jp