The New MOSSness

Do you have the, “cannot connect to configuration database” blues?  One of my clients did last week as well.  The first thing I noticed was this on the WFE:

The customer, nor I, were none to pleased to see this error.  I then looked at the services and noticed that the SharePoint administration service was not running.  I tried to start it, but it failed and then the event log displayed this:

The logs on the DB server were clean, so it didn’t appear to be a permissions issue.  I contacted a fellow SharePoint consultant, who is a developer name Bryan Phillips (His blog here) to see if he had any ideas.  And fortunately he did.  He told me to look at the groups on the local server to see if the SP groups were still there.  They weren’t; there were no WSS_ADMIN_WGP, no WSS_RESTRICTED_WPG, no WSS_WPG.  They were gone.  While doing research on this issue, I read in numerous places that this can happen if one tries to promote a member server to a domain controller, which makes total sense since DC’s don’t have “local” groups per se.  The other issue could be that some one may have tried to remove MOSS and then cancelled out at the last minute.  Regardless, we do not know the exact issue and can only theorize as to what happened and killed the server.

And here is the worst part; there were no backups for MOSS.  The backup script that had run had long since overwrote previous backups.  No server level backups either.  The great thing about this environment, was that it was totally virtual. 

Which brings me to why I love and whole heartedly advocate virtualization.  The technology allows so much flexibility and power that traditional server don’t offer.  For example, the DR potential is so much more simple to implement in a virtual environment than in a traditional physical data center.  And recovery from a VM failure can be so much quicker than a physical server failure.  Sure there is the whole issue of having a single point of failure with the one server, but that is why you buy two servers and make them into a highly available cluster! There can be other issues, but nothing that should be a deal breaker. *Rant over*

Ok, so since I have a virtual environment, I had the ability to take snapshots of all my VMs.  So I decided to take snapshots, and work from there.  First I recreated the groups on the WFE and used another server as guidance as to whom needed to be where; as in what users belong in which groups.  After I copied the setting from another server in the farm, I was able to start to the Administration service.  That was a good push, so I decided to try to to get to the CA site again, and it threw the same error that I referenced earlier.  So I decided I would run the config wizard on the WFE.  The odd thing was that when running the wizard, it seemed as if the server had never been a part of the farm.  So I told it to connect to an existing farm, entered the credentials for the farm account pointing to the correct DB server, and hit go.

And thus after telling the server to join an existing farm, and running the config wizard, the CA was site was now again functional.  All of the other sites were still throwing the error “cannot connect to the configuration database.”  I verified that the app pool accounts all had access to their appropriate content DBs and to the config db and they all did.  Permissions in IIS were correct as well.  So I thought I’d give the server a reboot.  After the server rebooted in 35 seconds (thanks to virtualization) I tried the sites, and all were now available.  My customer and I had a great sigh of relief. 

I hope these word may help you if you get into this situation.