Guest Author: Joan Resnick Ehrlich

The SharePoint 2010 Beta installation and product configuration went smoothly. There is not much to the process, which is rather straightforward. We ran into a show stopper post-installation when manually creating a few of the service applications. The problem also occurred on my home server, leading to a reformat/reinstall of that server as part of the troubleshooting process. That in turn led to an opportunity to install SharePoint using Kerberos authentication following the TechNet instructions here: Configure Kerberos authentication (SharePoint Server 2010), which I will step through in my next article. This article will explain the post-installation show stopper and subsequent troubleshooting. “Fasten your seat belts, it’s going to be a bumpy night.” (Said the late, great actress Bette Davis as Margo Channing in the 1950 movie classic “All About Eve”.)

Each instance of a service application has a service application proxy. This is part of the new shared service architecture in SharePoint 2010 which replaces the Shared Service Provider (SSP) architecture in MOSS. Several articles and blog posts later (links provided at article end), a few SharePoint Conference 2009 sessions, and one free (OMG, free) Critical Path Training webinar by principal owner and SharePoint MVP Andrew Connell, I think I understand the service application-proxy relationship. Using one service application instance and one web application as an example, keeping in mind that a service application can be associated with many web applications and a web application can have one or more instances of that service application:

A service application provides a service. “Consumer objects” on the web front end (aka inside a web application) consume the service. A consumer object is any SharePoint feature that provides functionality to the user; examples are web parts and code. Consumer objects do not communicate directly with the service application but rather, connect to the service application’s proxy, which also resides in the web application. It is the proxy that connects to and communicates with the service application. Now, here’s where it gets a bit complicated. The proxy knows it is associated with the service application but not on which application server the service application resides. The service application can reside on one or more application servers. (Think scaled-out deployments.) So the proxy asks SharePoint where its service application is. SharePoint knows which application servers host that service application. If the service application resides on more than one application server in the farm, SharePoint determines which server is next in line to handle a request (in a kind of round robin load-balancing way) and tells the proxy. The proxy then communicates with the service application on that server. The proxy passes the request from the consumer object to the service application. The service application handles the request and passes the results back to the proxy which passes it back to the consumer object.

Hopefully my communiqué is more “By George, she’s got it” (My Fair Lady – must be old movie night on the brain) than not. The gist is, proxies are a critical piece to the architecture.

Well, then… after manually creating the Excel Services and Access Services service applications, the corresponding proxies showed a “Stopped” status. So too did the proxy for the “WSS_UsageApplication” service application. Given a proxy’s role, this was Not Good. The show stopper? We could not figure out a way to start the proxies and nothing we tried worked. Not good at all.

A screenshot of the Excel Services and Access Services proxies showing “Stopped”:

The WSS_UsageApplication proxy also showing “Stopped”:

We had created the Excel Services and Access Services service applications in Central Administration via the Manage Service Applications “New” menu. Most service applications can be manually created this way. PowerShell can also be used.

Here is the “Create New Excel Services Application” settings page:

As the screenshot indicates, we used a new, dedicated application pool and dedicated managed account. We did the same for Access Services and all other service applications listed in the menu, which offer the same opportunity.

There are by my count – and it sure is hard to keep count – five service applications not listed in the menu but when created are listed on the “Manage Service Applications” page. Two are automatically created and configured by SharePoint when a farm is created. These are the Application Discovery and Load Balancer and Security Token service applications. You should therefore see these listed on the Manage Service Applications page when first launching Central Administration.

The other three are the Application Registry Service, State Service, and Usage and Health data collection service applications. All three can be created using the Farm Configuration Wizard. In fact, we could not find a way to create the Application Registry Service other than the Wizard, which in the end is how we did create it. PowerShell stumped us. We did not see a directly related “New-” command, e.g. “New-SPApplicationRegistryApplication”, or even a general “New-” command, e.g. “New-SPServiceApplication” to use. As it ends up, we don’t need the service because we have WSS only. The service provides backward compatibility to the SharePoint 2007 (MOSS) Business Data Catalog (BDC). In doing research for this article I found out that “Application Registry” was the original term for the BDC and the BDC is referenced internally (API, Object Model) by that term. (And do we all know that in 2010 the BDC has been renamed Business Data Connectivity and is part of the Business Connectivity Services (BCS) architecture?)

The State Service and Usage and Health data collection service applications can be created using PowerShell (New-SPStateServiceApplication, New-SPUsageApplication). We used PowerShell for the State Service. The Usage and Health data collection can also be created by using the “Configure usage and health data collection” link under Reporting in the Monitoring section in Central Admin. We used the link, which we discovered by following the instructions in Initial configuration under Configure farm services  in the SharePoint Server 2010 TechNet Library.

This leads to an interesting digression. Configuring usage and health data collection using the link created the WSS_UsageApplication service application. When we later ran the Farm Configuration Wizard (to create the Application Registry service application) the selection box for “Usage and Health data collection” was checked and greyed out, as is the case after the first instance of a service application is created. When I redid my home server I let the Farm Configuration Wizard create all service applications, including Usage and Health data collection, as part of the troubleshooting process. Instead of the expected WSS_UsageApplication, a “Usage and Health data collection” service application resulted:

Clicking the “Usage and Health data collection” service application led to the same configuration page as did “WSS_UsageApplication”. What’s with the name difference? I haven’t a clue.

Anyway, I could not find within Central Admin how to start a proxy separately from its service application, nor could I find a PowerShell command (did I miss it?).  So I resorted to working with the service application. I tested first with Excel Services and this is what did not work: stopping and restarting the service, using the farm account instead of a dedicated account, and using a new application pool. Resetting IIS and rebooting did not work. Deleting/recreating the service application, deleting/recreating the service application using a new name, and creating a second instance (first removing the first instance from the default proxy group) also did not work; with each new instance the proxy showed “Stopped”. (I suppose that means I was able to recreate the issue on demand.) I tested next with Access Services but after deleting/recreating once with no luck I gave up. (Glutton for punishment I am not.)

I did an exhaustive search online which turned up empty except that I determined we are one of four people in the entire world to have experienced this issue. I posted to the Microsoft SharePoint 2010 – Setup, Upgrade, Administration and Operation forum. Except for my own blethering updates as I worked through the issue the post went unanswered, perhaps because no one knew what to make of it (or me).

I did not think the issue was related to installation but to the manual creation process. Because I was not sure whether the manual creation process was the cause – the other service applications’ proxies did not experience the issue – I decided to start over with a clean slate. Hence, the reason I reformatted my home server and used the Farm Configuration Wizard right out of the gate to create all of the service applications. Voila! That worked. All proxies showed “Started”.

Poking around, I saw that the Excel Services and Access Services service applications had been deposited by the Farm Configuration Wizard into the “SharePoint Web Services Default” pool. This can be seen via the Security, Configure Service Applications page:

Spencer Harbar calls this application pool “the big Daddy” because this is the pool in which all of the service applications are placed when the Farm Configuration Wizard is run. Take a read of his articles SharePoint Server 2010 Worker Processes and More on SharePoint 2010 Application Pools.

Another side note but “quite important in the scheme of Beta things”: I found Mr. Harbor’s blog when researching Event ID 5048 from WAS (Windows Process Activation Service) in the System Event log, which I noticed when the SharePoint Product Configuration Wizard finished:

The error referenced the Central Admin web application pool. Mr. Harbor explains the error and provides the fix here: Application Pool Mis-configuration in SharePoint 2010. I have had to apply the fix for each web application after creation.

As for resolving the stopped proxy problem at work, I decided to attempt a less drastic approach rather than reformat/reinstall. I deleted the troublesome service applications (rebooting the server after) and recreated them using the Farm Configuration Wizard. That also worked; the proxies showed “Started”. The next troubleshooting step would have been to change the service applications’ configuration to use a dedicated application pool. I chose not to bother for the Beta.

Here are the links to articles and blog posts about the SharePoint 2010 service application architecture. There are numerous more; just search on “SharePoint 2010” and (“service application architecture” or “service application framework” or “shared service”). Don’t forget TechNet and MSDN.

SharePoint 2010 Shared Service Architecture Part 1 by the Microsoft SharePoint Escalation Team:
http://blogs.msdn.com/spses/archive/2010/01/20/sharepoint-2010-shared-service-architecture-part-1.aspx

SharePoint 2010 – New Shared Services by James Kemp, Microsoft Consulting Services, UK SharePoint team:
http://blogs.msdn.com/uksharepoint/archive/2009/10/21/sharepoint-2010-new-shared-services.aspx

The New Service Application Architecture in SharePoint 2010 by Andrew Connell:
http://www.andrewconnell.com/blog/archive/2009/10/19/the-new-service-application-architecture-in-sharepoint-server-2010.aspx

SharePoint Service Applications by Ted Pattison (also a Critical Path Training principal owner and SharePoint MVP):
http://www.sharepointproconnections.com/article/sharepoint-development/SharePoint-Service-Applications.aspx

SharePoint 2010 Service Architecture and Logical Architecture by Jason Apergis
http://www.k2distillery.com/2010/01/sharepoint-2010-service-architecture.html
http://www.k2distillery.com/2010/03/sharepoint-2o10-logical-architecture.html

Planning for the New Service Application Architecture in SharePoint 2010 by Matt Passannante
http://vspug.com/matt/2010/02/23/planning-for-the-new-service-application-architecture-in-sharepoint-2010/

Guest Author: Joan Resnick Ehrlich

Joan Resnick Ehrlich has been in the IT industry for 15 years and is Corporate IT Administrator for a mid-sized company on Long Island, NY. Prior to entering the industry Joan was a business researcher, and she enjoys combining her research skills with IT work. In addition to SharePoint, her primary responsibilities include Windows Server, Active Directory, Exchange Server, and SQL Server.