— New online course!  Getting Started with Azure DevOps — Special launch price 50% off —

TFS2010 Build Agent Cannot Talk to Controller: “There was no endpoint listening at http://servername:9191/Build/v3.0/Services/Controller/5 that could accept the message”

by

The Problem:

I ran into a problem with a customer’s Team Foundation Server 2010 build server today. The server’s been up and running for at least 6 months and then suddenly it wouldn’t run builds anymore – the builds would start and then hang with the message “Run On Agent (waiting for build agent)”. 

SNAGHTML48e6619

When I went to look at the server configuration in Team Foundation Server Administration Console’s Build Configuration dialog, the Build Controller and Build Agents both said that they were “Ready”.  The weird thing was that the Agent icon didn’t indicate that it was ready.  (see below)

SNAGHTML4905eba

In Team Explorer’s “Manage Build Controllers” dialog, that build agent was coming up as “Unavailable”.

SNAGHTML492fbd3

Back in the TFS Administration console, I went to the Build Agent Properties dialog and clicked “Test Connection”.

SNAGHTML4954de8

SNAGHTML493d733

 

After about 60 seconds I got a “Testing connections for the build agent timed out” message box.

SNAGHTML495f9ff

After this error message box popped up, we’d get a TFSBuildServiceHost error in the Event Log.

SNAGHTML49800a3

Here’s the complete text of the event log error:

Log Name:      Application
Source:        TFSBuildServiceHost
Date:          11/17/2010 3:45:43 PM
Event ID:      0
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HQDEVUTILITY
Description:
Service ‘HQDEVUTILITY – Agent1’ had an exception:
Exception Message: There was no endpoint listening at http://hqdevutility:9191/Build/v3.0/Services/Controller/5 that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. (type EndpointNotFoundException)

Exception Stack Trace:
Server stack trace:
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
   at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Microsoft.TeamFoundation.Build.Machine.IBuildControllerService.TestConnectionFromController(String agentUri)
   at Microsoft.TeamFoundation.Build.Machine.ServiceProxies.ServiceProxy`1.<>c__DisplayClass3.<Do>b__2(T channel)
   at Microsoft.TeamFoundation.Build.Machine.ServiceProxies.ServiceProxy`1.Do[TResult](Func`2 action)
   at Microsoft.TeamFoundation.Build.Machine.BuildAgentService.<>c__DisplayClass12.<TestConnection>b__11(Object )

Inner Exception Details:

Exception Message: Unable to connect to the remote server (type WebException)
Exception Stack Trace:    at System.Net.HttpWebRequest.GetRequestStream(Tran
sportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()

Inner Exception Details:

Exception Message: No connection could be made because the target machine actively refused it 127.0.0.1:18303 (type SocketException)

Exception Stack Trace:    at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

The Solution:

It turned out that there was a problem with the proxy settings in Internet Explorer for the build service account.  (Thanks to Richard Hundhausen for the suggestion to check the IE proxy settings.)

1. Log in as the same user as the build service account

2. Open Internet Explorer and go to Tools –> Internet Options

SNAGHTML49c7ae1

3. Choose the Connections tab
SNAGHTML49e1c3e

4. Click the “LAN settings” button

5. You should now be on the Local Area Network (LAN) Settings dialog.  Verify that the Proxy server settings are correct.  If you aren’t using a proxy server, the “Use a proxy server for your LAN” checkbox should be UNCHECKED.

SNAGHTML49f10d9

6. Restart your TFS Build Service

7. Verify that your build agent is running and that the icon indicates that the agent is available.

SNAGHTML4a15c2b

Summary:

What appears to have happened was that the Proxy Server settings were mistakenly and incompletely pushed to the build server via Windows Domain Group Policy.  Once the proxy settings were corrected, everything worked again.

-Ben

 

— Looking for help with your Team Foundation Server?  Problems with your TFS2010 Team Builds?  Want some coaching on Microsoft Visual Studio ALM?  Drop us at line: info@benday.com


5 Responses to "TFS2010 Build Agent Cannot Talk to Controller: “There was no endpoint listening at http://servername:9191/Build/v3.0/Services/Controller/5 that could accept the message”"
  1. Thank you. This was exactly the problem that I was experiencing.

    Since I couldn’t get the Group policy ammended, I added an entry into the hosts file to prevent the proxy from affecting it.

  2. Thanks – I wasn’t convinced this was my problem but logged on as the tfs account & found the rogue proxy checkbox was checked. Unchecked it & restarted the agent & all back to green.

  3. Thank you Ben,
    We were getting frustrated with this issue, we were doing all the steps but not the very first step “1. Log in as the same user as the build service account”

    Your post helped us to fix the issue.

    Thanks,
    Naveen

  4. Hi Ben

    I would like to give you a big thanks. I was really frustrated with this. And how in the world could you guess that this was the problem. But a big thanks to you for saving days of troubleshooting for so many people. Keep up the good work.

    Regards,
    Irfan

  5. This issue just hit one of my build controllers during a final build for push to QA at about midnight. This would of impacted hundreds of users (let alone my sleep) but your post fixed the issue.

    Nice work, and thanks for the sleep I will now be getting tonight.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: