Release Management 2015 randomly hangs during acceptance/approval/validation/deployment steps - ms-release-management

Question

We have been using RM for a while with no significant issues until we upgraded from 2013 to RM 2015. We upgraded to 2015 and also migrated the RM server to be hosted on a different virtual machine at the same time.
Ever since that upgrade, we have an intermittent issue where RM will completely hang for 10-15 minutes whenever a release is accepted, approved or validated. This is what happens when RM hangs:
1) Multiple deployment agents will disconnect from RM because they can't connect to the server.
2) The RM fat client becomes unresponsive
3) The RM website becomes unresponsive
4) If the hang occurs during the deployment step, often times the deployment will fail because the agent can not talk to the server
Things we have tried:
1) Per Microsoft recommendation, we turned off recycling the app pool on the server
2) We made sure all of our servers are accessing the drop location using UNC paths and not HTTPS
3) Increased Look for Packages to Deploy timeout
This issue does not seem to be tied to any particular release template or even a release. It can happen to any of them at any time. So far, the only pattern we have been able to identify is that it only happens when someone accepts, approves or validates a deployment. Even then, it does not occur every time.
When RM hangs, both the server and the agents log errors like this:
Timestamp: 1/6/2016 11:36:09 AM
Message: Root element is missing.: \r\n\r\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)
at Microsoft.TeamFoundation.Release.Common.ExtensionMethods.XmlExtensionMethods.ToXDocument(String value, Boolean preserveWhitespace)
at Microsoft.TeamFoundation.Release.Data.Model.ModelFactory.TransformXmlToModel[T](T model, String xml)
at Microsoft.TeamFoundation.Release.Data.Model.ModelFactory.Load[T](Int32 id)
at Microsoft.TeamFoundation.Release.DeploymentAgent.Services.Deployer.DeploymentEventFetcherBase.DeployNextComponent()
Category: General
Priority: -1
EventId: 0
Severity: Error
Title:
Machine: [Redacted]
Application Domain: DeploymentAgent.exe
Process Id: 1880
Process Name: C:\Program Files\Microsoft Visual Studio 12.0\Release Management\bin\DeploymentAgent.exe
Win32 Thread Id: 2436
Thread Name:
and this:
Timestamp: 1/6/2016 11:36:09 AM
Message: Error while converting string to XDocument: [Root element is missing.] [ at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)
at Microsoft.TeamFoundation.Release.Common.ExtensionMethods.XmlExtensionMethods.ToXDocument(String value, Boolean preserveWhitespace)].
Value is:
Category: General
Priority: -1
EventId: 0
Severity: Error
Title:
Machine: [Redacted]
Application Domain: DeploymentAgent.exe
Process Id: 1880
Process Name: C:\Program Files\Microsoft Visual Studio 12.0\Release Management\bin\DeploymentAgent.exe
Win32 Thread Id: 2436
Thread Name:
Extended Properties:
After about 10-15 minutes, RM will recover on its own and be usable again. It might proceed with not problems on a few more releases, or it might hang again on the very next approval/acceptance/validation gate.
Any help troubleshooting would be greatly appreciated.

Answer 1

Joe, you can enable verbose logs for RM service but following the steps in https://blogs.msdn.microsoft.com/visualstudioalm/2013/12/12/how-to-debug-release-management-components/. Follow section named "Enable Services Logs on Release Management Server".
You can then mail the logs to rm_customer_queries_at_microsoft_dot_com for faster resolution.