Correlation Failed, Remote Login. AspNet Core Identity Server - identityserver4

Question

Trying to gain some basic understanding of how this process works as I am receiveing the Correlation failed error. Let me first begin my describing the issue I'm encountering...
QAT is not working properly and is configured as follows:
I have an Identity Server running behind a load balancer for QAT.
All requests sent to the load balancer are https.
The traffic being forwarded to each application server (2 seperate servers in this case) is http.
The Netscaler is adding all necessary X-Forwarded items to the header.
I have another application that also sits behind the load balancer for QAT.
There are 2 seperate servers hosting this application which the netscaler will forward the traffic to.
This application is configured to use the X-Forwarded info from the netscaler.
It is designed to authenticate using the above-mentioned Identity Server.
My issue is that I end up with a never ending loop between the second application and the Identity Server when I deploy to QAT. This is strange to me as my SYS environment works perfectly. My sys environment has a seperate instance of Identity Server and the second Application mentioned (except that there is only a single instance of each application being forwarded to). This also goes through the netscaler and does all the X-Forwarded magic mentioned earlier.
In both situations the setup is identical. The only difference is that QAT has multiple servers hosting each app and SYS only has 1 server hosting each app.
My question is why would this behave differently?
Why would this work in sys but not in qa?
I think at this point we can rule out the callback path, cookie settings, etc... b/c it work in SYS.
Could it be that I need to implement some sore of Data Protection Key middleware in both the identity server and the other applciation? On that note, I really dont understand the Data Protection Keys. Would both the identity server and the seperate application need to store their Keys in the same location (whether that be in the database or filesystem) in order to be able to decrypt the information stored in the cookie...
Any help is greatly appreciated.

Answer 1

It was definitely the Data Protection Keys that were the issue. My solution was simple. Save the encryption key as part of the deployment process, Create an IXmlRepository, and then add that to the startup. Easy Peasy.
using Microsoft.AspNetCore.DataProtection.Repositories;
using System;
using System.Collections.Generic;
using System.Text;
using System.Xml.Linq;
namespace Myapp.Encryption.Repositories
{
public class EncryptionRepository : IXmlRepository
{
private String Key { get; set; }
public EncryptionRepository()
{
var year = Convert.ToString(DateTime.Now.Year + 2);
var key = "<key id=\"983440f7-626b-46e4-8bfa-7c3d6d9d4619\" version=\"1\">" +
" <creationDate>2019-11-13T17:42:58.889085Z</creationDate>" +
" <activationDate>2019-11-13T17:42:58.3843715Z</activationDate>" +
" <expirationDate>" + year + "-02-11T17:42:58.3843715Z</expirationDate>" +
" <descriptor deserializerType=\"Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60\">" +
" <descriptor>" +
" <encryption algorithm=\"AES_256_CBC\" />" +
" <validation algorithm=\"HMACSHA256\" />" +
" <masterKey p4:requiresEncryption=\"true\" xmlns:p4=\"http://schemas.asp.net/2015/03/dataProtection\">" +
" <value>{{Your Encryption Key }}</value>" +
" </masterKey>" +
" </descriptor>" +
" </descriptor>" +
"</key>";
Key = key;
}
public IReadOnlyCollection<XElement> GetAllElements()
{
var collection = new List<XElement>();
collection.Add(XElement.Parse(Key));
return collection;
}
public void StoreElement(XElement element, String friendlyName)
{
// Not required as key is hard coded
}
}
}
services.AddSingleton<IXmlRepository, EncryptionRepository>();
services.AddDataProtection().AddKeyManagementOptions(a => a.XmlRepository = (services.BuildServiceProvider()).GetService<IXmlRepository>());