Accessing claims from ProfileService - identityserver4

Question

I have a simple project with identityserver4 and ElmahCore.
I added a custom IProfileService and IResourceOwnerPasswordValidator
Authentication works like a charm, but custom claims I add in the Profile Service do not show up in the principal when I try to restrict Elmah access.
services.AddElmah<SqlErrorLog>(
options => {
options.CheckPermissionAction = context => context.User.Identity.IsAuthenticated;
}
);
The User Identity exists and is authenticated, but the only claims that exist are sub, name, auth_time, idp and amr.
Other custom claims do not show up.
The value of sub is set to the value I expect it to have.
I also added this line before calling services.AddIdentityServer(), but nothing changed:
services.AddScoped<IUserClaimsPrincipalFactory<HaproUser>, AppClaimsPrincipalFactory>();
The application configuration is set up like this:
public void Configure(IApplicationBuilder app)
{
app.UseForwardedHeaders(
new ForwardedHeadersOptions
{
ForwardedHeaders = ForwardedHeaders.XForwardedProto
}
);
if (Environment.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseFileServer();
app.UseIdentityServer();
app.UseMvcWithDefaultRoute();
app.UseElmah();
}
And service configurations is this:
services.AddMvc();
services.AddScoped<IUserClaimsPrincipalFactory<HaproUser>, AppClaimsPrincipalFactory>();
var builder = services.AddIdentityServer()
.AddInMemoryIdentityResources(Config.GetIdentityResources())
.AddInMemoryApiResources(Config.GetApis())
.AddInMemoryClients(Config.GetClients());
builder.Services.AddSingleton<IUserRepository, UserRepository>();
builder.AddProfileService<HaproProfileService>();
builder.AddResourceOwnerValidator<HaproPasswordValidator>();
if (Environment.IsDevelopment())
{
builder.AddDeveloperSigningCredential();
}
The ProfileService is fairly straight forward:
public class HaproProfileService : IProfileService
{
// IsActiveAsync omitted
public Task GetProfileDataAsync(ProfileDataRequestContext context)
{
var sub = context.Subject.GetSubjectId();
var claims = context.RequestedClaimTypes.Select(type => MapClaim(type, user)).ToList();
context.AddRequestedClaims(claims);
return Task.FromResult(0);
}
private static Claim MapClaim(string type, HaproUser user)
{
switch (type)
{
case "name":
return new Claim(type, user.DisplayName);
// Omitted cases here
}
}
}