What I am trying to do:
We have a Task Scheduler that kicks off an EXE, which in the course of its runtime, will connect to SQL Server.
So that would be:
taskServer.myDomain triggers the Task Scheduler action
taskServer.myDomain exe runs locally
taskServer.myDomain initiates a connection to sqlServer.myDomain
The scheduled task is associated with a service account (svc_user) that is set to run with highest privilege, run whether the user is logged in or not, and store credentials for access to non-local resources.
The actual behavior
What we are seeing is the Task Scheduler is indeed running as svc_user. It triggers the EXE as expected, and the EXE is also running as svc_user. When the EXE initiates a connection to SQL Server, it errors on authentication.
Looking at the Event Viewer we can see the failure trying to initialize the connection to SQL
Exception Info: System.Data.SqlClient.SqlException
at System.Data.SqlClient.SqlInternalConnectionTds..ctor(System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data.SqlClient.SqlConnectionString, System.Data.SqlClient.SqlCredential, System.Object, System.String, System.Security.SecureString, Boolean, System.Data.SqlClient.SqlConnectionString, System.Data.SqlClient.SessionData, System.Data.ProviderBase.DbConnectionPool, System.String, Boolean, System.Data.SqlClient.SqlAuthenticationProviderManager)
And then looking at the SQL Server logs we can see the root of the issue
Logon,Unknown,Login failed for user 'NT AUTHORITY\ANONYMOUS LOGON'. Reason: Could not find a login matching the name provided.
The connection initialized by the EXE to SQL Server is trying to authenticate as ANONYMOUS LOGON.
What I have tried
Background
This issue popped up when our IT team started deploying a GPO lockdown in our environments. So in order to get to this point, we first had to add some GPO exceptions to allow the svc_user to:
log on locally
log on as batch job
Progress?
This is where we started being able to capture the ANONYMOUS LOGON error in SQL Server. From there we tried a handful of other GPO exceptions including
Allow Credential Save
Enable computer and user accounts to be trusted for delegation
The actual issue?
So it would appear that this is a double hop delegation issue. Which eventually led me here and then via the answer, here and here.
So I tried adding GPO policies to allow delegating fresh credentials using the WSMAN/* protocol + wildcard.
Two issues with this:
the Fresh credentials refer to prompted credentials while the EXE is running as a service during off-hours and inheriting the credentials from the TaskScheduler
the WSMAN protocol appears to be used for remote PowerShell sessions (via the original question in the serverfault post) and not SQL Service connections.
So, I added the protocol MSSQLSvc/* to the enabled delegation and tried all permutations of Fresh, Saved and Default delegation. (This was all done in Local Computer Policy -> Computer Configuration -> Administrative Templates -> system -> Credentials Delegation)
Where it gets weird
We have another server, otherServer.myDomain, which we setup with the same TaskSchedule. It is setup with the same GPO memberships, but seems to be able to successfully connect to SQL Server. AFAIK, the servers are identical as far as setup and configuration.
The Present
I have done a bit more digging into anywhere I could think that might offer clues as to how I can feed the credentials through or where they might be falling through. Including watching the traffic between the taskServer and the sqlServer as well as otherServer and sqlServer.
I was able to see NTLM challenges coming from the sqlServer to the taskServer/otherServer.
In the case of taskServer, the NTLM response only has a workstationString=taskServer
On otherServer, the NTLM response has workstationString=otherServer, domainString=myDomain, and userString=svc_user.
Question
What is the disconnect between hop 1 (task scheduler to EXE) and hop 2 (EXE to SQL on sqlServer)? And why does this behavior not match between taskServer and otherServer?
So I finally have an update/solution for this post.
The crux of the issue was a missing SPN. The short answer:
Add an SPN for sqlServer associated with the service account SQL services are running as (not the svc_user)
example: SetSPN -S MSSQLSvc/sqlServer.myDomain myDomain\svc_sql_user
Add another SPN like above but w/ the sql service port
example: SetSPN -S MSSQLSvc/sqlServer.myDomain:1433 myDomain\svc_sql_user
Set the SQL service user account to allow delegation like so
Related
I have a strange one for you. I'm maintaining several databases prior to a migration. One of them is a 2008R2 instance. This instance has multiple errors in the logs (the infrastructure has been poorly maintained), so I set up a bunch of alerts (16-25) and tried using Database Mail to send them. But the mail registry settings keep resetting and preventing it from working. I can't tell if someone is maliciously going in behind me and reverting the settings in the registry (this is possible in the poisonous environment I'm working in) or whether it's some kind of obscure problem.
Just to confirm... I've created the same alerts with the same mail settings on the 2017 instances that I'm also monitoring with no problem. Equally, on the 2008R2 instance, I can successfully set the Database Mail parameters, send myself a test email AND execute a job, sending a 'completed' email using the same Database Mail profile and user via an Operator.
Setting the parameters using xp_instance_regwrite or sp_set_sqlagent_properties didn't work either, although I realised early on that the parameters weren't sticking because of a lack of admin rights on the server, so I got the infrastructure guys to give me access. I then:
logged in to the server
shut down the Agent (it isn't doing anything at all)
configured the registry settings (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.<instance>\SQLServerAgent\UseDatabaseMail = 1, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10_50.<instance>\SQLServerAgent\DatabaseMailProfile = <my-mail-profile>)
restarted the Agent.
I then confirmed from SSMS that the 'Mail session' parameters (Enable mail profile, Mail system and Mail profile) were correctly set. A day later, the log is full of errors, I have no emails and all of the Agent properties are empty and greyed out!
Anyone seen this before?
My windows service is using integrated authentication and running under Local System account and got the below exception.
The target principal name is incorrect. Cannot generate SSPI context.
The SQL Server Service is running under domain admin user e.g. "domain\administrator". If I change the SQL Server Service to run under Local System account then it fixes the above error.
Can anyone explain why it's happening like this? We have an InstallShield wizard which installs our application on client side i don't know how we can handle this behavior through the wizard. Also changing the user for SQL Server Service is not realistic as well because the client may not allow it.
Note: Once when my windows service works fine and I revert the SQL Server run under the admin account my service runs fine. I guess there are some permissions are set to the local system account.
Before it, I ran the Kerberos which generated the following script to run and fixed the issue. After this it was not required to change the user for SQL Server Service.
SetSPN -d "MSSQLSvc/FQDN" "domain\machine$"
SetSPN -s "MSSQLSvc/FQDN" "domain\administrator"
Please explain why it's happening and what is the best way to handle the situation?
When running under the Local System account, sql-server registers an spn for every service it controls automatcially up to active-directory, and attempts to unregister them when the service shuts down. The Local System account has the ability to communicate over the network as the computer account and thus can indicate to Active Directory as to when to make changes about itself and the SPN SQL Service wants to register. When you change the SQL Server account over to an AD domain user account, the Local System account immediately loses it's ability to control this; therefore you must manually delete the existing SPNs previously registered for that SQL service by Local System before registering new SPNs. You should now notice why its nice that the SQL server script helpfully calls for a deletion of the old SPN followed by the registration of a new one in order to prevent issues. When this isn't done properly - you'll get an authentication error when the kerberos clients obtain a ticket for the old invalid SPN - because it was never deleted and any Kerberos-aware service will always reject a ticket for a wrong SPN. After you make SPN changes, always be sure to restart the SQL Server service and right after that if you’re testing with a user have that user log out and log back in. This answers your main question here.
Please see this Microsoft document for further reading on the subject: Register a Service Principal Name for Kerberos Connections. There's also a very good youtube video on this exact problem, that's where I learned about it and how to resolve it. Ignore "SSRS" in the title, I've watched the entirety and the guidance applies to any and all services by SQL which have SPNs.
You had a secondary question at the very end of your question regarding what is the best way to handle the situation. If you're talking about solving it programmatically that would be very difficult to answer as all environments are different in some way and you will come across SQL instances running in all sorts of different security contexts. In an online forum like this you would probably get different answers from different people. If this were your only question, I think it would get closed by the moderators for "being primarily opinion-based" and likely to attract spam answers. I would suggest you incorporate some kind of guidance about the problem in some form of a Readme file that you should package with the InstallShield wizard.
Side note: I think you should add the kerberos tag to this question - as SPNs are relevant to Kerberos only - and not to any other authentication protocol.
I run the same script as an application account under IIS, and as a Scheduled task, and i get very different results.
From IIS (as a web page), i can read e.g. field properties, and run very simple queries like
SELECT * FROM SYSTEMINFO
So connection to database is there.
Anything more "advanced", gives get all sorts of errors:
SELECT SYSTEMINFOID FROM SYSTEMINFO
or
SELECT COUNT(*) FROM SYSTEMINFO
or
INSERT INTO SYSTEMINFO ( SYSTEMINFOID ) VALUES (1)
typically gives "[Microsoft][ODBC Microsoft Access Driver] Unknown (SQL-42000)" or "[Microsoft][ODBC Microsoft Access Driver] Overflow (SQL-42000)" which means "missing access or syntax error" according to some ms error message overviews (Quite broad group of errors...)
If the application account is member of Administrators, all works fine (but not a good solution), any other memberships (user, power user, backup operator) gives the same error. (need to reboot server for such changes to take effect, so debugging is time consuming.)
User has local policies to "Replace a process token", and "Adjust memory quotas" granted (as well as run as batch + service).
Environment looks the same (except that in IIS web environment is added), and i can see in task manager that the scripts are run as the application account.
connection string is
DBI:ODBC:driver=Microsoft Access Driver (*.mdb);DBQ=C:\folder\Demo.mdb;
running CGI, Perl 5.18, DBI,
on IIS 8.5 Win2012 std, appliction pool uses the application account,with load user profile, and Impersonate user as True
Setting up the same on Win 2008 R2 works fine.
Connection to e.g. a local ms sql express database gives SSL errors, even if SSL is not in use, and using a sql account for authentication.
Spent four days on looking at all odd combinations, any suggestions for settings I may have overlooked?
(summary: when running as application account, not being an administrator under iis 8.5 the script is running, connection to database works, but anything more than trivial queries results in odd errors. )
Finally found a solution:
in IIS 8.5, for a service account to work, the same user (as pool user) must be set as Application Pass-trough authentication.
Creating an application under the Web Site,
Selecting Basic Settings for this Application, assigning it to a separate pool run by the service account was not enought.
In addtion, the "Pass-throught autentication" must be set as the same user, by selecting the "Connect as..."-button.
PROBLEM BACKGROUND
Sorry if this is a bit tedious to read, but please bear with me.
I have been tasked to determine the most restrictive security permissions...or rather investigate if more restrictive security settings can be configured for the SQL server login our program uses, yet still function as normal.
Currently the program runs as a Windows service configured to log on using a Windows user account that has been configured in SQL server with trusted auth. The login used has been assigned a db_owner role and the service works fine like that.
So to narrow the permissions for this user I removed the db_owner rights and assigned it to the db_datareader and db_datawriter roles. Unfortunately this causes a problem and when I start up the service I get an error dialogue displaying:
Error 1053: the service did not respond to the start or control request in a timely fashion.
and in the event viewer under the System events are logged:
event 7009 (timeout waiting for..to connenct)
event 7000 (the service did not respond to the start or control )
My problem is the code base is really large and I'm not sure what exactly to look for that would require db_owner permissions (it sets permissions maybe?).
QUESTION
What should I be looking for in a program that executes SQL that would cause it to require db_owner permissions?
In case the first question is too general: is there an easy way/any tools I can use to figure out what a Windows service is trying to do during start-up 'SQL wise' if I get system error events logged:
event 7009 (Timeout (30000 milliseconds) waiting for the ... service to connect)
event 7000 (The service did not respond to the start or control request in a timely fashion).
BTW I tried running profiler with all audit events selected, but still get nothing logged when starting the service.
This is such a broad question without knowing the architecture of your service and how it communicates with SQL Server. Are you using in-line SQL? Stored Procedures?
I think you'd best tackle this issue by starting from the service's code and tracing the execution path from the start and see what is being executed on/against SQL Server.
Alternatively, if you are using stored procedures, you may want to script them all out into a file and search on some common T-SQL commands limited to a db_owner, such as CREATE, DROP, ALTER.
Executing SqlScript at the remote DB causes an error:
Failed to connect to SQL database. (-2147467259 myDB1)
The SqlScript is the following:
<sql:SqlString
Id='UpdateSomething1'
SqlDb='myDB1'
ExecuteOnInstall='yes'
User='SQLUser'
ContinueOnError='no'
ExecuteOnReinstall='no'
ExecuteOnUninstall='no'
Sequence='26'
SQL='[SqlString]'/>
where the Db is:
<sql:SqlDatabase
Id='myDB1'
Database='myDB1'
Server='[DATABASE_SERVER]'
CreateOnInstall='yes'
DropOnInstall='no'
DropOnUninstall='no'
ContinueOnError='no'/>
and the user is:
<util:User
Id="SQLUser"
Name="myUserName1"
Password="password1"/>
The problem does not occur with the local DB.
We extracted more specific error message from the IP traffic (the actual error that the remote MSSQL server throws):
Can not open database "myDb1"
requested by the login. The login
failed. {remote machine name} Login
failed for user {user name}
Thank you for any help and information.
Max
I would need more information to be sure but here are some general observations I've had over the years.
In MSI, you typically run deferred custom actions with no impersonation so that they run as Administrator to support managed/elevated installs where the invoking user doesn't have admin either because they really don't or because UAC hasn't elevated their process.
In InstallShield, and I'm sure WiX is similar, this typically causes a problem for remote database connections. If you have a dialog in the UI sequence to test the connection it will succeed ( when expected to ) because the interactive user has permissions to that database/instance. And if installing locally it will succeed because SYSTEM (typically) has permissions the database/instance. But when installing to a remote instance it will frequently fail because SYSTEM can't authenticate against SQL on the remote machine. Your mileage will improve if using sql authentication ( e.g. SA ).
Personally I have some practices that I follow. If I'm creating a single tier system, I restrict the database to (local). If I'm creating a 2 tier system, I create two installers: one for my database layer which I restrict to (local) and one for my application layer which I then reuse the sqllogin dialog to verify connectivity and write the values out to a web.config or app.config. This allows me to loosely couple the layers and service them independently of each other.
I hope this helps to understand the types of issues that can be encountered. I don't know your exact problem without seeing your environement.
The WiX custom actions are just using standard OLEDB commands to connect to the remote server. If the credentials work locally but not remotely then I'd start by ensuring the credentials are correct. There isn't anything different in the WiX custom actions between local and remote servers.
Looking at your database element I would say that you have not added the User attribute to the sql:SqlDatabase so it is creating the database impersonating the current user.
Try:
<sql:SqlDatabase
Id='myDB1'
Database='myDB1'
Server='[DATABASE_SERVER]'
User='SQLUser'
CreateOnInstall='yes'
DropOnInstall='no'
DropOnUninstall='no'
ContinueOnError='no' />