We have been using JBoss Wildfly 8.2 for about 9 months and never had this issue until about two weeks ago (Nov 9th 2015 approximately). We use IIS 7.5 on Windows 2008R2. We serve .war files with Java/JavaScript and also server ColdFusion separately. We connect to MSSQL Server 2012. All of this has been the same with out the errors. We also use Spring and SOLR. We use SiteMinder.
What happens:
User goes to our website, they see a blank screen with the IE11 tab stating 500 error. After some minutes, the user can refresh and the site will be up fine. We have confirmed that two users at once will experience the same thing. As far as suspected browser configuration issues, the user does not need to do anything. They just wait 5-15 mins and then click refresh. Now that I think of it ... maybe the 500 error resolves after I have logged in remotely, but I am not 100% sure.
This does not happen every day, and we have 4 different sites with the exact same software and VM setup (IIS / .war file / etc is the same) and it happens to different sites randomly.
It is ALWAYS the first users in the morning that I have seen so far.
There are no server logs after 02:00 AM ... the site is trying to be accessed at around 6:00 AM. JBoss shows no indication of any errors and everything looks just fine in the back end. Our last error is: 23 Nov 15 02:00:03,915 ERROR [stderr] (Timer-6) java.net.ConnectException: Connection refused: connect -- this is expected since that is DB maintenance time. After that ... nothing.
The Application Logs in Server Manager show the following at the time of the 500 error:
New virus definition file loaded
Then 6+ occurrences:
Failed to initialize the message bus
SiteMinder agent has encountered initialization errors and will not service requests
Server already running Duplicate LLAWP processes not allowed, exiting
Then I log into the server successfully and shortly after I see
The Software Protection service is starting
Then, random or not, the site seems to be up and logs say Software Protection service has started.
What the heck is going on and how do I fix this? It seems coincidental that the 500 server error goes away after I log in, but still...
Related
I know similar questions have been asked before...
I am using SQL server 2005, with SSRS 2005 installed on the same box. (aka. production DB, Report DB/TempDB, Database engine, and SSRS all in the same box).
We have about 200 reports deployed in the box.
SSRS/DB is running on a W2k3 64-bit VM.
Now the problem...
Occasionally almost on a daily basis our users get the 'operation timeout' error (error in XML document....). At first I thought it was a report size problem, but then when I try the Report Manager URL (http://<>/reports), nothing appears on the browser. The only thing I can do is to recycle the Report server IIS pool and it will work again. Everytime when the 'operation timeout' happens, the Report Manager URL will not work, and I can't find any logs in IIS to indicate there's a problem.
I researched on the net and found that some people have put a dummy report as part of the SQL server agent job which runs every 10 minutes from 9-5 to 'warm up' the SSRS. The dummy report made a small connection to the DB on one row from a very small table. The operation timeout problem seems to have disappeared for 95% of time, but it still happens. Strange enough, when the operation timeout problem happens, I notice the dummy report job has also stopped working. In this case, I had to recycle the IIS pool, and start the SQL server job again, and then SSRS will work again (until the same problem happens next time)
The error I got from the SQL server job is:
System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host
However I am totally confused by how the IIS issue on the report server somehow affects the SSRS job. Maybe I am on the wrong track but that's bizzare.
My observation so far is if it takes forever for the Report Manager URL (http://<>/reports) to appear it is a bad sign that something has gone terribly wrong on SSRS.
I have also added a new task which call the SSRS Report Manager http://<>/reports URL using PowerShell in order to 'warm-up' the IIS but it does not seem to make much difference.
Can someone point me to the right direction? Thanks. WM
In the past, after much research, I've found memory allocation for SSRS to be the root of many issues. You can try this.
Add the following into the <Service> node in the rsreportserver.config file
<WorkingSetMaximum>4000000</WorkingSetMaximum>
The file is typically in c:\program files\Microsoft SQL Server\MSRS11.iMIS\Reporting Services\ReportServer
This sets the maximum memory available for the report which also set the minimum memory to 60% of the maximum.
https://msdn.microsoft.com/en-us/library/ms159206(v=sql.110).aspx
I have error in my log for a few weeks, I searched a lot but I couldn't found useful answer.
I did close SQL Server port for public IP, But I have problem yet.
Error: 17806, Severity: 20, State: 14.
SSPI handshake failed with error code 0x8009030c, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The Windows error code indicates the cause of failure. The logon attempt failed [CLIENT: 10.10.3.25]
Time raised: 27 Jan 2015 2:23 PM
It was raised error while this system was off.
The Scenario –
A couple of separate individual Windows ID’s started generating these errors while attempting connections, all other windows logins were working properly. The connections were initially happening through applications, but also occurred through sqlcmd. When logged in to the server locally with the offending ID’s the connections to SQL would succeed.
The Troubleshooting process –
Check all the regular SSPI issues, I wont bore you with the details as they are easily searchable
A relatively easy way of checking the “easy” authentication issues If possible/appropriate is to log into the SQL Server locally with the offending ID and fire up sqlcmd and connect to the server via sqlcmd –Sservername,port –E (by specifying the port you force TCP/IP instead of LPC, thereby forcing the network into the equation)
Verify whether the login is trying to use NTLM or Kerberos (many ways to do this but simplest is to see if there are any other KERBEROS connections on the machine)
SELECT DISTINCT auth_scheme FROM sys.dm_exec_connections
If Kerberos is in use, there are a few additional things to verify related to SPN’s, since only NTLM was in use on this server I skipped that
Determine if the accounts were excluded from connecting to the machine through the network through a group policy or some other AD setting
After all of these checked out OK, I began to try and figure out what the error code 0x8009030c meant, turns out, its fairly obvious what the description is : sec_e_logon_denied. This description was so helpful I thought about making this server into a boat anchor but, luckily for my employer the server room is located many miles away and has armed guards.
Since I knew we could logon locally to the SQL Server with the ID that SQL was rejecting with logon denied something else was trying to make my life miserable.
We didn’t have logon failure security auditing turned on so, I had no way of getting a better error description, As luck would have it though this would prove instrumental in finding the root cause. To get a better error message, I found this handy KB article detailing steps needed to put net logon into debug mode.
Say hello to my new best friend! — nltest.exe
After downloading nltest & using it to enable netlogon debugging on the SQL Server, I got this slightly better message in the netlogon.log file
06/15 14:15:39 [LOGON] SamLogon: Network logon of DOMAIN\USER from Laptop Entered
06/15 14:15:39 [CRITICAL] NlPrintRpcDebug: Couldn’t get EEInfo for I_NetLogonSamLogonEx: 1761 (may be legitimate for 0xc0000064)
06/15 14:15:39 [LOGON] SamLogon: Network logon of DOMAIN\USER from Laptop Returns 0xC0000064
The error code 0XC0000064 maps to “NO_SUCH_USER”
Since I was currently logged in to the server with the ID that was returning no such user, something else was obviously wrong, and luckily at this point I knew it wasn’t SQL.
Running “set log” on the server revealed that a local DC (call it DC1) was servicing the local logon request.
After asking our AD guys about DC1 and its synchronization status, as well as whether the user actually existed there, everything still looked OK.
After looking around a bit more I discovered this gem of a command for nltest to determine which DC will handle a logon request
C:\>nltest /whowill:Domain Account
[16:32:45] Mail message 0 sent successfully (\MAILSLOT\NET\GETDC579)
[16:32:45] Response 0: DC2 D:Domain A:Account (Act found)
The command completed successfully
Even though this command returned “act found” it was returning from DC2. (I dont exactly understand why the same account would authenticate against 2 different DC’s based on a local desktop login or a SQL login but it apparently can)
After asking the AD guys about DC2 the light bulbs apparently went off for them as that server actually exists behind a different set of firewalls, in a totally different location. While DC2 would return a ping, the console wouldn’t allow logons for some reason. After a quick reboot of DC2, and some magic AD pixie dust (I am not an AD admin, if it wasn’t totally obvious from my newfound friend nltest) the windows Id’s that were having trouble started authenticating against DC3 and our SSPI errors went away.
Interesting tidbit — During troubleshooting, I found that this particular SQL Server was authenticating accounts against at least 5 different DC’s. Some of this might be expected since there are different domains at play but, I haven’t heard a final answer from the AD guys about whether it should work that way.
The solution
Reboot the misbehaving DC, of course there may be other ways to fix this by redirecting requests to a different DC without a reboot but, since it was misbehaving anyway, and the AD experts wanted to reboot so we went with that. A reboot of SQL would have likely solved this problem too but, I hate reboot fixes of issues, they always seem to come back!
reference
I am developing a WCF service, so far I have added only a few simple interfaces. Testing the service in the VS debug environment, all is well.
When I published the service to IIS 7 for further testing, all was fine, at least after I added db_datareader and db_datawriter permissions to the relevant databases for user IIS APPPOOL\ASP.NET v4.0.
Now the service is failing silently. Calling the service through a browser, I get the following message: "The server encountered an error processing the request. See server logs for more details." Fiddler says this is a result of an HTTP 400 error message (Bad Request).
I rolled back the code to return a hard-coded value, so I am sure that the issue is at the DB level, not the IIS 7 server installation.
The problem is that the Event Viewer shows no meaningful error messages. This despite the fact that all the code is surrounded by try/catch, with any exception caught going to the event viewer.
There is one message stating "Starting up database 'ReportServer$SQLEXPRESSTempDB'", but as far as I can tell that appears every 10 minutes, without reference to any attempts to access the database. Just to make sure, I gave the .NET 4 user R/W permission to access that DB as well.
In addition, I don't see any messages in the SQL Server Logs.
How can I debug this?
The problem was that db_datareader and db_datawriter permissions are not enough to run stored procedures. I upgraded the permissions and all was good.
The only thing I don't understand is why the exception wasn't written to the event log.
I have an MVC3 application hosted by third party hosting provider. The site has been running well for the past 3 months without any problems. Today suddenly the Application started throwing following Exception as recorded in my logs part of which is shown below.
System.Data.ProviderIncompatibleException: The provider did not return
a ProviderManifestToken string. --->
System.Data.SqlClient.SqlException: Timeout expired. The timeout
period elapsed prior to completion of the operation or the server is
not responding.
The message is self explanatory and I first thought I should increase the connect timeout, but then the exception was still thrown suggesting the other part (Server Not Responding). I contacted my hosting provider and he said there was nothing wrong on his part. So I am stuck with a down website and don't know what to do.
Any ideas why the provider is throwing the exception listed above. Also, is it possible for me to remotely connect to the database on the hosting server with limited authority. Any tools for that ? I don't have an exposure in database subject, except for application programming.
This occurs due to the Timeout, the default timeout is 30 seconds, for time out there are 2 common reasons.
Long running tasks or uncommitted transactions. Refer to the Timeout expired to know about this.
I have a SQL Server Reporting Services 2005 installation that has worked great for a few years now, but has been giving me occasional glitches recently.
I'm using the default reporting services website instead of embedding reports into my application, which has proven a very efficient way to deploy the reports. The OS is Windows Server 2003 and the reports are being viewed on IE8 running XP.
It's not a big deal. Refreshing the browser brings the report right up each time. I guess it might be a cacheing issue. It does scare users, so it would be nice to squash this bug.
Any ideas?
Server Error in '/Reports' Application.
Execution 'j0cnbcqzv5lxg1imzqouqxya' cannot be found (rsExecutionNotFound)
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: Microsoft.Reporting.WebForms.ReportServerException: Execution 'j0cnbcqzv5lxg1imzqouqxya' cannot be found (rsExecutionNotFound)
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[ReportServerException: Execution 'j0cnbcqzv5lxg1imzqouqxya' cannot be found (rsExecutionNotFound)]
Microsoft.Reporting.WebForms.ServerReport.GetExecutionInfo() +289
Microsoft.Reporting.WebForms.ServerReport.SetExecutionId(String executionId, Boolean fullReportLoad) +120
Microsoft.Reporting.WebForms.ServerReport.LoadFromUrlQuery(NameValueCollection requestParameters, Boolean fullReportLoad) +101
Microsoft.Reporting.WebForms.ReportDataOperation..ctor() +321
Microsoft.Reporting.WebForms.HttpHandler.GetHandler(String operationType) +458
Microsoft.Reporting.WebForms.HttpHandler.ProcessRequest(HttpContext context) +56
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +181
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75
Version Information: Microsoft .NET Framework Version:2.0.50727.3603; ASP.NET Version:2.0.50727.3082
Try this (extending the "report session" timeout)
http://blogs.msdn.com/b/jgalla/archive/2006/10/11/session-timeout-during-execution.aspx
Here is a thread that might help...not sure if you tried it already.
In this case updating Microsoft.ReportViewer.WebForms.dll was a fix.
I also saw where it could be caused by a problem with NT Authority\Network Service account.
A quick "Google" with SSRS 2005 Execution "cannot be found" produced quite a few other options as well.
HTH
Take a look here :
http://blogs.msdn.com/jgalla/archive/2006/10/11/session-timeout-during-execution.aspx
This can happen when a report takes a long time to run, using a script with rs.exe to increase the RS session lifetime may well help.
I have answered this question in this thread.
It is to do with the execution context in the Session expiring.
In my case, this error ocurred only in the Browsers Internet Explorer 8 and Internet Explorer 9 with reports that takes more than 20 minutes for execute.
I solved this problem optimizing the sql query procedure ( before optimize took 30 minutes to execute ) and now takes 3 minutes.
I tried to configure increasing the session timeout in ReportManager configuration but anything solve the problem.
This error is weird because in Google Chrome and Firefox works fine!
the reason of this issue may be below:
1 - > your login has expired OR
2 - > The server on which SSRS report deployed, below services has stopped
a - > MS SQL SERVER
b - > SQL Server Agent is not running
to resolve go to the server and start these services
None of the aforementioned things worked for me. My reports aren't working from the Report browser interface, even though it used to. It does however work from the server URL.
eg) The /ReportServer context works fine, and NOT the /Reports context whilst browsing to reports.
This is ok since I can still upload reports, and our applications use the server URL.