Quite suddenly, after years of running and developing this application, the log-file shown at Google developers console is filling up with entries marked with the double exclamation mark error, but with no error description when expanded. It has been going on for days now, and is making the logging absolutely useless. Everything else is drowning in these seemingly false error messages.
What can I do, except pay $150 for the right to send them an email?
tl:dr Don't have print statements in your request handler!
With the help of a friendly google person looking at my issue report, I found the trigger for the empty errors in the log file:
Print statements in the request handler was the cause. I had forgotten to remove them after a quick local test. Updating a version with those removed stopped the errors immediately, so it seems pretty clear.
Related
I have recently changed to use custom Go runtime on GAE, and noticed many errors like this from logs:
internal.flushLog: Flush RPC: Call error 3: invalid security ticket: 6c8027dc99b3ed3e
internal.flushLog: Flush RPC: Canceled: (timeout)
The server is still running well, but I have no idea about that error, as well as why it happens.
I'm using a custom Go runtime by using Dockerfile, and App Engine Release is 1.9.37.
Any help to clarify the error would be highly appreciated. Thanks.
This is a known issue with the Go runtime on App Engine Flexible. It tends to happen when a line is logged right before the end of a request/response.
What happens is that when the line is logged it is actually put in a list of log lines to be batched together and sent to the application server as an RPC at periodic intervals. The security ticket is canceled at the end of a request/response which sometimes can happen before the log lines have been flushed. It's harmless, except that you may lose a log line or two. :\
We're actively working on fixing it.
Some requests silently fail in my python app, intermittently and unpredictably. The hallmarks of the failure are:
Request returns a 200, so the client doesn't know there's a problem.
Request does NOT successfully execute on the server.
No logging statements are recorded for the request.
Below is an example from my logs of a bunch of requests which are each supposed to write an entity to the datastore. You can see for the lower, successful request, a blue 'i' is present, indicating that info level logs were recorded. When I examine the datastore, an entity was successfully written for this request.
However, for the failed request, you can see there is just a white box, and there are no logging statements present at all. While the server returned a 200, no entity was written to the datastore for this request.
Has anyone encountered something like this before on App Engine? Any ideas on how to debug it? I've seen it in multiple different apps myself, but I've never been able to figure it out.
EDIT
To clarify, the main problem here is that code doesn't execute, as measured by the failure to write an entity. The spurious 200 and lack of logging is an associated symptom.
From a comment originally, but seems to be the resolution path for this issue:
Given that there are no log statements at all in the line and you appear to unpack the arguments and log them as soon as you enter the handler, this starts to look like an infrastructure/platform issue.
In such a case, it's best to open a public issue tracker issue, with "Type-Production" as a tag, including your app's app id and a timeframe, and as much information about your app and request handler involved as possible, and platform support will pick up the issue in the course of triage.
That said, it's worth examining the handler to make absolutely sure there's no way you could be exiting from the handler and sending a 200 without logging anything or seeing an exception. It all depends on what the code handling the request is capable of, what stack of libraries it's build upon, etc.
We have built a website based on the design of the Kigg project on CodePlex:
http://kigg.codeplex.com/releases/view/28200
Basically, the code uses the repository pattern, with a repository implementation based on Linq-To-Sql. Full source code can be found at the link above.
The site has been running for some time now and just about a year ago we started to get errors like:
There is already an open DataReader associated with this Command which must be closed first.
ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.
These are the closest error examples I can find based on my memory. These errors started to occur when the site traffic started to pick up. After banging my head against the wall, I figured out assumed that the problem is inherit within Linq-To-Sql and how we are using the same connection to call multiple commands in a single web request.
Evenually, I discovered MARS (Multiple Active Result Sets) and added that to the data context's connection string and like magic, all of my errors went away.
Now, fast forward about 1 year and the site traffic has increased tremendously. Every week or so, I will get an error in SQL Server that reads:
A severe error occurred on the current command. The results, if any, should be discarded
Immediately after this error, I receive hundreds to thousands of InvalidCastException errors in the error logs. Basically, this error shows up for each and every call to the Linq-To-Sql data context. Only after I restart the web server do these errors clear up.
I read a post on the Micosoft Support site that descrived my problem (minus the InvalidCastException errors) and stating the solution is that if I'm going to use MARS that I should also use Asncronous Processing=True. I tried this, but it did not solve my problem either.
Not really sure where to go from here. Hopefully someone here has seen and solved this problem before.
I have the same issue. Once the errors start, I have to restart the IIS Application Pool to fix.
I have not been able to reproduce the bug in dev despite trying many different scenarios involving multi-threading, leaving connections open, etc etc.
One possible lead I do have is that amongst the errors in the server Event Log is an OutOfMemoryException for the Application Pool. Perhaps this is the underlying cause of the spurious SQL Datareader errors (a memory leak elsewhere). Although again I haven't been able to reproduce this in dev.
Obviously if you are using a 64 bit OS then this is probably not the cause in your case.
So after much refactoring and re-architecting, we figured out that problem all along is MARS (Multiple Active Result Sets) itself. Not sure why or what happens exactly but MARS somehow gets result sets mixed up and doesn't recover until the web app is restarted.
We removed MARS and the errors stopped.
If I remember correctly, we added MARS to solve the problem where a connection/command was already closed using LinqToSql and we tried to access an object graph that hadn't been loaded. Without MARS, we'd get an error. But when we added MARS, it seemed to not care about it. This is really a great example of us not really understanding what the heck we were doing and we learned some valuable (and expensive) lessons from this.
Hope this helps others who have experienced this.
Thanks to all how have contributed their comments and answers.
I understand you figured out the solution..
Following is not a direct solution to the problem; but it is good for others to take a look at
What does "A severe error occurred on the current command. The results, if any, should be discarded." SQL Azure error mean?
http://social.msdn.microsoft.com/Forums/en-US/bbe589f8-e0eb-402e-b374-dbc74a089afc/severe-error-in-current-command-during-datareaderread
We use custom settings in a SalesForce app. We access it like so:
MySettings__c settings = MySettings__c.getOrgDefaults();
This was working fine, but today the app completely crashed. By that I mean the page doesn't load at all, I just get a white screen telling me an internal error occurred. We traced it down to this line of code - when it is commented out the page loads as well as it can without those settings (but at least it loads).
Running that single line of code in the System Log (using the Execute functionality) also causes a report of Internal System Error. The only thing the system log reports is "FATAL_ERROR Internal Salesforce.com Error." The Apex code modal reports "Internal System Error: 1018505045-332 (-920440070)"
The setting has values for the organization - we've also tried deleting the settings and recreating them to no affect. So far SalesForce has been no help beyond telling us to ask on their website.
This is very frustrating as it was working fine on Friday and today it was broken before anyone touched anything.
What you have there is a platform error. Whenever you get those you should report them to SFDC support and they will be able to see further internal logging to sort it out.
Nothing anyone out here can do to help I am afraid.
Paul
try setting the apiVersion of the affected code back to version 21.0. We had the same issue and making this change has provided an effective workaround.
This was a bug in Salesforce's infrastructure, which has been reported resolved. If you're still seeing this error with API version 22.0, you should create a case with salesforce support.
I'm trying to generate ics-files, see sample here: http://deap.nu/myprogram.ics
It validates alright here: (URL removed: service no longer exists)
but when trying to import it into a google calendar I get the message
"Failed to import events: Unable to process your ical/CSV file."
which isn't very informative. I've tried to slim the file down, and googled a lot but can't find whats wrong. Any input on this is appreciated.
Importing to outlook doesn't work either.
After some sleep and more trial and error I managed to pinpoint the problem to the Organizer elements in the .ics file. So I removed them for now, adding the information to the description instead.
I have another recommandation for a successful import to GCalendar (calendar.google.com). I failed several times to import an ics-File with meaningless error messages.
I tried anything to make the ics-file standard conform, left out all UID rows etc. but nothing helped.
I used Google Chrome, which is my normal every day browser.
I thought: Which browser could be better to use it with Google-Services?
But: When I logged in to google-calendar using Microsoft Edge (Chromium-based) the ics-import just worked out of the box.
Hope it helps others with the same problem...
Then
I just got past a problem with the same description. I could leave Organizer in, but, editing the file with emacs on linux, I noticed that the Description contained carriage returns (^M), whereas the other line breaks had been normalized to linux standard when I had saved the .ics file. I removed the ^Ms and it then imported fine.
I realize this is an old post, but I'm adding this in to help others that might run into issues trying to import an .ics file.
As far as Google Calendar, the errors are not descriptive or outright misleading. I had a permissions error, but that was due to a UID (Unique Event ID) being in the file. As described below from Google's Calendar Community Forum, the solution was to remove that line from the file, save it, then try to import it again. It worked!
some of Google Calendar's error messages are not to be taken at face
value... this particular error often means that your UIDs (unique
event IDs) are not formatted to Google Calendar's liking, so my
question is where was this ics file generated?
I would try opening the file in a good text editor, and deleting every
line starting with UID as they are not needed for inserting new
events, and will cause this error if not formatted correctly.