I work on a full stack application that is composed of:
Django (Django Rest Framework)
React
PostgreSQL database
Redis
Celery
It is deployed through docker. Whole application works well and has no bugs that cannot be traced.
However, when I try to delete Project item from database (this is domain specific), I get error 500 and no specific trace.
I figured this bug out on deployed application. While inspecting Networking tab in Developer Tools I found the request and saw 500 return code. However, nothing was returned in returned in Response.
However, I think something should have been returned. Code is as such:
class ProjectCRUD(GenericAPIView):
# [...]
def delete(self, request, pk):
try:
# [...] code that deletes all referenced values and current project
except ProtectedError as e:
return JsonResponse({
"error": "Project still referenced",
"details": str(e)
}, status=400)
except Exception as e:
return JsonResponse({"error": "Wrong project id"}, status=status.HTTP_400_BAD_REQUEST)
return JsonResponse({
'message': f'Project with id {project_id} was deleted successfully!'
}, status=status.HTTP_204_NO_CONTENT)
# [...]
This "Wrong project id" assumption is by all means bad and this will be refactored as soon as this bug is also found. This code makes sure that if exception is raised, it is caught, something is returned with at least some amount of information given. If exception is not caught, return 204.
So I go to the application, I create a new project, try to delete it and error 500 with nothing in Networking appears.
Next step is trying things locally. I start local server using python manage.py runserver. This doesn't go through docker because redis and celery are not used for this feature. I create a new project, try to delete it and console logs writes 204, which means it passed.
I start docker. Repeat process. Everything works, 204 is returned.
Next I check docker logs of deployed application. This is where it starts to be really weird. Backend logs show 204 as it did locally. Frontend logs show 204 as well. However, client (ie browser) in networking displays error 500.
From searching I concluded that the bug happens somewhere between Frontend and client.
My questions are:
any idea why is this happening
where should I look next in order to catch a bug
So the whole application works as expected except for this feature.
Thanks.
Related
I am seeing an odd intermittent authorization failure on publish. My publisher is running on App Engine Standard (Python). Because of that, I am using the "old" python client library. So the code looks like this:
from googleapiclient.discovery import build
build('pubsub','v1').projects().topics().publish(topic=topic,body=body).execute()
This works just fine. The identity gets picked up and everything is authenticated. However, again intermittently, it will stop working and I get 403 forbidden errors. Then later it will start working again (even with the same topic and body). In the meantime, no code changes, no deployments.
I have had to wrap the publish to catch this error, throw it on a task queue and have the request repeat with decaying frequency until it finally starts working again a few hours later. This is OK in the very short term, but obviously this will not work for us.
To summarize, this is on the publish side, GAE Standard ... it works, then stops working, then works again.
Thanks for any insight or help.
It turns out, of course, that in fact there were deployments when I wasn't aware. So I thought, "no code change - no deployments", but there were deployments. And the issue was that the person making these deployments had an old library (or other dependency) for google_api_python_client. Once corrected, pubsub is working just fine.
Recently I have revamped a website which is created on a development server. Then after that i started migrating it onto the main server. Initially I got a unicode error while uploading the database on the live server. I googled it and found a solution on stack overflow itself (#1273 – Unknown collation: ‘utf8mb4_unicode_520_ci’). I used the method suggest by sabba and it worked. Later when I Changed the config file and loaded that link. Its giving me a 503 error.. It error is as follows:
Service Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Additionally, a 503 Service Unavailable error was encountered while trying to use an Error Document to handle the request
Go through the steps and check it,
Enable WP_DEBUG
But since the 503 error often locks you out of your WordPress admin, we shall use WP_DEBUG and WP_DEBUG_LOG, WP_DEBUG_DISPLAY and #ini_set constants available to WordPress.
To enable debug mode in WordPress and write errors to a log file, follow these steps:
1. Open the wp-config.php file
2. Scroll down to where WP_DEBUG is defined. It looks like this define ('WP_DEBUG', false);. If it is missing, we will add it just above the line that says /*That's all, stop editing! Happy blogging.*/
3. Insert the DEBUG magic codes. Just change the above define ('WP_DEBUG', false); code to:
define ('WP_DEBUG', true);
define ('WP_DEBUG_LOG', true);
define ('WP_DEBUG_DISPLAY', false);
#ini_set ('display_errors', 0);
4. Save changes
Now, reload your site to provoke the error. Next, locate a file known as debug.log inside your wp-content folder in your WordPress directory.
This file contains all the errors on your website. If your 503 service unavailable error is caused by a custom code snippet, it will show up somewhere with details of the error.
Eliminate/replace the problematic code and reload your site. If the 503 error persists, the problem could lie in your web server.
I cannot say exactly since when we have this problem, since our project is still in development and we only periodically deploy to test things.
In our most recent test the PubSub authentication does not seem to work anymore. When subscribing to a topic, the following error comes up in the logs:
21:10:29.577 Error subscribing to '123456'-topic: googleapi: Error 401: The request does not have valid authentication credentials., unauthorized
This error is new. It worked fine before. We cannot trace it back to any code change on our part.
This error is also not occuring on the devappserver with the PubSub-Emulator.
The subscriber is in the same project as the PubSub-Service. Again, everything here is happening in one project.
Here is some insight into what we are doing:
client, err := pubsub.NewClient(ctx, "lol123", option.WithTokenSource(google.AppEngineTokenSource(ctx, pubsub.ScopePubSub)), option.WithHTTPClient(urlfetch.Client(ctx)))
[...] more code [...]
_, err := client.NewSubscription(ctx, subname, topic, 0, &pubsub.PushConfig{
Endpoint: endpoint,
})
Pretty standard stuff.
Here are things we tried to play with:
Adding a second scope:
option.WithTokenSource(google.AppEngineTokenSource(ctx, pubsub.ScopePubSub, pubsub.ScopeCloudPlatform)
Using the default tokensource:
tokenSource, _ := google.DefaultTokenSource(ctx, pubsub.ScopePubSub)
option.WithTokenSource(tokenSource)
Adding an endpoint, in case https://experimental.pubsub.googleapis.com/ is somehow used:
option.WithEndpoint("https://pubsub.googleapis.com/")
Also, we switched from the previous cloud.WithTokenSource(...)-stuff to the new option.WithTokenSource(...)-libraries. This migration to the new Cloud-Libraries worked well. However, they did not help with the permission problem.
We also played around with every imaginable combination of Service Accounts and Permissions. E.g., the default Appengine Service Account has been added as Owner to all PubSub-Topics. Remember, that this error is new and it worked before and we cannot trace it back to changes on our part.
Any ideas and suggestions? We'll try pretty much anything at this point. ;-)
My EPiServer application is throwing following error:
A site with siteId is already registered. Call the Reset method firs.
There were no code changes before the exception occurred and there is duplicate of this instance which is running without any problems.
My first try was the most obvious, to change the site id in web.config to something else. This did not help. However changing the configuration restarted application and after refreshing the page it shown another exception:
This slave site failed when validating master license information
After another refresh, third exception appeared:
ClassFactory not initialized
Following refreshes bring back original exception about duplicated site id.
I have no idea what could happen. Searching in google did not bring any results. Every restart of the application (through iisreset or by configuration change) causes the site to throw all those three exceptions, always in the same order: license, class factory, and until next restart duplicated site id.
I will be very grateful for any help that could lead me to the solution.
EDIT: I am using EPiServer 5
EDIT 2: I am now pretty sure that "A site with siteid ..." exception is not relevant here. It is just a consequence of previous fail.
This is what is happening basing on exceptions (it does not seem very logical, but well):
IIS is started
Application reaches static initializer and tries to validate the master license. It fails, exception is thrown and the site is not yet registered (in the database there is no entry made in tblSiteConfig)
The page is refreshed
Application reaches static initializer and successfully validates the license, the site is then registered, the code goes further and fails on plugin initialization method (class factory not initialized)
The page is refreshed
Application reaches static initializer and again successfully validates the license, the site fails because it was already registered
Step 6 is happening until next IIS restart
According to messages, is looks like you are using EPiServer Multisite feature.
First thing to check are /sites/site sections of episerver.config - they should contain different siteId attribute.
Next, in episerver.framework.config file, clear siteHostMapping section, so it should be completely empty:
<siteHostMapping />
Then, start sites one by one.
Possible cause to issue is exception during site startup - so you can still experience similar or the same issue if these errors won't be fixed. Be sure to turn on logging and examine logs.
After you've changed the siteId you need to clear some tables in the database. I don't have a EPi5-database at hand but I recall it's at least tblSite.
I have a fairly big application which went over a major overhaul.
The newer version uses lot of JSONP calls and I notice 500 server errors. Nothing is logged in the logs section to determine the error cause. It happens on JS, png and even jersey (servlets) too.
Searching SO and groups suggested that these errors are common during deployment. But it happens even after hours after deployment.
BTW, the application has become slightly bigger and it even causes deadline exception while starting few instances in few rare cases. Sometimes, it starts & serves within 6-10secs. Sometimes it goes to more than 75secs thereby causing a timeout for the similar request. I see the same behavior for warmup requests too. Nothing custom is loaded during app warmup.
I feel like you should be seeing the errors in your logs. Are you exceeding quotas or having deadline errors? Perhaps you have an error in your error handler like your file cannot be found, or the path to the error handler overlaps with another static file route?
To troubleshoot, I would implement custom error pages so you could determine the actual error code. I'm assuming Python since you never specified what language you are using. Add the following to your app.yaml and create static html pages that will give the recipient some idea of what's going on and then report back with your findings:
error_handlers:
- file: default_error.html
- error_code: over_quota
file: over_quota.html
- error_code: dos_api_denial
file: dos_api_denial.html
- error_code: timeout
file: timeout.html
If you already have custom error handlers, can you provide some of your app.yaml so we can help you?
Some 500s are not logged in your application logs. They are failures at the front-end of GAE. If, for some reason, you have a spike in requests and new instances of your application cannot be started fast enough to serve those requests, your client may see 500s even though those 500s do not appear in your application's logs. GAE team is working to provide visibility into those front-end logs.
I just saw this myself... I was researching some logs of visitors who only loaded half of the graphics files on a page. I tried clicking on the same link on a blog that they did to get to our site. In my case, I saw a 500 error in the chrome browser developer console for a js file. Yet when I looked at the GAE logs it said it served the file correctly with a 200 status. That js file loads other images which were not. In my case, it was an https request.
It is really important for us to know our customer experience (obviously). I wanted to let you know that this problem is still occurring. Just having it show up in the logs would be great, even attach a warm-up error to it or something so we know it is an unavoidable artefact of a complex server system (totally understandable). I just need to know if I should be adding instances or something else. This error did not wait for 60 seconds, maybe 5 to 10 seconds. It is like the round trip for SSL handshaking failed in the middle but the logs showed it as success.
So can I increase any timeout for the handshake or is that done on the browser side?