We are facing this below error in Vespa, after restarting the cluster we got this below issue.
1600455444.680758 10.10.000.00 1030/1 container Container.com.yahoo.filedistribution.fileacquirer.FileAcquirerImpl info Retrying waitFor for file 'e0ce64d459828eb0': 103 -- Request timed out after 60.0 seconds.
1600455446.819853 10.10.000.00 32752/146 configproxy configproxy.com.yahoo.vespa.filedistribution.FileReferenceDownloader info Request failed. Req: request filedistribution.serveFile(e0ce64d459828eb0,0)\nSpec: tcp/10.10.000.00:19070, error code: 103, set error for connection and use another for next request
We faced this issue second time, earlier we kept it ideal and it was resolved automatically, but this time it is persistent.
Looks like the configproxy is unable to talk to the config server (which is listening to port 19070 on the same host: Spec: tcp/10.10.000.00:19070). Is the config server really runnning and listening on port 19070 on this host? Try running the vespa-config-status script to see if all is well with the config system
Related
This code times out in a CakePHP 4.3.3 template with a
503 - Service Temporarily Unavailable.
In a none-CakePHP script it works [it shows after 6 minutes DONE].
ini_set("max_execution_time", "0");
sleep(360);
die("DONE");
The template is called like this:
$this->render("/General/index", "Study");
I didn't find anything regarding a max execution time for rendering but as I can clearly reduce it to the above MCVE, it must be something CakePHP relevant because in CakePHP it stops after 60 seconds, in a PHP file timeout.php, placed directly in the webroot directory, it works.
What am I doing wrong, what I haven't set up properly?
I think, I should present the log entry:
[proxy_fcgi:error] [pid 1813:tid 140621247407872] (70007)The timeout
specified has expired: [client XXXX:56968] AH01075: Error dispatching
request to :
The server is running under Apache/nginx.
I don't understand why a stand-alone script on the same server and same domain works, but under CakePHP it ran into the timeout - with the exactly identical 3 lines of code.
The solution was - after investigating the log error - the Apache2 Timeout.
Additionally, I had to set the nginx timeout settings:
proxy_connect_timeout 3600s;
proxy_send_timeout 3600s;
proxy_read_timeout 3600s;
fastcgi_send_timeout 3600s;
fastcgi_read_timeout 3600s;
After applying the above settings for Apache and nginx it works also under CakePHP.
ABP version 4.4 APP with identity server with angular EF core
my angular client send the username and password to identity server that is part of my abp application (mean same URL for api and identity server) that returns me token but after very next request for "api/abp/application-configuration" return with CORs error.
same application work for QA local environment with local config.
same call for "api/abp/application-configuration" work on app start because that time token is not attached in request header
i check my app setting CorsOrigins are matching with IdentityServerClientCorsOrigins
no space issue both angular and front end on https but with different port. you can verify with logs. one thing is strange in log no protocol attach # respond. (domain:1000)
I check and confirm that this is Authentication issue. by comment these two line and Authorize attribute from class
My application works fine. no issue for cors
app.UseAuthentication();
app.UseJwtTokenMiddleware();
system log
2021-09-04 10:42:24.731 -04:00 [INF] Request starting HTTP/2 OPTIONS https://domain:1000/.well-known/openid-configuration - -
2021-09-04 10:42:24.745 -04:00 [INF] CORS policy execution successful.
2021-09-04 10:42:24.751 -04:00 [INF] Request finished HTTP/2 OPTIONS https://domain:1000/.well-known/openid-configuration - - - 204 - - 19.7919ms
2021-09-04 10:42:24.793 -04:00 [INF] Request starting HTTP/2 GET https://domain:1000/.well-known/openid-configuration - -
2021-09-04 10:42:24.794 -04:00 [INF] CORS policy execution successful.
2021-09-04 10:42:46.050 -04:00 [ERR] Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'System.String'.
System.IO.IOException: IDX20804: Unable to retrieve document from: 'System.String'.
System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (domain:1000)
System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
It is not a CORS issue, instead it is because your API (the UseJwtTokenMiddleware part) can't reach your IdentityServer over HTTP(s). I guess you got some DNS/Certificate/Network issue to resolve.
I have a API web server, there are two version api coexist, they are routed by url path, which is /shine/v7 and /shine/v8.
I made this by haproxy, but when I request /shine/v7/admin, it sometimes went to shine_v8_backend, and sometimes went to shine_v7_backend, I don't know why this happend, is anyone can help me?
there is my haproxy.conf
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
daemon
defaults
log global
option http-server-close
frontend http *:5000
mode tcp
timeout client 86400000
default_backend shine_v8_backend
acl shine_v7 path_dir /shine/v7
use_backend shine_v7_backend if shine_v7
backend shine_v8_backend
mode tcp
option httpclose
balance roundrobin
timeout server 86400000
timeout connect 5000
server host_0 127.0.0.1:5001
backend shine_v7_backend
mode tcp
option httpclose
balance roundrobin
timeout server 86400000
timeout connect 5000
server host_0 127.0.0.1:5002
I try to request /shine/v7/admin many times, there is the logs
$ sudo haproxy -f haproxy.conf -d
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
00000000:http.accept(0004)=0006 from [127.0.0.1:55026]
00000000:shine_v7_backend.srvcls[0006:0007]
00000000:shine_v7_backend.clicls[0006:0007]
00000000:shine_v7_backend.closed[0006:0007]
00000001:http.accept(0004)=0006 from [127.0.0.1:55028]
00000001:shine_v8_backend.srvcls[0006:0007]
00000001:shine_v8_backend.clicls[0006:0007]
00000001:shine_v8_backend.closed[0006:0007]
00000002:http.accept(0004)=0006 from [127.0.0.1:55030]
00000002:shine_v7_backend.srvcls[0006:0007]
00000002:shine_v7_backend.clicls[0006:0007]
00000002:shine_v7_backend.closed[0006:0007]
00000003:http.accept(0004)=0006 from [127.0.0.1:55032]
00000003:shine_v8_backend.srvcls[0006:0007]
00000003:shine_v8_backend.clicls[0006:0007]
00000003:shine_v8_backend.closed[0006:0007]
00000004:http.accept(0004)=0006 from [127.0.0.1:55034]
00000004:shine_v7_backend.srvcls[0006:0007]
00000004:shine_v7_backend.clicls[0006:0007]
00000004:shine_v7_backend.closed[0006:0007]
I have found the problem, It can't use mode tcp here, must use mode http :(
and after fix, the haproxy.conf is
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
daemon
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
frontend http *:5000
timeout client 86400000
acl shine_v8 path_dir /shine/v8
acl shine_v7 path_dir /shine/v7
use_backend shine_v8_backend if shine_v8
use_backend shine_v7_backend if shine_v7
backend shine_v8_backend
option httpclose
balance roundrobin
timeout server 86400000
timeout connect 5000
server host_0 127.0.0.1:5001
backend shine_v7_backend
option httpclose
balance roundrobin
timeout server 86400000
timeout connect 5000
server host_0 127.0.0.1:5002
it works fine.
I am trying to upload some data to my local datastore in appengine.
The command I am using is the next one:
appcfg.py upload_data --config_file="C:\config.yml" --filename="C:\mycsv.csv" --url=http://localhost:8888/remote_api --kind=MyEntity
The problem is that I'm working behind my company proxy and I am getting the next ERROR even trying to connect to the localhost server:
Error Code: 502 Proxy Error. The ISA Server denied the specified Uniform Resource Locator (URL). (12202)
It seems the authentication is ok, but somehow the proxy tries to filter my connection to my own computer.
Some ideas about how can I solve this?
Thanks.
Remove/disable proxy settings of your network then try the above command.
I was facing the similar issue and this issue resolved when i disable my proxy settings.
I am using Tomcat 7.0.29 fronted with Apache 2.2.22 modproxy.
Configured Ajp as the protocol in httpd.conf and AjpNioProtocol in server.xml.
After the server starts, the logs are filled with the following message:
Severe: Invalid message received with signature 20599
com.apache.coyote.ajp.AjpMessage processHeader
There are no requests sent to the web or tomcat server and it still throws that error. Access logs in tomcat and apache show that no request is coming in.
What is causing the invalid message error?
Here is the configuration :
httpd.conf
ProxyPass /wl ajp:// ip : port /wl
ProxyPassReverse /wl ajp:// ip : port /wl
server.xml
<Connector port="port"
protocol="org.apache.coyote.ajp.AjpNioProtocol"
connectionTimeout="20000"
acceptorThreadCount="2"
maxThreads="1600"
redirectPort="8443" />
For me, the problem was simple. I was sending HTTP requests but the connector was configured with AJP protocol. My connector in server.xml was configured like this:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443"/>
But when I changed it to this:
<Connector port="8009" protocol="HTTP/1.1" redirectPort="8443"/>
The error went away.
Hopefully that will help someone with this error.
This may also happen when buffer sizes are not same at both ends: logs mention invalid AJP message and browser receive 400 error code.
I have fixed the situation with both packetSize on AJP connector and ProxyIOBufferSize in Apache2 configuration.
In Tomcat server.xml:
<Connector protocol="AJP/1.3" port="8009"
connectionTimeout="20000"
packetSize="65536"
proxyName="yourproxy.domain.ltd" proxyPort="80"
/>
In Apache2 mod_proxy_ajp configuration, add the statement ProxyIOBufferSize 65536.
It was found that one of the internal processes was calling that port and sending http requests causing the "Invalid message.." error. So I ended up adding an additional http Connector for those internal processes
There are no requests sent to the web or tomcat server and it still throws that error. Access logs in tomcat and apache show that no request is coming in. What is causing the invalid message error?
Just some hint for other people because I forgot the same in one of my configs by accident: The mentioned Connector in server.xml is listening globally, because only port is specified, without any address. The latter is defined to listen globally by default:
By default, this port will be used on all IP addresses associated with the server.
https://tomcat.apache.org/tomcat-7.0-doc/config/http.html
So without any additional firewall or such it might be that bad clients are simply testing for open ports using various protocols, which might or might not be HTTP and therefore resulting in error messages with different signatures. Without very good reasons, there shouldn't be any need to make AJP globally available, especially not in case of a proxy-setup like used by the thread starter.
<Connector address="localhost" port="port"
protocol="org.apache.coyote.ajp.AjpNioProtocol"
connectionTimeout="20000"
acceptorThreadCount="2"
maxThreads="1600"
redirectPort="8443" />
I got a similar message today:
Nov 18, 2016 4:25:00 PM org.apache.coyote.ajp.AjpMessage processHeader
SEVERE: Invalid message received with signature 65524
The root cause of my problem was that selinux wasn't letting apache connect to tomcat. I'm a little confused as to how this error was a result - I'd expect that there would be no connection, period. Best guess, i probably attempted to manually connect to that port with telnet. Doing that certainly gives a similar message.
Regardless, perhaps this selinux reminder will be helpful to someone else who ends up here.