I'm getting an error while syncing using ntp by running the command
sudo ntpd -gq.
28 Sep 05:19:30 ntpd[1919]: receive:
KoD packet from 172.19.79.12 has inconsistent xmt/org/rec timestamps. Ignoring.
What does it mean? How can I fix it?
It means your server has received the Kiss-o-Death
A client or peer receiving a kiss-of-death packet performs a set of
sanity checks to minimize security exposure. If this is the first
packet received from the server, the client assumes an access denied
condition at the server. It updates the stratum and reference
identifier peer variables and sets the access denied (test 4) bit in
the peer flash variable. If this bit is set, the client sends no
packets to the server. If this is not the first packet, the client
assumes a client limit condition at the server, but does not update
the peer variables. In either case, a message is sent to the system
log.
Furthermore your log tells you exactly why it happened;
172.19.79.12 has inconsistent xmt/org/rec timestamps. Ignoring.
The time source 172.19.79.12 is broken and is being ignored. You need to check that server for further answers.
As you didn't give your ntp.conf file I can only presume you are using one server or peer statement. If that's the case you should take some time to read up on ntp to see why that's a bad thing and (like now) it breaks your timing.
Related
I am trying to implement fault tolerance and trying to collect a list of SQL error numbers that are considered transient. but I keep running into error numbers that have multiple documentations.
Consider error number 10053:
(1) A transport-level error has occurred when receiving results from the server (from here)
(2) Could not convert the data value due to reasons other than sign mismatch or overflow.
(from here)
It gets weirder. Consider error number 11001:
(1) Non-NULL value successfully returned.
(2) An error has occurred while establishing a connection to the server....
Both from the same Microsoft docs page adjacent to each other.
What's going on?
As #Jeroen Mostert said,
Error 10053 means that an established connection has been dropped.
An established connection was aborted by the software in your host machine.
The TCP/IP Connection was aborted by Windows. This was possibly due to a data transmission timeout or protocol error.
The virtual circuit was terminated due to a time-out or other failure. The application should close the socket as it is no longer usable.
SQL server error 11001 occurs when the SQL Server can't be found on Network.
This happens if the IP address is not reachable or the TCP port number is either not open or is not the correct port number or is blocked by a firewall
This error mainly occurs when the SQL Server client can't connect to the server. This may happen when the client cannot resolve the name of the server or the name of the server is incorrect.
So we should check firewall list in Azure sql first.
In our production environment there are sometimes peaks of ORA-12599 errors the connection works otherwise.
The description is not quite clear to me.
Cause: The data received is not the same as the data sent.
Action:
Attempt the transaction again. If error persists, check (and correct)
the integrity of your physical connection.
https://www.oraexcel.com/database-oracle-11gR2-ORA-12599
Does that mean there is a problem with my network cable? Or any other Hardware?
Because our DBA said the error can be caused by a configuration issue on the client side.
Edit:
It started without any changes on the client or server side. And there are spikes. 80 errors in an hour and then no errors for 4 hours while the load is more or less constant
It can mean a mismatch in how the client and the server are wishing to communicate (in encrypted fashion). For example, trying to connect to an 11g database with 10g JDBC drivers will get this because the encryption implementation changed between these versions.
Check your local (ie, client) sqlnet.ora file - you may need to opt for a different method or change the order of them.
The error can be typically be ignored (its informational) but using the latest drivers will normally resolve the issue.
We had sporadic ORA-12599 errors and also ORA-24300 ("bad value for mode") in a Python web app using Flask and uwsgi. It turned out to be caused by multiple worker processes writing and reading to/from the same TCP connection to the DB, giving chaotic communications on the socket.
In our setup this was caused by creating the TCP connections to the DB before uwsgi forked the workers, and the file descriptors being inherited. We fixed this by using uwgsi's lazy-apps mode.
See also this question.
I'm trying to write a TCP client with write function following this: guide.
I shut down my network and try to send a PING package with 2 bytes of data.
Since it's a TCP client, I expect some kind of error returned because there is no network at all and the server won't be able to receive the message, hence no ACK packet will be returned. However, write function returns 2 bytes were written, seem like it doesn't verify ACK packet from server side at all.
How could I check for ACK to make sure whether my message is sent successfully and server receive it?
If you run both client and server on the same machine they'll communicate through the loopback interface, so unplugging or deactivating the outbound network interface will not change anything. Your server and client communicate happily with each other. If there truly isn't any connection between server and client the connect() should've failed with errno set to ENETUNREACH or ECONNREFUSED if the server refuses the connection. So what you probably want is to run the client on your machine and the server either another computer or in a VM with a bridged network. Note that write() successfully returning only means that the bytes are buffered to be send. It may even return fewer bytes than your message contains, indicating the buffer is currently full and you need to try sending the remaining bytes later.
The best way to analyze protocol implementations is using a tool like wireshark to see what packages are actually sent over the wire (or through the loopback interface).
I had a perfectly working SQL Server Service Broker this morning, until I tested how it recovers from crashing.
I forced a system shutdown on the sender during a messaging session between servers over a network. I was sending binary messages of about 5mb size. There are automatic procedures for sending, replying and receiving messages and ending conversations from both sides in place and my setup uses certificates for security.
I am now unable to send any messages from the server side.
Both sides of the messaging chain have queues on and it does not seem like poison message handling would be causing this. The sender side accepts new messages but is not sending them.
The sender side transmission queue has messages with transmission_status
The Service Broker endpoint cannot listen for connections due to the following error: '10013(An attempt was made to access a socket in a way forbidden by its access permissions.)'.
Running ALTER ENDPOINT myendpoint STATE = STARTED returns the same error as above.
Running select * from sys.endpoints shows the endpoint with state_desc = STARTED anyhow..
Running select state_desc from [sender_database].sys.conversation_endpoints shows state_desc = CONVERSING for all results.
Running SELECT COUNT(*) FROM dbo.sender_queue returns 0.
There is no other traffic to the port my endpoint is using, at least not any that is visible with netstat or the TCPView tool. The ports have rules to allow traffic from the firewall and sqlagent and sqlsrvr processes also have extra rules to be allowed.
Using ssbdiagnose tool with ssbdiagnose -level info configuration from service... from the sender side shows a (not new) error
The route for service sender_service is classified as REMOTE. This will result in the message being forwarded.
along with some other errors about certificates that have always been there even when messaging was working. Ssbdiagnose with RUNTIME flag shows nothing at all.
Ssbdiagnose from the target side now says an exception occurs during connection. The target database also has a couple of reply messages stuck in the transmission queue with an empty transmission_status.
Edit: Seems that occasionally the status on the target side changes to the error 10060 connection failed...
What more can I do to diagnose the problem and fix it?
Edit: I tried changing the port the endpoint uses but the same error is thrown.
Edit: I am able to ping the servers from each other. Ssbdiagnose with RUNTIME option on target side says it cannot find the connection to the SQL Server that corresponds to the routing address of my sender endpoint/database.
The Service Broker endpoint cannot listen for connections due to the following error: '10013(An attempt was made to access a socket in a way forbidden by its access permissions.)'
WSAEACCESS (10013) is a rather unusual socket listen error. I never encountered it before. A quick search reveals KB3039044: Error 10013 (WSAEACCES) is returned when a second bind to a excluded port fails in Windows which is an acknowledged bug in Windows Server 2008R2, 2012 and 2012R2 when excluding a range of ports (netsh ... add excludedportrange ...). So my first question is, are you on one of the affected server OSes and are you actually using a network port exclusion range?
I strongly urge you to open a Microsoft support case for this issue and follow up with them, making sure networking guys are involved (again, WSAEACCESS is rather unusual symptom). This is not one of the usual issues and it is difficult to diagnose over forums discussion.
Hi Socket Programming experts,
I am writing a proxy server on Linux for SQL server 2005/2008 running on Windows.
The proxy is coded using bsd sockets and in C, and it is working fine with a problem described below.
When I use a database client (written in JAVA, and running on a Linux box) to fire queries (with a concurrency of 100 or more) directly to the Database server, not experiencing connection resets. But through my proxy I am experiencing many connection resets.
Digging deeper I came to know that connection from 'DB client' to 'Proxy' always succeeds
but when the 'Proxy' tries to connect to the DB server the connection fails, due to the SYN packet getting RST,ACK.
That was to give some background. The question is :
Why does sometimes SYN receives RST,ACK?
DB client(linux) to Server(windows) ----> Works fine
DB client(linux) to Proxy(Linux) to Server(windows) -----> problematic
I am aware that this can happen in "connection refused" case but this definitely is not that one. SYN flooding might be another scenario, but that does not explain fine behavior while firing to Server directly.
I am suspecting some socket option setting may be required, that the client does before connecting and my proxy does not. Please put some light on this. Any help (links or pointers) is most appreciated.
Additional info:
Wrote a C client that does concurrent connections, which takes concurrency as an argument. Here are my observations:
-> At 5000 concurrency and above, some connects failed with 'connection refused'.
-> Below 2000, it works fine.
But the actual problem is observed even at a concurrency of 100 or more.
Note: The problem is time dependent sometimes it never comes at all and sometimes it is very frequent and DB client (directly to server) works fine at all times .
SQL Server needs worker threads to accept incoming connections. If your server is worker starved (which can be easily diagnosed by a high number of entries in sys.dm_os_tasks in PENDING state) then attempting to open new connection will fail. So likely what I suspect it happens is that you're pushing to the server more workload that it can handle. You need to optimize the workload or get a beefier server.
Clients like Java client make effective use of connection pooling and, even under a high load, do not need to open new connection hence you do not see this problem, instead you see only delays in requests completion.
A listening socket keeps queue of established connections and connections in establishment process (e.g. SYN got, SYNACK replied, but not ACK from client yet). If a established queue overflows, IP stack reaction differs on OS. The most traditional approach was to ignore newcoming SYNs, waiting when userland accept()s and frees a slot in the queue. With SYN flooding attacks in mid-90s, the new method was invented named "SYN cookies" which drops need for establishment queue totally, in cost of need to support special TCP option.
OTOH I have heard that Windows stacks change their behavior - under some conditions, the reaction to queue overflow is RST response. In earlier stacks (e.g. Win95) this was the main response and client side was correspondingly changed to ignore RST response to SYN:(
That's why I guess that some proxy host feature triggers RST in Windows stack.
Another guess is that DB server closes listening socket at all under some condition (e.g. detected overload peak) which appears only with proxy.
When the SYN receives the RST response, it should not be the problem of SQL-SERVER.
Because the application is able to accept the socket only after the tcp handshake finished.
Is there any device between the Proxy and the SQL-Server machines?
Try to make sure that the RST response come from sql-server machine.
You connection count is far from SYN-FLOOD, I think.