I am constantly getting Azure SQL Server sessions with (blocked by) "Blk By" column of -5.
What could this session be please? I have searched for negative number SPID and I can see information for -2, -3, -4 but not for -5.
(I have removed identifiable information for hostname, login dbname columns)
Copied below is a listing from sp_who2
According to the documentation for blocking_session_id
-5 = Session ID of the blocking latch owner could not be determined because it is not tracked for this latch type (for example, for an SH latch).
Edit: I came across this from a Microsoft engineer. The bit I found interesting:
A blocking session id of -5 alone does not indicate a performance problem. The addition of -5 is just an indication that the session is waiting on an asynchronous action to complete where-as prior to the addition, the same session wait would have showed blocking session = 0 but was still in a wait state.
Related
Consider the use case where we need to find the pattern for a attack like 10 failed logons from the same device and same username followed by a success logon from different device but same username. This should happen within 10 mins.
Let us say we have 10 login failed windows events with user A as username and B as devicename and we have a success logon from user A with different device C, we should raise an alert. Please let me know how flink CEP can be used to solve the case.
This is rather similar to Apache Flink - Matching Fields with the same value. In this case you might try MATCH_RECOGNIZE with something this:
PARTITION BY user
...
PATTERN (F{10} S) WITHIN INTERVAL '10' MINUTE
DEFINE
F.status = 'failure' AND (LAST(F.device, 1) IS NULL OR F.device = LAST(F.device, 1)),
S AS S.status = 'success' AND S.device <> LAST(F.device, 1)
The idea is to check that each new F is for the same device as the previous one, and S is for a different device.
BTW, in practice you might rather specify F{10,} so that the pattern matches 10 or more failed attempts in a row, rather than exactly 10.
Well known issue when many clients query on a sqlite database : database is locked
I would like to inclease the delay to wait (in ms) for lock release on linux, to get rid of this error.
From sqlite-command, I can use for example (4 sec):
sqlite> .timeout 4000
sqlite>
I've started many processes which make select/insert/delete, and if I don't set this value with sqlite-command, I sometimes get :
sqlite> select * from items where code like '%30';
Error: database is locked
sqlite>
So what is the default value for .timeout ?
In Perl 5.10 programs, I also get sometimes this error, despite the default value seems to be 30.000 (so 30 sec, not documented).
Did programs actually waited for 30 sec before this error ? If yes, this seems crasy, there is at least a little moment where the database is free even if many other processes are running on this database
my $dbh = DBI->connect($database,"","") or die "cannot connect $DBI::errstr";
my $to = $dbh->sqlite_busy_timeout(); # $to get the value 30000
Thanks!
The default busy timeout for DBD::Sqlite is defined in dbdimp.h as 30000 milliseconds. You can change it with $dbh->sqlite_busy_timeout($ms);.
The sqlite3 command line shell has the normal Sqlite default of 0; that is to say, no timeout. If the database is locked, it errors right away. You can change it with .timeout ms or pragma busy_timeout=ms;.
The timeout works as follows:
The handler will sleep multiple times until at least "ms" milliseconds of sleeping have accumulated. After at least "ms" milliseconds of sleeping, the handler returns 0 which causes sqlite3_step() to return SQLITE_BUSY.
If you get a busy database error even with a 30 second timeout, you just got unlucky as to when attempts to acquire a lock were made on a heavily used database file (or something is running a really slow query). You might look into WAL mode if not already using it.
I have a use case where a large no of logs will be consumed to the apache flink CEP. My use case is to find the brute force attack and port scanning attack. The challenge here is that while in ordinary CEP we compare the value against a constant like "event" = login. In this case the Criteria is different as in the case of brute force attack we have the criteria as follows.
username is constant and event="login failure" (Delimiter the event happens 5 times within 5 minutes).
It means the logs with the login failure event is received for the same username 5 times within 5 minutes
And for port Scanning we have the following criteira.
ip address is constant and dest port is variable (Delimiter is the event happens 10 times within 1 minute). It means the logs with constant ip address is received for the 10 different ports within 1 minute.
With Flink, when you want to process the events for something like one username or one ip address in isolation, the way to do this is to partition the stream by a key, using keyBy(). The training materials in the Flink docs have a section on Keyed Streams that explains this part of the DataStream API in more detail. keyBy() is the roughly same concept as a GROUP BY in SQL, if that helps.
With CEP, if you first key the stream, then the pattern will be matched separately for each distinct value of the key, which is what you want.
However, rather than CEP, I would instead recommend Flink SQL, perhaps in combination with MATCH_RECOGNIZE, for this use case. MATCH_RECOGNIZE is a higher-level API, built on top of CEP, and it's easier to work with. In combination with SQL, the result is quite powerful.
You'll find some Flink SQL training materials and examples (including examples that use MATCH_RECOGNIZE) in Ververica's github account.
Update
To be clear, I wouldn't use MATCH_RECOGNIZE for these specific rules; neither it nor CEP is needed for this use case. I mentioned it in case you have other rules where it would be helpful. (My reason for not recommending CEP in this case is that implementing the distinct constraint might be messy.)
For example, for the port scanning case you can do something like this:
SELECT e1.ip, COUNT(DISTINCT e2.port)
FROM events e1, events e2
WHERE e1.ip = e2.ip AND timestampDiff(MINUTE, e1.ts, e2.ts) < 1
GROUP BY e1.ip HAVING COUNT(DISTINCT e2.port) >= 10;
The login case is similar, but easier.
Note that when working with streaming SQL, you should give some thought to state retention.
Further update
This query is likely to return a given IP address many times, but it's not desirable to generate multiple alerts.
This could be handled by inserting matching IP addresses into an Alert table, and only generate alerts for IPs that aren't already there.
Or the output of the SQL query could be processed by a de-duplicator implemented using the DataStream API, similar to the example in the Flink docs. If you only want to suppress duplicate alerts for some period of time, use a KeyedProcessFunction instead of a RichFlatMapFunction, and use a Timer to clear the state when it's time to re-enable alerts for a given IP.
Yet another update (concerning CEP and distinctness)
Implementing this with CEP should be possible. You'll want to key the stream by the IP address, and have a pattern that has to match within one minute.
The pattern can be roughly like this:
Pattern<Event, ?> pattern = Pattern
.<Event>begin("distinctPorts")
.where(iterative condition 1)
.oneOrMore()
.followedBy("end")
.where(iterative condition 2)
.within(1 minute)
The first iterative condition returns true if the event being added to the pattern has a distinct port from all of the previously matching events. Somewhat similar to the example here, in the docs.
The second iterative condition returns true if size("distinctPorts") >= 9 and this event also has yet another distinct port.
See this Flink Forward talk (youtube video) for a somewhat similar example at the end of the talk.
If you try this and get stuck, please ask a new question, showing us what you've tried and where you're stuck.
In SQL XE for sp/rpc/stmt completed events it will be great if we can include wait types like IO/Network waits etc. Just like we can see reads/cpu/duration, If we can also gets other resource waits, we can get a good idea why sql is slow during scenarios where the duration/CPU is high and reads are low.
You can actually track the wait statistics of a query but the other way around - by tracking the wait statistics themselves. Take a look at the following snippet and the result image:
CREATE EVENT SESSION [Wait Statistics] ON SERVER
ADD EVENT sqlos.wait_info
(
ACTION(sqlserver.database_name,sqlserver.session_id,sqlserver.sql_text)
WHERE (
opcode = N'End'
AND duration > 0
)
),
ADD EVENT sqlos.wait_info_external
(
ACTION(sqlserver.database_name,sqlserver.session_id,sqlserver.sql_text)
WHERE (
opcode = N'End'
AND duration > 0
)
)
Here we capture the end of every wait (this is done because at this point SQL Server knows the duration of the statistic, so we can output it int he result) that has has duration greater than 0. In the ACTION part we retrieve the database name, the text of the query that caused the statistic and the session id of the query.
Beware though. Tracking wait statistics through Extended Events(and not through sys.dm_os_wait_stats that collects aggregate data) can generate a ton of data overwhelmingly fast. Should you choose this method, you should define very carefully which wait statistics you want to keep track of and what from what duration on the statistic causes you problem.
How can you find out what are the long running queries are on Informix database server? I have a query that is using up the CPU and want to find out what the query is.
If the query is currently running watch the onstat -g act -r 1 output and look for items with an rstcb that is not 0
Running threads:
tid tcb rstcb prty status vp-class name
106 c0000000d4860950 0 2 running 107soc soctcppoll
107 c0000000d4881950 0 2 running 108soc soctcppoll
564457 c0000000d7f28250 c0000000d7afcf20 2 running 1cpu CDRD_10
In this example the third row is what is currently running. If you have multiple rows with non-zero rstcb values then watch for a bit looking for the one that is always or almost always there. That is most likely the session that your looking for.
c0000000d7afcf20 is the address that we're interested in for this example.
Use onstat -u | grep c0000000d7afcf20 to find the session
c0000000d7afcf20 Y--P--- 22887 informix - c0000000d5b0abd0 0 5 14060 3811
This gives you the session id which in our example is 22887. Use onstat -g ses 22887
to list info about that session. In my example it's a system session so there's nothing to see in the onstat -g ses output.
That's because the suggested answer is for DB2, not Informix.
The sysmaster database (a virtual relational database of Informix shared memory) will probably contain the information you seek. These pages might help you get started:
http://docs.rinet.ru/InforSmes/ch22/ch22.htm
http://www.informix.com.ua/articles/sysmast/sysmast.htm
Okay it took me a bit to work out how to connect to sysmaster. The JDBC connection string is:
jdbc:informix-sqli://dbserver.local:1526/sysmaster:INFORMIXSERVER=mydatabase
Where the port number is the same as when you are connecting to the actual database. That is if your connection string is:
jdbc:informix-sqli://database:1541/crm:INFORMIXSERVER=crmlive
Then the sysmaster connection string is:
jdbc:informix-sqli://database:1541/sysmaster:INFORMIXSERVER=crmlive
Also found this wiki page that contains a number of SQL queries for operating on the sysmaster tables.
SELECT ELAPSED_TIME_MIN,SUBSTR(AUTHID,1,10) AS AUTH_ID,
AGENT_ID, APPL_STATUS,SUBSTR(STMT_TEXT,1,20) AS SQL_TEXT
FROM SYSIBMADM.LONG_RUNNING_SQL
WHERE ELAPSED_TIME_MIN > 0
ORDER BY ELAPSED_TIME_MIN DESC
Credit: SQL to View Long Running Queries