There is a list of IPs (their quantity is changing every few minutes) stored in a log-file. The logfile only contains unique IPs (sort -u):
IP-Pool.log [Example]:
192.168.1.1
192.168.1.2
10.1.1.0
.
.
I would like to filter occuring connection events with rsyslog and sort them by:
Events where $msg contains one IP of Array[IP-Pool-List]
Events where $msg doesn't contain any IP of Array[IP-Pool-List]
Therefore I would have to put the IPs into an array so they can be compared but I haven't found a way to do so.
if $msg contains 'connection-event-id' then {
if $msg contains 'Array[IP-Pool-List] then {
-/var/log/event_contains_ip.log
STOP
}
else{
-/var/log/event_doesnt_contain_ip.log
STOP
}
}
Would I also have to restart the rsyslogd service in order to refresh the array [IP-Pool-List], since the file is changing every few minutes ?
Is there a way to solve this with rsyslog, or is there an easier/better way to do this ?
EDIT
I used crontab and inotifywait to keep the IP-Pool-File up to date.
#reboot inotifywait -q -m -e modify /var/log/client-ips.log |
while read -r filename events; do sort -u /var/log/client-ips.log
-o /var/log/ips-unique.log ;done
I also asked this question via mail to the "rsyslog-community" their contact mail is listed at the bottom of the rsyslog documentation (https://lists.adiscon.net/mailman/listinfo/rsyslog)
They gave me the advise to use "Lookup-Tables" to solve this. Currently reading how to use them correctly. Since I am a slow learner, this should take a while.
EDIT 2
A new plan is starting to form:
To get Lookup-Table going, I need to create a JSON-File, which
contains the values I am going to build my filter with
Update the JSON File with "reload_lookup_table" or "JQ", whenever a new IP is added to the
IP-Pool-List.
Use RSYSLOG Lookup-Table to filter for the given IPs and split the
results into two seperate log files | Match.log | NoMatch.log
Related
I was hoping someone could help me. I am new to powershell and struggling with trying to find the right way to approach something in my script. The script is to do a restore check on our backups using the veeam powershell commands. We have 7 backup jobs with various servers in each job and various drives being backed up in each job. Sometimes the servers are in multiple jobs as we have to split the drives across jobs as we copy the backup jobs to USB every day (so we have to balance out the amount of data we backup in each job so that we can copy the drives to the USB drives). I can write the entire script and get it to work but its around 800 lines long and is very inefficient although easy to understand for my team (and me later on!). My current approach pretty much revolves around performing certain actions for each backup job then each server within that job on each disk. I would like to cut it down using an array/loop. I have figured out how to use basic loops and arrays but I am struggling with being able to link the drives and servers in an array for example:
$Backupjob1 = "Backup Job 1E"
$Backupjob1Servers =#('Server1','Server2')
Somekind of array that allows different numbers of multiple drives for Server1 and Server 2.
$Backupjob1ServeDrives =#(Server1.Drive1 ='C', Server1.Drive2 ='F', Server2.Drive1 = 'C')
and then I need to loop through it so that on each loop, it performs an action on server1 and its first drive then does the next drive. Once server1 has finished, it performs the action on server2 on its first drive and then repeats on the second drive.
I understand the basics of looping through an array but I am struggling to understand how i would create an array of arrays that can deal with the above and then loop through it correctly.
I am stuck as not sure what array works and what options there are.
One way would be to externalise your config into a json file or similar and feed that to your script. Here is an example using a json string:
$serverConfig = #"
[{
"serverName": "server1",
"backupDrives": [
"D","F"
]
},
{
"serverName": "server2",
"backupDrives": [
"D","G","H"
]
}]
"#
$serverList = $serverConfig | ConvertFrom-Json
foreach($server in $serverList) {
Write-Host "Server: $($server.serverName)"
foreach($drive in $server.backupDrives) {
Write-Host "Backing up drive $drive..."
# Logic for the backup
}
}
This demonstrates handling the array of arrays you reference, you have an array of server objects and each server object has an array of drives, so on each iteration of the servers you iterate the server's drives too.
If you save the string into a config file your script could read that in, and then if you make any changes to the backup configuration you only need to change the config file and not the script.
It sounds like you want a dictionary type - thankfully PowerShell has a builtin unordered dictionary type called a hashtable that can be used.
In order to construct a hashtable literal, use #{...} instead of #(...):
$driveMapping = #{
Server1 = #{
Drive1 = 'C'
Drive2 = 'F'
}
Server2 = #{
Drive1 = 'C'
}
}
To traverse this data structure, use a simple nested loop:
foreach($server in $drivemapping.psbase.Keys){
Write-Host "About to backup the drives on server '$server'"
foreach($drive in $drivemapping[$server].psbase.Keys){
$driveLetter = $drivemapping[$server][$drive]
Write-Host "About to backup drive '$drive' with letter '$driveLetter' on server '$server'"
}
}
I own a small minecraft server, and I would like to create a google spreadsheet for calculating user playtime data. I wan't this data because it would help let me know if my advertising campaigns are working or not. You can try to eyeball this stuff, but a solid data set would be alot more effective than guessing if the advertising is effective. The problem lies in the fact that manually searching for data from the server logs is really hard. I would appreciate anyone who could help me build a simple script or something that reads a .txt file and extracts the data I need. The script needs to be able to:
Detect lines with "User Authenticator" and "Disconnected" then print the entire line.
Format the text in some way? Possibly alphabetize the lines so that were not all over the place looking for specific users logins and logouts, defeating the purpose of the script. Not sure if this is possible.
Exclude lines with certain text (usernames), we want normal player data, not admin data.
I am sorry if did anything wrong, this is my first time on the site.
UPDATE: The admin data would be stored in a file called "admins.txt". By "alphabetizing" i meant it, example: Player A joins at 06:00, Player B joins at 06:30, then, Player A leaves at 06:45, Player B leaves at 07:00. If the data was flat, it would end up reading something like: A: 6:00, B: 6:30, A:6:45, B:7:00. But I would rather it be: A: 6:00, A: 6:45, B: 6:30, B: 7:00. That would make it easier to chart it out and make a calculation. Sorry for the long text.
Also typical server logging looks like this:
[15:46:30] [User Authenticator #1/INFO]: UUID of player DraconicPiggy is (UUID)
[15:46:31] [Server thread/INFO]: DraconicPiggy[/(Ip address)] logged in with entity id 157 at ([world]342.17291451961574, 88.0, -32.04791955684438)
The following awk script will report only on the two line types that you mentioned.
/User Authenticator|Disconnected/ {
print
}
I'm guessing "alpabetize" means sort. If so then you can pass the awk output to sort via a pipe.
awk -f script.awk | sort
I'm assuming the file is a log that's already in date-time sequence, with the timestamps at the start of the line. In this case you'll need to tell sort what to sort on.sort /? will tell you how to do this.
Multiple input files
To process all log files in the current directory use:
awk -f script.awk *.log
Redirect output to file
The simplest way is by adding > filtered.log to the command, like this:
awk -f script.awk *.log > filtered.log
That will filter all the input files into a single output file. If you need to write one filtered log for each input file then a minor script change is needed:
/User Authenticator|Disconnected/ {
print >> FILENAME ".filtered.log"
}
Filtering admins and redirecting to several files
The admins file should be similar to this:
Admin
DarkAdmin
PinkAdmin
The admin names must not contain spaces. i.e DarkAdmin is OK, but Dark Admin woud not work. Similarly the user names in your log files must not contain spaces for this script to work.
Execute the following script with this command:
awk -f script.awk admins.txt *.log
Probably best to make sure the log files and the filtered output are in separate directories.
NR == FNR {
admins[ $1 ] = NR
next
}
/User Authenticator|Disconnected/ {
if ( $8 in admins ) next
print >> FILENAME ".filtered.log"
}
The above script will:
Ignore all lines that mention an admin.
Creat a filtered version of every log file. i.e. if there are 5 log files then 5 filtered logs will be created.
Sorting the output
You have two sort keys in the file, the user and the time. This is beyone the capabilities of the standard Windows sort program, which seens very primitive. Yo should be able to do it with Gnu Sort:
sort --stable --key=8 test.log > sorted_test.log
Where:
--key=8 tells it to sort on field 8 (user)
--stable keeps the files in date order within each user
Example of sorting a log file and displaying the result:
terry#Envy:~$ sort --stable --key=8 test.log
[15:23:30] [User Authenticator #1/INFO]: UUID of player Doris is (UUID)
[16:36:30] [User Disconnected #1/INFO]: UUID of player Doris is (UUID)
[15:46:30] [User Authenticator #1/INFO]: UUID of player DraconicPiggy is (UUID)
[16:36:30] [User Disconnected #1/INFO]: UUID of player DraconicPiggy is (UUID)
[10:24:30] [User Authenticator #1/INFO]: UUID of player Joe is (UUID)
terry#Envy:~$
I am trying to integrated two systems using SNMP protocol, here is my question:
as you know to get queries from for example MySQL database i have ability to add conditions in query string and get required result, some thing like below:
SELECT * FROM table WHERE a='foo' AND b='bar'
is there any possibility to request queries using SNMP with conditions
It depends if you need to select data based on the mib instance or the mib data.
You can select data based on the instance, like:
snmpwalk -v2c -cpublic 1.2.3.4 ifOperStatus
This will give you all interface statuses in a device.
SNMP do not support getting only ifOperStatus = "up", in this case you need to retrieve all instance and make your selection
snmpwalk -v2c -cpublic 1.2.3.4 ifOperStatus | grep up
Another more over the top solution is to collect the data and store it in a database, then you can use the SQL syntax you mentioned in your question
Say you have an LDAP with the following structure:
dc=corp,dc=com
|--dc=security
|--ou=users
|--ou=corporate
| |--ou=it
| |--it-user1
| |--it-user2
|--user1
|--user2
|--user3
I need a search query that will look at all entries under the users ou, including those under corporate and it.
Currently I am trying the following:
uid=it-user2,ou=users,dc=security,dc=corp,dc=com
The scope of the search is set as subtree. I was under the impression that the subtree scope would cause the LDAP to search recursively through the entire tree, but that does not seem to be the case. However, if I add the full path into the search as I have below, the entry is found.
uid=it-user2,ou=it,ou=corporate,ou=users,dc=security,dc=corp,dc=com
Could someone give me an idea of where I am going wrong? Thanks.
You need to set your search context (i.e., the search base) to where your object/entry is stored. Based on your example, the search context is ou=users,dc=security,dc=corp,dc=com. When you set the search scope to subtree, it should find the entry or entries that match your critera (i.e., search filter). For example,
ldapsearch -h SERVER -b ou=users,dc=security,dc=corp,dc=com -s sub "(uid=it-user2)"
Of course, with the 'subtree' search scope, you could even set the search context to a higher level container (e.g., dc=security,dc=corp,dc=com). Your entry would still be found as long as it matches the criteria specified by your filter. Since you're searching for all entries under the ou=users container, your query would probably look like this:
ldapsearch -h SERVER -b ou=users,dc=security,dc=corp,dc=com -s sub "(uid=*)"
or
ldapsearch -h SERVER -b ou=users,dc=security,dc=corp,dc=com -s sub "(objectclass=*)"
I fought this for hours - CN=Users LDAP Directory Entry in .Net - not working with OU=Users
This may seem silly and stupid, but the default tree setup in Active Directory is not OU=Users,dc=domain,dc=com but rather CN=Users,dc=domain,dc=com (Note the CN= not the OU= for Users.)
uid=it-user2,ou=users,dc=security,dc=corp,dc=com does not exist. The LDAP client must provide a base object to the search request which exists.
see also
LDAP: Search Best Practices
How do I filter an Active Directory LDAP query to groups containing the authenticated/bound user (or any user at all)? This works fine:
(&(objectClass=group)(member=*))
>>> lots of results
But I can't go any more detail:
(&(objectClass=group)(member=*S*))
>>> nothing
The MSDN mentions using a filter like this:
(member:1.2.840.113556.1.4.1941:=(cn=user1,cn=users,DC=x))
But even ignoring the crazy hyper magic number involved in that, I always get 0 results when I try to filter with that (even replacing cn=user1,cn=users,DC=x with my own distinguishedName, even replacing it with *).
You need the full DN of the user i.e
(&(member=CN=Your Name,OU=Your OU,DC=company,DC=com)(objectClass=group))
take note you cannot use * in this one
So the crazy hyper magic number involved in recursive search is explained in Search Filter Syntax.
To find in one search (recursively) all the groups that "user1" is a member of:
Set the base to the groups container DN; for example root DN (dc=dom,dc=fr)
Set the scope to subtree
Use the following filter: (member:1.2.840.113556.1.4.1941:=cn=user1,cn=users,DC=x)
explicited using LDIFDE.EXE the command line tool included in Windows Server it gives:
ldifde -f user1Grps.ldf -d "dc=societe,dc=local" -r "(member:1.2.840.113556.1.4.1941:=cn=user1,ou=Monou,dc=societe,dc=local)"
If you are running that on a W2K8 or W2K8 R2 server be careful to run as administrator.
If you are programming in C# you can use:
/* Retreiving a principal context
*/
Console.WriteLine("Retreiving a principal context");
PrincipalContext domainContext = new PrincipalContext(ContextType.Domain, "WM2008R2ENT:389", "dc=dom,dc=fr", "jpb", "PWD");
/* Look for all the groups a user belongs to
*/
UserPrincipal aUser = UserPrincipal.FindByIdentity(domainContext, "user1");
PrincipalSearchResult<Principal> a = aUser.GetAuthorizationGroups();
foreach (GroupPrincipal gTmp in a)
{
Console.WriteLine(gTmp.Name);
}