A masterdata descriptor like this one: $masterdata:"pool_name" is converted in the Movilizer client to an integer number like 113.
We are building a logic that sends back to the backend the poolid (113) and the key modified (key="key1") in a DataContainer.
The DataContainer key is formed like this: "poolid$$key", my question is:
Is the poolid integer number the same accross participant/devices?? (Always 113), or is it random depending on the client?
I need to know this to send some string with the poolname instead of the poolid for this to work.
If I have to guess I'd say the poolID mapping is the same over all participants/devices and depends on the order of pools as they are created in a systemID. Probably the first pool that is created in the sysID gets the mapping 0 or 1 ... and so on. But this is just a wild guess and I am not sure if your approach is advisable at all. In the end it could all rely on an unsorted data structure and might change the ordering of the id's in unexpected ways based on that.
Related
I have a use case where a large no of logs will be consumed to the apache flink CEP. My use case is to find the brute force attack and port scanning attack. The challenge here is that while in ordinary CEP we compare the value against a constant like "event" = login. In this case the Criteria is different as in the case of brute force attack we have the criteria as follows.
username is constant and event="login failure" (Delimiter the event happens 5 times within 5 minutes).
It means the logs with the login failure event is received for the same username 5 times within 5 minutes
And for port Scanning we have the following criteira.
ip address is constant and dest port is variable (Delimiter is the event happens 10 times within 1 minute). It means the logs with constant ip address is received for the 10 different ports within 1 minute.
With Flink, when you want to process the events for something like one username or one ip address in isolation, the way to do this is to partition the stream by a key, using keyBy(). The training materials in the Flink docs have a section on Keyed Streams that explains this part of the DataStream API in more detail. keyBy() is the roughly same concept as a GROUP BY in SQL, if that helps.
With CEP, if you first key the stream, then the pattern will be matched separately for each distinct value of the key, which is what you want.
However, rather than CEP, I would instead recommend Flink SQL, perhaps in combination with MATCH_RECOGNIZE, for this use case. MATCH_RECOGNIZE is a higher-level API, built on top of CEP, and it's easier to work with. In combination with SQL, the result is quite powerful.
You'll find some Flink SQL training materials and examples (including examples that use MATCH_RECOGNIZE) in Ververica's github account.
Update
To be clear, I wouldn't use MATCH_RECOGNIZE for these specific rules; neither it nor CEP is needed for this use case. I mentioned it in case you have other rules where it would be helpful. (My reason for not recommending CEP in this case is that implementing the distinct constraint might be messy.)
For example, for the port scanning case you can do something like this:
SELECT e1.ip, COUNT(DISTINCT e2.port)
FROM events e1, events e2
WHERE e1.ip = e2.ip AND timestampDiff(MINUTE, e1.ts, e2.ts) < 1
GROUP BY e1.ip HAVING COUNT(DISTINCT e2.port) >= 10;
The login case is similar, but easier.
Note that when working with streaming SQL, you should give some thought to state retention.
Further update
This query is likely to return a given IP address many times, but it's not desirable to generate multiple alerts.
This could be handled by inserting matching IP addresses into an Alert table, and only generate alerts for IPs that aren't already there.
Or the output of the SQL query could be processed by a de-duplicator implemented using the DataStream API, similar to the example in the Flink docs. If you only want to suppress duplicate alerts for some period of time, use a KeyedProcessFunction instead of a RichFlatMapFunction, and use a Timer to clear the state when it's time to re-enable alerts for a given IP.
Yet another update (concerning CEP and distinctness)
Implementing this with CEP should be possible. You'll want to key the stream by the IP address, and have a pattern that has to match within one minute.
The pattern can be roughly like this:
Pattern<Event, ?> pattern = Pattern
.<Event>begin("distinctPorts")
.where(iterative condition 1)
.oneOrMore()
.followedBy("end")
.where(iterative condition 2)
.within(1 minute)
The first iterative condition returns true if the event being added to the pattern has a distinct port from all of the previously matching events. Somewhat similar to the example here, in the docs.
The second iterative condition returns true if size("distinctPorts") >= 9 and this event also has yet another distinct port.
See this Flink Forward talk (youtube video) for a somewhat similar example at the end of the talk.
If you try this and get stuck, please ask a new question, showing us what you've tried and where you're stuck.
I could need some help with a Anylogic Model.
Model (short): Manufacturing scenario with orders move in a individual route. The workplaces (WP) are dynamical created by simulation start. Their names, quantity and other parameters are stored in a database (excel Import). Also the orders are created according to an import. The Agent population "order" has a collection routing which contains the Workplaces it has to stop in the specific order.
Target: I want a moveTo block in main which finds the next destination of the agent order.
Problem and solution paths:
I set the destination Type to agent and in the Agent field I typed a function agent.getDestination(). This function is in order which returns the next entry of the collection WP destinationName = routing.get(i). With this I get a Datatype error (while run not compiling). I quess it's because the database does not save the entrys as WP Type but only String.
Is there a possiblity to create a collection with agents from an Excel?
After this I tried to use the same getDestination as String an so find via findFirst the WP matching the returned name and return it as WP. WP targetWP = findFirst(wps, w->w.name == destinationName);
Of corse wps (the population of Workplaces) couldn't be found.
How can I search the population?
Maybe with an Agentlink?
I think it is not that difficult but can't find an answer or a solution. As you can tell I'm a beginner... Hope the description is good an someone can help me or give me a hint :)
Thanks
Is there a possiblity to create a collection with agents from an Excel?
Not directly using the collection's properties and, as you've seen, you can't have database (DB) column types which are agent types.1
But this is relatively simple to do directly via Java code (and you can use the Insert Database Query wizard to construct the skeleton code for you).
After this I tried to use the same getDestination as String an so find via findFirst the WP matching the returned name and return it as WP
Yes, this is one approach. If your order details are in Excel/the database, they are presumably referring to workplaces via some String ID (which will be a parameter of the workplace agents you've created from a separate Excel worksheet/database table). You need to use the Java equals method to compare strings though, not == (which is for comparing numbers or whether two objects are the same object).
I want a moveTo block in main which finds the next destination of the agent order
So the general overall solution is
Create a population of Workplace agents (let's say called workplaces in Main) from the DB, each with a String parameter id or similar mapped from a DB column.
Create a population of Order agents (let's say called orders in Main) from the DB and then, in their on-startup action, set up their collection of workplace IDs (type ArrayList, element class String; let's say called workplaceIDsList) using data from another DB table.
Order probably also needs a working variable storing the next index in the list that it needs to go to (so let's say an int variable nextWorkplaceIndex which starts at 0).
Write a function in Main called getWorkplaceByID that has a single String argument id and returns a Workplace. This gets the workplace from the population that matches the ID; a one-line way similar to yours is findFirst(workplaces, w -> w.id.equals(id)).
The MoveTo block (which I presume is in Main) needs to move the Order to an agent defined by getWorkplaceByID(agent.workplaceIDsList.get(nextWorkplaceIndex++)). (The ++ bit increments the index after evaluating the expression so it is ready for the next workplace to go to.)
For populating the collection, you'd have two tables, something like the below (assuming using strings as IDs for workplaces and orders):
orders table: columns for parameters of your orders (including some String id column) other than the workplace-list. (Create one Order agent per row.)
order_workplaces table: columns order_id, sequence_num and workplace_id (so with multiple rows specifying the sequence of workplace IDs for an order ID).
In the On startup action of Order, set up the skeleton query code via the Insert Database Query wizard as below (where we want to loop through all rows for this order's ID and do something --- we'll change the skeleton code to add entries to the collection instead of just printing stuff via traceln like the skeleton code does).
Then we edit the skeleton code to look like the below. (Note we add an orderBy clause to the initial query so we ensure we get the rows in ascending sequence number order.)
List<Tuple> rows = selectFrom(order_workplaces)
.where(order_workplaces.order_id.eq(id))
.orderBy(order_workplaces.sequence_num.asc())
.list();
for (Tuple row : rows) {
workplaceIDsList.add(row.get(order_workplaces.workplace_id));
}
1 The AnyLogic database is a normal relational database --- HSQLDB in fact --- and databases only understand their own specific data types like VARCHAR, with AnyLogic and the libraries it uses translating these to Java types like String. In the user interface, AnyLogic makes it look like you set the column types as int, String, etc. but these are really the Java types that the columns' contents will ultimately be translated into.
AnyLogic does support columns which have option list types (and the special Code type column for columns containing executable Java code) but these are special cases using special logic under the covers to translate the column data (which is ultimately still a string of characters) into the appropriate option list instance or (for Code columns) into compiled-on-the-fly-and-then-executed Java).
Welcome to Stack Overflow :) To create a Population via Excel Import you have to create a method and call Code like this. You also need an empty Population.
int n = excelFile.getLastRowNum(YOUR_SHEET_NAME);
for(int i = FIRST_ROW; i <= n; i++){
String name = excelFile.getCellStringValue(YOUR_SHEET_NAME, i, 1);
double SEC_PARAMETER_TO_READ= excelFile.getCellNumericValue(YOUR_SHEET_NAME, i, 2);
WP workplace = add_wps(name, SEC_PARAMETER_TO_READ);
}
Now if you want to get a workplace by name, you have to create a method similar to your try.
Functionbody:
WP workplaceToFind = wps.findFirst(w -> w.name.equals(destinationName));
if(workplaceToFind != null){
//do what ever you want
}
I have a guid which Sysinternals tools named Sysmon left.
It looks like this.
3/18 C591B94E-4BDD-5AAE-0000-001073B13706
4/4 C591B94E-1BFA-5AC5-0000-0010E76F3903
4/29 C591B94E-A33F-5AE5-0000-001074CA4C26
5/2(different windows account) C591B94E-E23B-5AE9-0000-0010DD40EF32
5/2(on the virtual machine) A15730FB-E3DA-5AE9-0000-0010AB2C0800
It's generated when the process is created(Event id 1) in my computer on different days and different environment.
And I Found the uuid format (https://en.wikipedia.org/wiki/Universally_unique_identifier)
xxxxxxxx-xxxx-Mxxx-Nxxx-xxxxxxxxxxxx(M indicate the UUID version, and the one to three most significant bits of digit N indicate the UUID variant)
According to this, my 3/18 example is C591B94E-4BDD-5AAE-0000-001073B13706. It means M is 5, N is 0, In other words, UUID version is 5, variant is 0. It means It's SHA-1 Hash Value(Version 5) and Variant is 0.
I really wonder what the other number does mean. Because the sysmon's documents says that guid is helpful for correlation BUT they never explain what does this number mean.
I can guess the first group is related to PC information. because only when I chanaged the PC(5/2 on the virtual machine) the first group is changed(C591B94E -> A15730FB). So I thought It's related to Mac or IP address. But even if I changed the MAC and IP address, It stayed A15730FB or C591B94E.
I'm sure the second group is related to time.
But I can't figure out what does this exactly mean.
The GUID does not specifically mean anything in itself. Its purpose is to allow you to correlate and filter process events when Windows reuses process IDs (in this way you can think of it as a completely unique process ID).
From: https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon
"Includes a process GUID in process create events to allow for correlation of events even when Windows reuses process IDs."
I see the document
https://developers.google.com/appengine/docs/python/ndb/keyclass#Key_integer_id
Returns the integer id in the last (kind, id) pair, or None if the key
has an string id or is incomplete.
see I think the id of a key can be a int ; so I write
r = ndb.Key(UserSession, int(id)).get()
if r:
return r.session
but the dev_server.py , will always raise
File "/home/bitcoin/down/google_appengine/google/appengine/datastore/datastore_stub_util.py", line 346, in CheckReference
raise datastore_errors.BadRequestError('missing key id/name')
BadRequestError: missing key id/name
I chanage the int(id) -> str(id)
seems right ;
so my question is , How to use ndb key with integer_id ?
the model is
class UserSession(ndb.Model):
session = ndb.BlobProperty()
The type of the id you use when reading the entity must match the type of the id you used when you wrote the entity. Normally, integer ids are assigned automatically when you write a new entity without specifying an id or key; you then get the id out of the key returned by entity.put(). It is generally not recommended to assign your own integer ids; when the app assigns the keys, the convention is that they should be strings.
There's an easier way to fetch:
UserSession.get_by_id(int(id))
https://developers.google.com/appengine/docs/python/ndb/modelclass#Model_get_by_id
If that doesn't work, I suspect that id is wrong or empty.
There must be something wrong with your variable 'id'.
Your code here should be no problem, and it's better to user long instead of int.
You can try your code on interactive console of development server with specific integer id.
It may be easier to identify your entities in the sessions with their keys instead of their ids. There really is no need to extract the ID from the key to identify the session (other than maybe saving a bit of memory. I think the way your thinking is based on a RDB. I learned that using the key actually makes entity/session identifications easier.
'id' is also a python builtin function. Maybe you are taking that by mistake.
I would like to get some information from the user profiles of a certain domain.
www.domain.com/profiles/[userID]
The main problem is that the user ID is like:
4fc34f1ad3d85a000300b5a4
4fc34f1ad3d85a000300b5a4
4f4bdd96848740000300026a
4f09884f9cc1590001019c98
4f1bcd5f33aa850001011811
4faa2801c619ad0003011c7e
4e4f506ac9d69600010010ca
so if I wnat the user profile of a certain user I have to put
www.domain.com/profiles/4fc34f1ad3d85a000300b5a4
www.domain.com/profiles/4f4bdd96848740000300026a
www.domain.com/profiles/4f09884f9cc1590001019c98
www.domain.com/profiles/4f1bcd5f33aa850001011811
www.domain.com/profiles/4faa2801c619ad0003011c7e
www.domain.com/profiles/4e4f506ac9d69600010010ca
What kind of ID is that? How I know the subjacent canonical order?
Other times was a lot easier because the user ID only was integers.
www.domain.com/profiles/1
www.domain.com/profiles/2
www.domain.com/profiles/3
www.domain.com/profiles/4
www.domain.com/profiles/5
I would like to know if this is some kind of encoding or what.
What kind of ID on the URL is that?
Thank you in advance.
Looks like an MD5 hash. So either it's a random hash, or it might a hash of the previously used ID, or something completely different. Either way apart from brute force guessing, there's no simple way to decode them.
http://en.wikipedia.org/wiki/MD5
These ID's use hexadecimal - notice there are no letters above 'f'.
The similarities between ID's are interesting - could you provide the ID's of some consecutively created pages, to check if there is a pattern to how the ID changes.
Or the website may just generate a random number between certain limits and convert to hex.