I see EAN'S signature's document like the following:
http://developer.ean.com/docs/getting-started/api-authentication
SIG = MD5("{APIKey}{APISecret}{UnixTimestamp in second}")
When using the digital signature, the value of sig is calculated by generating an MD5 hash made up of the API key, the API user’s shared secret, and a UNIX timestamp.
The system accepts timestamps up to five minutes before or after the server timestamp to accommodate for reasonable clock drift.
And they show the sample url only contains ApiKey, not include timestamp
http://api.ean.com/ean-services/rs/hotel/v3/avail?cid=[yourCID]&apiKey=[yourAPIKey]&sig=[youSigValue]&minorRev=[current minorRev #]&customerUserAgent=[xxx]&customerIpAddress=[xxx]&locale=en_US¤cyCode=USD&hotelId=201252
My question is :
How they detected my request whether in five minutes before or after the server time???
Compare MD5 result 600 times??
Related
I have a use case where a large no of logs will be consumed to the apache flink CEP. My use case is to find the brute force attack and port scanning attack. The challenge here is that while in ordinary CEP we compare the value against a constant like "event" = login. In this case the Criteria is different as in the case of brute force attack we have the criteria as follows.
username is constant and event="login failure" (Delimiter the event happens 5 times within 5 minutes).
It means the logs with the login failure event is received for the same username 5 times within 5 minutes
And for port Scanning we have the following criteira.
ip address is constant and dest port is variable (Delimiter is the event happens 10 times within 1 minute). It means the logs with constant ip address is received for the 10 different ports within 1 minute.
With Flink, when you want to process the events for something like one username or one ip address in isolation, the way to do this is to partition the stream by a key, using keyBy(). The training materials in the Flink docs have a section on Keyed Streams that explains this part of the DataStream API in more detail. keyBy() is the roughly same concept as a GROUP BY in SQL, if that helps.
With CEP, if you first key the stream, then the pattern will be matched separately for each distinct value of the key, which is what you want.
However, rather than CEP, I would instead recommend Flink SQL, perhaps in combination with MATCH_RECOGNIZE, for this use case. MATCH_RECOGNIZE is a higher-level API, built on top of CEP, and it's easier to work with. In combination with SQL, the result is quite powerful.
You'll find some Flink SQL training materials and examples (including examples that use MATCH_RECOGNIZE) in Ververica's github account.
Update
To be clear, I wouldn't use MATCH_RECOGNIZE for these specific rules; neither it nor CEP is needed for this use case. I mentioned it in case you have other rules where it would be helpful. (My reason for not recommending CEP in this case is that implementing the distinct constraint might be messy.)
For example, for the port scanning case you can do something like this:
SELECT e1.ip, COUNT(DISTINCT e2.port)
FROM events e1, events e2
WHERE e1.ip = e2.ip AND timestampDiff(MINUTE, e1.ts, e2.ts) < 1
GROUP BY e1.ip HAVING COUNT(DISTINCT e2.port) >= 10;
The login case is similar, but easier.
Note that when working with streaming SQL, you should give some thought to state retention.
Further update
This query is likely to return a given IP address many times, but it's not desirable to generate multiple alerts.
This could be handled by inserting matching IP addresses into an Alert table, and only generate alerts for IPs that aren't already there.
Or the output of the SQL query could be processed by a de-duplicator implemented using the DataStream API, similar to the example in the Flink docs. If you only want to suppress duplicate alerts for some period of time, use a KeyedProcessFunction instead of a RichFlatMapFunction, and use a Timer to clear the state when it's time to re-enable alerts for a given IP.
Yet another update (concerning CEP and distinctness)
Implementing this with CEP should be possible. You'll want to key the stream by the IP address, and have a pattern that has to match within one minute.
The pattern can be roughly like this:
Pattern<Event, ?> pattern = Pattern
.<Event>begin("distinctPorts")
.where(iterative condition 1)
.oneOrMore()
.followedBy("end")
.where(iterative condition 2)
.within(1 minute)
The first iterative condition returns true if the event being added to the pattern has a distinct port from all of the previously matching events. Somewhat similar to the example here, in the docs.
The second iterative condition returns true if size("distinctPorts") >= 9 and this event also has yet another distinct port.
See this Flink Forward talk (youtube video) for a somewhat similar example at the end of the talk.
If you try this and get stuck, please ask a new question, showing us what you've tried and where you're stuck.
In the link agent, I came across attributes like maxPropagationDelay and reservationGuardTime. What is the role of these attributes? Where I can find more information about these attributes.
These are parameters of specific LINK protocols.
maxPropagationDelay is used to determine timeouts based on expected round-trip-times in the network. It should be set to a value that depends on the geographical size of your network if the network is small enough for a single hop connection between any pair of nodes. Otherwise it should be set to a value based on the maximum communication range of your modem.
reservationGuardTime is a small extra time that a channel is reserved for, to allow for practical timing jitter of modems. Usually the default value for this provided by the agent will be good enough for most purposes.
The Underwater networks handbook to be released with the upcoming version of UnetStack3 will provide a lot more guidance on many of these parameters, and on how to set up various types of networks using Unetstack.
You can access more information about any parameters of any of the Agents in UnetStack using the help command. For the Link Agent, you'll see this in UnetStack 1.4.
> help link
link - access to link agent
Examples:
link // access parameters
link.maxRetries = 5 // set maximum retries for reliable delivery
link << new DatagramReq(to: 2, data: [1,2,3], reliability: true)
// send reliable datagram
Parameters:
MTU - maximum data transfer size
maxRetries - maximum retries for reliable delivery
reservationGuardTime - guard period (s)
maxPropagationDelay - maximum propagation delay (s)
dataChannel - channel to use for data frames (0 = control, 1 = data)
reservationGuardTime is the additional guard time that can be added to the frame duration when reserving a channel (using MAC) to ensure channel reservations have some delay in between for the nodes to be able react.
maxPropagationDelay is used to estimate the maximum time that an acknowledge to a request (or a series of requests if fragmentation is needed) might take and used to set timeouts for transmissions, or to make channel reservations (if using a MAC). Depending on your simulation/setup, you can change this number to be longest time (one-way) between two nodes which can communicate.
I am using librdkafka C API consumer (specifically using rd_kafka_consumer_poll to read and I did call rd_kafka_poll_set_consumer before this)
Problem I see is that in my google test I do following
write 3 messages to kafka
init/start kafka consumer (rd_kafka_consumer_poll)
in rebalance_cb I set each partition offset to RD_KAFKA_OFFSET_STORED and assign them to handle
At this point I believe it should read 3 messages but it reads only last message but surprisingly offset for each partition is already updated!
Am I missing something here using Kafka consumer?
And one more question is I initially thought stored offset is in kafka broker and there is unique offset for topic + consumer group id + partition combination.
So I thought different consumer groups reading same topic should have different offset.
However, it doesn't look like the case. I am always reading from same offset when used different consumer groups.
I am suspecting this may be related to offset commit but not sure where to tackle this.
Any insight?
Configuration to look at : auto.offset.reset
From Kakfa consumer documentation :
What to do when there is no initial offset in Kafka or if the current
offset does not exist any more on the server
From librdkafka documentation :
Action to take when there is no initial offset in offset store or the
desired offset is out of range: 'smallest','earliest' - automatically
reset the offset to the smallest offset, 'largest','latest' -
automatically reset the offset to the largest offset, 'error' -
trigger an error which is retrieved by consuming messages and checking
'message->err'. Type: enum value
Default value is latest.
Furthermore,
#define RD_KAFKA_OFFSET_STORED -1000
So, you're trying to set partition offset to -1000 which is obviously not a valid offset.
Apparently, librdkafka reads last message in this case (I didn't check code).
References:
id scheme
Format: id:<namespace>:<document-type>:<key/value-pairs>:<user-specified>
http://docs.vespa.ai/documentation/content/buckets.html
http://docs.vespa.ai/documentation/content/idealstate.html
its possible to structure data in user defined bucketing logic by using 32 LSB in document-id format (n / g selections).
however, the query logic isn't very clear on how to route queries to a specific bucket range based on a decision taken in advance.
e.g., it is possible to split data into a time range (start-time/end-time) if i can define n (a number) compressing the range. all documents tagged such will end up in same bucket (that will follow its course of split on number of documents / size as configured).
however, how do i write a search query on data indexed in such manner?
is it possible to indicate the processor to choose a specific bucket, or range of buckets (in case distribution algorithm might have moved buckets)?
You can choose one bucket in a query by specifying the streaming.groupname query property.
Either in the http request by adding
&streaming.groupname=[group]
or in a Searcher by
query.properties().set("streaming.groupname","[group]").
If you want multiple buckets, use the parameter streaming.selection instead, which accepts any document selection expression: http://docs.vespa.ai/documentation/reference/document-select-language.html
To specify e.g two buckets, use set streaming.selection (in the HTTP request or a Searcher) to
id.group=="[group1]" and id.group=="[group2]"
See http://docs.vespa.ai/documentation/streaming-search.html
Note that streaming search should only be used when each query only need to search one or a few buckets. It avoids building reverse indexes, which is cheaper in that special case (only).
The &streaming.* parameters is described here http://docs.vespa.ai/documentation/reference/search-api-reference.html#streaming.groupname
This only applies to document types which are configured with mode=streaming, for default mode which is index you cannot control the query routing http://docs.vespa.ai/documentation/reference/services-content.html#document
Performance Engineering tool: LoadRunner 11
Protocol: Silverlight
The scenario:
1. A call is made (by the Silverlight application) to the web server to generate a unique identifier (UID)
2. The server returns a response with the UID
3. The application uses that UID to save Patient record (in this scenario, UID is Unique Patient Id)
The problem:
We would like to use the ID received in the Step 2's Web Response (say, as a local variable) and replace it in the next subsequent request sent by LoadRunner.
Please advise whether the same is possible.
*In parallel trying to figure out if "web_reg_save_param" will solve our problem.*
------------ LoadRunner data --------------------
1. Call to generate Uid
ignored as it is too large to paste here and irrelevant
2. Response from server (The UID is UNI-0000001544)
HTTP/1.1 200 OKCache-Control: privateContent-Type: application/msbin1Content-Encoding: gzipVary: Accept-EncodingServer:
Microsoft-IIS/7.5X-AspNet-Version: 4.0.30319X-Powered-By: ASP.NETDate:
Fri, 06 Jul 2012 05:41:27 GMTContent-Length:
188#GenerateSequenceResponsehttp://tempuri.org/#GenerateSequenceResult™UNI-0000001544
3. Next LoadRunner request where the UID (UNI-0000001543) has to be replaced with the UID received in the response
web_custom_request("SubmitChanges",
"URL=http://infinityappload/ClientBin/Infinity-Web-Services-ActorDomainService.svc/binary/SubmitChanges",
. . .
, "BodyBinary=#\rSubmitChanges\
. . .
\tPatientID\\x99\\x0EUNI-0000001543\
... LAST);
This is, as James said, the idea of correlation. The simplified approach would go as follows:
web_reg_save_param("UID2", "LB=GenerateSequenceResult™", "Savelen=14", LAST);
//The TM symbol will have to be replaced by whatever HTML code is used.
web_custom_request("WebRequest1", ....);
web_custom_request("SubmitChanges", "URL=http://infinityappload/...",
...
"BodyBinary=#\rSubmitChanges\",
"\tPatientID\\x99\\x0E{UID2}\",
LAST);
Values that are to be correlated are going to be determined prior to the request they will be populated by. They can then be used in any subsequent requests. Just make sure that, when using a correlated parameter, it is inside a quoted string.
The concept you are referring to is the management of correlated variables, it is a core concept in the use of performance test tools, LoadRunner included. This particular concept is the subject of nearly 1/3 of the standard LoadRunner script development product training. I would refer you to information on correlation for both web and silverlight protocols in the system documentation. Also, it is critical for your short and long term success that you pair yourself with both a strong mentor as well as attending some form of standard product training.