Changing CloudWatch log output from a "Kinesis Data Analytics for Apache Flink" app - apache-flink

Does anyone know how to change the CloudWatch log output from a "Kinesis Data Analytics for Apache Flink" app.?
There are two things I'd like to change:
The fields in the JSON written to CloudWatch
The contents/format of the "message" field (i.e., format of each "LOG.info", "LOG.warn", etc. - line)
#1 is most important.
The default format written to CloudWatch looks like this:
{
"locationInformation": "",
"logger": "",
"message": "",
"threadName": "",
"applicationARN": "arn:aws:kinesisanalytics:eu-west-1:...",
"applicationVersionId": "23",
"messageSchemaVersion": "1",
"messageType": "INFO"
}
Is it somehow possible to change the output, so that each CloudWatch entry becomes this instead:
{
“EventTime”: "20201224T23:59:59.999Z",
“LogLevel”: 5,
“EventSource”: "ApplicationURI/Name",
“Message”: ”foobar”
}
Using SLF4J is mentioned here (https://docs.aws.amazon.com/kinesisanalytics/latest/java/cloudwatch-logs-writing.html), although the format mentioned on the same page is the default described above.
The pom.xml file of the Java project includes aws-java-sdk-logs. It also excludes log4j and slf4j.
<artifactSet>
<excludes>
<exclude>org.slf4j:*</exclude>
<exclude>log4j:*</exclude>
</excludes>
</artifactSet>
I've had a look at this:
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html
But when testing locally, changing log4j.properties changes the "message" field of the log entry. That does not seem to be loaded when running on AWS, despite existing in the root directory of the .jar file. Even if I could make aws-java-sdk-logs pick up changes in log4j.properties, changing this file doesn't seem capable of changing the JSON fields written to CloudWatch (only the "message" format).
When the app starts on AWS, I can see it prints this:
-Dlog4j.configuration=file:/etc/flink/log4j-console.properties
-Dlogback.configurationFile=file:/etc/flink/logback-console.xml
I was hoping to perhaps copy them out on startup, changing them, and including them while setting the props to point at the JAR. However, both these files seem empty when trying to read them on startup from inside the Flink app code.
Is there some relatively straight-forward way to:
Rename/remove/add fields to the JSON written to CloudWatch?
Change the format of the "message" field?

Related

How to deploy a smart contract on KADENA

Please how do I deploy a smart contract to the testnet or mainnet WITHOUT Chainweaver web UI? I know I need a YAML file for that, but what do I do with it and where exactly do I send it?
Do I need to run a pact server, chainweb api or...? I couldn't find any guide for that
Step 0: Install the Prerequisites
Install Pact
Step 1: Create the Pact Module
We will be deploying the following pact module. For simplicity's sake the pact code we are deploying is not using a transaction's data field (read-keyset is one such pact function that makes use of this field). Otherwise, the accompanying YAML file will have to change. We also assume that this pact code is saved as test.pact.
(namespace 'free)
(module someModuleName AUTONOMOUS
(defcap AUTONOMOUS ()
true)
(defun dummy ()
(+ 1 2)
)
)
Step 2: Create the YAML file
The following YAML file will be used along with pact -a to sign and produce the escaped JSON needed to submit a transaction to Testnet.
codeFile: /Users/linda.ortega.cordoves/pact/test.pact
networkId: testnet04
publicMeta:
chainId: "0"
gasLimit: 1000
ttl: 28000
creationTime: 1585056536
sender: "testing"
gasPrice: 0.00001
keyPairs:
- public: 1d877a7b4524b6724a6ae708cf9ea7396d6ee9d17b10098b7793800177669c1d
secret: 33fcd94b8a42057bd4e3190f8983e3a73ec96c3f60df95c9e2aa3f13602c714f
nonce: step02
This file makes a couple of assumptions that might change depending on your specific implementation:
The full path of the pact we want to upload is: /Users/linda.ortega.cordoves/pact/test.pact
We want to submit a transaction to Testnet, whose network id is testnet04
We want to submit to the zero'th chain on Testnet, which has a chain id of "0"
That the current creation time in UNIX Epoch time is 1585056536 seconds. This value MUST CHANGE, so calculate it by either navigating to this website or running date +%s on the command line.
That "testing" is the account paying for gas (aka the "sender") on the Testnet network. To create a Testnet account and fund it some coins, navigate to the Testnet Coin Faucet. You will need to have generated an ED22519 public-private key pair to use the faucet. You can use pact -g to generate this key pair. Make sure to save it somewhere save.
That the key pair specified in "keyPairs" corresponds to the key pair used to create the gas payer account, which in this example is "testing". This must change from the defaults provided.
That we saved this YAML file as /Users/linda.ortega.cordoves/pact/test.yaml.
Step 3: Submit Transaction to Testnet
We will now submit the example pact module we created by hitting the /send endpoint of a Testnet node. In the command line, run the following command:
pact -a /Users/linda.ortega.cordoves/pact/test.yaml | curl -H "Content-Type: application/json" -d #- https://us1.testnet.chainweb.com/chainweb/0.0/testnet04/chain/0/pact/api/v1/send
Some of the assumptions we made when creating the YAML file become important here:
The network id must match the node endpoint we submit to. Since the network id we chose is testnet04, we must submit to /chainweb/0.0/testnet04/. And the node we submit to (in this case us1.testnet.chainweb.com) must have this network id.
The chain id must also match. We chose chain id of "0", so we must submit to /chain/0/.
That we saved the yaml file to /Users/linda.ortega.cordoves/pact/test.yaml.
If we submitted the transaction successfully we will see the following:
{"requestKeys":["Vetli41gi_S4-dZlro0visI8QT15brHoPe4vxMmhdek"]}
This means that our transaction was successfully added to the blockchain's mempool and is waiting to be mined. Make note of the request key returned from /send as we will use it when polling for the result of the transaction.
It is also possible that our transaction will fail node validation when we attempt to submit it. If this happens, you will receive a validation failure message instead of the request key.
Step 4: Verify the Result of the Transaction
We will now try to get the results of the transaction we submitted to the Testnet network by hitting the /poll endpoint. In the command line, run the following command:
curl -H "Content-Type: application/json" -d '{"requestKeys":["Vetli41gi_S4-dZlro0visI8QT15brHoPe4vxMmhdek"]}' -X POST https://us1.testnet.chainweb.com/chainweb/0.0/testnet04/chain/0/pact/api/v1/poll
Again, we make a couple of assumptions in this step:
That the Testnet node we want to poll from is us1.testnet.chainweb.com.
That the network id is testnet04. Note that part of the endpoint is /chainweb/0.0/testnet04/.
That the chain id we are polling from is chain "0". Note that part of the endpoint is /chain/0/.
That the request key we are polling for is Vetli41gi_S4-dZlro0visI8QT15brHoPe4vxMmhdek.
If the transaction was successfully mined and thus added to the blockchain, then /poll will return the following JSON object:
{
"Vetli41gi_S4-dZlro0visI8QT15brHoPe4vxMmhdek": {
"gas": 58,
"result": {
"status": "success",
"data": "Loaded module free.linda-test, hash n0g99JhWnO2F7X7f8o_zcAiSHBAWS_QSAfn4yUaqpps"
},
"reqKey": "Vetli41gi_S4-dZlro0visI8QT15brHoPe4vxMmhdek",
"logs": "0KzZQDJmEgnAKvPnO20UeGoE7KGCIN22nhjraeyp1aw",
"metaData": {
"blockTime": 1585056990071469,
"prevBlockHash": "dIYmpjBQge9yw0Yzhn0Sau-wJFwsLOFBmGbV3_0xYeE",
"blockHash": "yULpC5C-7tzRcc9sWm-f1bOC3JDvtxwT61hruW0aXrA",
"blockHeight": 261712
},
"continuation": null,
"txId": 266084
}
}
Please note that it is possible that a transaction fails at the pact level, but still gets added to the blockchain and gas gets charged. If this happens the result.status field will be failure.
If a transaction has not be mined yet, /poll will return {}. Keep retrying until you receive the JSON object shown above.
source: https://gist.github.com/LindaOrtega/1c219f887d9782c6745dbd827bdbfb4d

How to import Linux logs into ELK stack for log analysis?

I'm building a log analysis environment with the purpose of analyzing linux logs such as: /var/log/auth.log, /var/log/cron, /var/log/syslog, etc. The goal is to be able to upload such a log file and analyze it properly with Kibana/Elasticsearch. To do so, I created a .conf file as seen below, which includes the proper patterns to pars auth.log and the information needed in the input and output section. Unfortunately, when connecting to Kibana I cannot see any data in the "Discover" panel and cannot find the related "index pattern". I tested the grokk pattern and they works well.
input {
file {
type => "linux-auth"
path => [ "/home/ubuntu/logs/auth.log"]
}
filter {
if [type] == "linux-auth" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time} %{WORD:method}\[%{POSINT:auth_pid}\]\: %{DATA:message} for %{DATA:user} from %{IPORHOST:IP_address} port %{POSINT:port}" }
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:time} %{WORD:method}\[%{POSINT:auth_pid}\]\:%{DATA:message} for %{GREEDYDATA:username}" }
}
}
}
output{
elasticsearch {
hosts => "elasticsearch:9200"
}
}
Example of auth.log:
2018-12-02T14:01:00Z sshd[0000001]: Accepted keyboard-interactive/pam for root from 185.118.167.241 port 64965 ssh2
2018-12-02T14:02:00Z sshd[0000002]: Failed keyboard-interactive/pam for invalid user ubuntu from 36.104.140.175 port 57512 ssh2
2018-12-02T14:03:00Z sshd[0000003]: pam_unix(sshd:session): session closed for user root
Here is the few recommendations which i would like to give:
You can run logstash on debug mode like below to check what is the exact error.
bin/logstash --debug -f file_path.conf
Check with stdout in output section which will print the incoming data. So that you will be sure that logstash reading the file correctly.
The most important as you mention you want to read system log and need to visualize the data, I would recommend to use filebeat with system modules. Filebeat is especially build for such use cases like reading from file.
It is simple setup where in filebeat under the system module you just need to specify which system log file you need read. Mention the Elasticsearch endpoint and run the filebeat.
It will start reading and pushing the data to the elasticsearch.
Also You don't need build the custom dashboard in kibana (As you going to build in case of logstash). Filebeat comes with pre configured dashboards for system logs.
You can check more on above official document.

Unable to push vespa metrics to cloudwatch

Basically I need to monitor vespa metrics and for that I am trying to implement method to push metrics to cloudwatch.
This is the document that I am referring to https://docs.vespa.ai/documentation/monitoring.html
I have added the credentials file and putMetricData permission in the IAM role attached. The service.xml file that I am using in my code looks like this:
<admin version="2.0">
<adminserver hostalias="admin0"/>
<configservers>
<configserver hostalias="admin0"/>
</configservers>
<monitoring>
</monitoring>
<metrics>
<consumer id="my-cloudwatch">
<metric-set id="vespa" />
<cloudwatch region="ap-south-1" namespace="vespa">
<shared-credentials file="~/.aws/credentials" profile="default" />
</cloudwatch>
</consumer>
</metrics>
</admin>
I have deployed the code using vespa-deploy prepare application.zip && vespa-deploy activatebut I am still not seeing any metrics updated on my cloudwatch.
Also, I have tried to add:
<monitoring>
<interval>1</interval>
<systemname>vespa</systemname>
</monitoring>
But getting this error when deploying:
Request failed. HTTP status code: 400
Invalid application package: default.default: Error loading model: XML error in services.xml: element "interval" not allowed here; expected the element end-tag [9:16], input:
How can I fix this issue. Or atleast debug the issue that I am facing.
I suggest to use absolute path to the credentials file, as the ~ may not resolve to the directory you intended at runtime.
A couple more things:
I recommend using the default metric set, as vespa contains a lot of metrics, which will drive your CloudWatch cost higher. If you need additional metrics, you can add them with the metric tag inside consumer.
The monitoring element doesn't do anything useful in this context, so you should just drop it.
If you still don't see any metrics, please check for warnings or errors in the vespa log file (use vespa-logfmt) and the Telegraf log file: /opt/vespa/logs/telegraf/telegraf.log. (Vespa uses Telegraf internally to emit metrics to CloudWatch.)

I am trying to create a flash briefing skill. But i get error everytime when i deploy my skill

Error Message: There was an error parsing the feed. Please ensure that
the feed is valid JSON or RSS.
Please look into the code below:
[
{
"uid": "urn:uuid:99098951-c0b1-4881-b3f9-b07d49f594a9",
"updateDate": "2019-02-1T21:22:14.0Z",
"titleText": "A hello world audio clip",
"mainText": "",
"streamUrl": "https://feeds.feedburner.com/ndtvsports-cricket",
"redirectionUrl": "https://www.ndtv.com/rss"
},
{
"uid": "urn:uuid:a687993c-5664-4261-a938-79a8526097db",
"updateDate": "2019-02-1T23:57:33.0Z",
"titleText": "A hello world audio clip",
"mainText": "",
"streamUrl": "https://feeds.feedburner.com/ndtvcooks-latest",
"redirectionUrl": "https://www.ndtv.com/rss"
}
]
i have create a json file of the above code and uploaded into S3 bucket of aws.amazon.com
After the i copied the path of the file and than paste in the actual code.
Finally when i deploy with code. I get an error message...
Please help with it...
Goal to create an audio flash briefing skill without using alexa template.
even if i put same path in developer portal by using template.. i ended up with same error message.
please help to resolve this or suggest me on add more than one feed.
Try to adopt your URL. Seems that your are not using the real rss path but some website which is build around.
Try an url like this:
https://feeds.feedburner.com/ndtvsports-cricket?fmt=xml
See: Getting raw XML data from a Feedburner RSS feed
You need to give a real rss path and not a path to a website which is showing a feed. (details here: https://developer.amazon.com/docs/flashbriefing/flash-briefing-skill-api-feed-reference.html)

How to understand log files downloaded from App Engine?

What is the format of Google App Engine log files, as downloaded by appcfg.sh request_logs?
As far as I've been able to determine, the format of the log file is as follows:
CLIENT_IP_ADDRESS - USERNAME [DATE:TIME TIMEZONE] "METHOD URL HTTP/VERSION" RESPONSE_CODE ??? URL USER_AGENT
LOG_LEVEL:TIMESTAMP MESSAGE
:
LOG_LEVEL:TIMESTAMP MESSAGE
:
LOG_LEVEL:TIMESTAMP MESSAGE
:
Each of the indented lines is associated with the non-indented line above it - that's how you determine which request each of them relates to, I think. The solitary colons are used to separate one log message from the next.
The non-indented lines are in reverse chronological order, but the groups of indented lines below them are in chronological order.
A strange client IP address like 0.1.0.2 indicates a within-App-Engine post from your app to your app's task queue. Actually, though, you can also see that it is within-App-Engine by looking at the user agent for that request.

Resources