I want to send a transaction in RSK network and I get this message in logs: Not enough gas for transaction execution.
I got the gas limit parameter from my testing environment, using web3.eth.estimateGas.
RSK nodes have a JSON-RPC for eth_estimateGas,
which is the most reliable way to perform gas estimations.
You can do this from the terminal using curl:
curl \
-X POST \
-H "Content-Type:application/json" \
--data '{"jsonrpc":"2.0","method":"eth_estimateGas","params":[{"from": "0x560e6c06deb84dfa84dac14ec08ed093bdd1cb2c", "to": "0x560e6c06deb84dfa84dac14ec08ed093bdd1cb2c", "gas": "0x76c0", "gasPrice": "0x3938700", "value": "0x9184e72a", "data": "" }],"id":1}' \
http://localhost:4444
{"jsonrpc":"2.0","id":1,"result":"0x5208"}
Alternatively, using web3.js:
web3.eth.estimateGas({"to": "0x391ec8a27d29a42c7601651d2f38b1e1895e27a1", "data": "0xe26e496319a16c8ccae126f4aac7e3010123927a4739288cd1ace12feafae9a2"})
23176
While this is the same JSON-RPC found in geth (Ethereum) and other Ethereum-compatible nodes,
note that the gas calculations in RSK and Ethereum are different.
Thus their implementations differ.
For example, the price of certain VM opcodes are different.
Another notable point of difference related to gas estimation,
is that Ethereum implements EIP-150,
whereas RSK does not.
This means that the 1/64 reduction in gas estimation does not apply to RSK.
(The detailed implications of this on gas estimation are perhaps beyond the scope of this question.)
This means that you will expect incorrect values when running against ganache-cli (previously testrpc),
which is used by default in common developer tools such as Truffle.
To get the correct gas,
using the RSK-specific calculations,
the best way is to use RSK Regtest
when invoking eth_estimateGas
for local development and testing.
In other scenarios you may also use
RSK Testnet and Mainnet.
The following other scenarios are also relevant, but not directly related to your question, but are also good to know:
When invoking smart contract functions
that have the pure or view modifiers,
no gas (and therefore gas estimation) is necessary.
When performing certain transactions that have a define invariant gas price,
simply you may use that as a hard-coded constant.
For example the transfer of the native currency (RBTC in this case),
the invariant gas price is 21000.
This assumes that no data (sometimes referred to as "message")
was sent with the transaction.
Related
I am looking at using Blockfrost.io API in order to read cardano transactions, I am looking to get the bare minimum which is:
Address from
Address to
Assets transfered (type + amount)
Fees
So far I can not find how to retrieve a transaction addresses from and to while using:
https://docs.blockfrost.io/#tag/Cardano-Transactions/paths/~1txs~1%7Bhash%7D/get
Am I missing something?
So to answer my question:
Cardano does use something called "utxos" for the way it handles transactions and I would invite everyone to read about these.
Regarding Blockfrost.io, this means you need to have a look at the transactions api:
https://docs.blockfrost.io/#tag/Cardano-Transactions/paths/%7E1txs%7E1%7Bhash%7D/get
and also combine it with the utxos api:
https://docs.blockfrost.io/#tag/Cardano-Transactions/paths/~1txs~1{hash}~1utxos/get
There is an AWS document that explains how to do it for oneself, i.e. how to allow only one's pages to hotlink and reject all others: https://aws.amazon.com/blogs/security/how-to-prevent-hotlinking-by-using-aws-waf-amazon-cloudfront-and-referer-checking/
I'd like to know if WAF is the right choice for my use case, which is a bit different from the one above.
At the company I work for, we intend to sell data through a JS widget.
We'd like to restrict access to those data so that only authorized REFERERs are able to show our data to their users, while rejecting all other REFERERs.
The possibility of spoofing the REFERER is not an important threat for us.
We expect to grow our customer base to some hundreds.
The reason I'm asking this question is due to noticing that there are some strict limits on WAF: https://docs.aws.amazon.com/waf/latest/developerguide/limits.html, according to which I understand that for our use case, WAF wouldn't scale nicely.
WAF is not the right tool for that job.
First, even if there are max 10 rules, each with max 10 conditions, each with max 10 filters, there is a strong max of 100 string conditions per AWS account.
Second, conditions and filters do not compose nicely for our use case. Conditions of a rule get composed with an AND and filters of a condition get composed with an OR. For example, a rule like r(x) := (x=a + x=b + x=c) * (x=d + x=e) would give r(d) = false without ever getting to test x=d.
In my case scenario, Flink is sending the metrics to Datadog. Datadog Host map is as shown below { I have no Idea why is showing me latency here }
Flink metrics are sent to localhost. The issue here is that when
flink-conf.yaml file configuration is as follows
# adding metrics
metrics.reporters: stsd , dghttp
metrics.reporter.stsd.class: org.apache.flink.metrics.statsd.StatsDReporter
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125
# for datadog
metrics.reporter.dghttp.class: org.apache.flink.metrics.datadog.DatadogHttpReporter
metrics.reporter.dghttp.apikey: xxx
metrics.reporter.dghttp.tags: host:localhost, job_id : jobA , tm_id : task1 , operator_name : operator1
metrics.scope.operator: numRecordsIn
metrics.scope.operator : numRecordsInPerSecond
metrics.scope.operator : numRecordsOut
metrics.scope.operator : numRecordsOutPerSecond
metrics.scope.operator : latency
The issue is that Datadog is showing 163 metrics which I don't understand, which I will explain in a while
I don't understand the metrics format in datadog as it shows me metrics something like this
Now as shown in above Image
Latency is expressed in time
Number of events per second is event /sec
count is some value
So my question is that which metric is this?
Also, the execution plan of my job is something like this
How do I relate the metrics in Datadog with execution plan operators in Flink?
I have read in Flink API 1.3.2 that I can use tags, I have tried to use them in flink-conf.yaml file but I don't have complete Idea what sense they make here.
My ultimate goal is to find operator latency, number of records out and in /second at each operator in this case
There are a variety of issues here.
1. You've misconfigured the scope formats. (metrics.scope.operator)
For one the configuration doesn't make sense since you specify "metrics.scope.operator" multiple times; only the last config entry is honored.
Second, and more importantly, you have misunderstood for scope formats are used for.
Scope formats configure which context information (like the ID of the task) is included in the reported metric's name.
By setting it to a constant ("latency") you've told Flink to not include anything. As a result, the numRecordsIn metrics for every operator is reported as "latency.numRecordsIn".
I suggest to just remove your scope configuration.
2. You've misconfigured the Datadog Tags
I do not understand what you were trying to do with your tags configuration.
The tags configuration option can only be used to provide global tags, i.e. tags that are attached to every single metrics, like "Flink".
By default every metric that the Datadog reports has tags attached to it for every available scope variable available.
So, if you have an operator name A, then the numRecordsIn metric will be reported with a tag "operator_name:A".
Again, I would suggest to just remove your configuration.
Suppose I have normal "Add Users" module that I want to automate using Java scripting, how I can avoid data duplication to avoid error message such as "User already exist"?
There are numerous ways in which this can be automated. You are receiving 'User already exists' due to fact that you're (probably) running your 'Add Users' test cases using static variables.
Note: For the following examples I will consider a basic registration flow/scenario of a new user: name, email, password being the required fields.
Note-002: My language of choice will be JavaScript. You should be able to reproduce the concept with Java with ease.
1.) Pre-pending/Post-pending a unique identifier to the information you're submitting (e.g.: Date() returns the number of seconds that have elapsed since January 1, 1970 => it will always be unique when running your test case)
var timestamp = Number(new Date());
var email = 'test.e2e' + timestamp + '#<yourMainDomainHere>'
Note: Usually, the name & password don't need to be unique, so you can actually use hardcoaded values without any issues.
2.) The same thing can also be achieved using the Math.random() (for JS), which returns a value between 0 and 1 (0.8018194703223693), 18 digits long.
var almostUnique = Math.random();
// You can go ahead and gen only the decimals
almostUnique = almostUnique.toString().split('.')[1];
var email = 'test.e2e' + almostUnique + '#<yourMainDomainHere>'
!!! Warning: While Math.random() is not actually unique, in hundreds of regression runs of 200 functional test cases, I didn't have the chance of seeing a duplicate.
3.) (Not so elegant | Harder exponentially harder to implement) If you have access to your web-apps backend API and through it you can execute different actions in the DB, then you can actually write yourself some scripts that will run after your registration test cases, like a cleanup suite.
These scripts will have to remove the previously added user from your database.
Hope this helps!
You don't mention what language you are coding in, but use whatever language you use's random function to generate random numbers and/or text for the user id. It won't guarantee that there will not be a duplicate, but the nature of testing is such that you should be able to handle both situations anyway. If this is not clear or I don't understand your question correctly, you'll need to provide a lot more information: what you've tried, what language you use, etc.
Edit: See my answer. Problem was in our code. MR works fine, it may have a status reporting problem, but at least the input readers work fine.
I ran an experiment several times now and I am now sure that mapreduce (or DatastoreInputReader) has odd behavior. I suspect this might have something to do with key ranges and splitting them, but that is just my guess.
Anyway, here's the setup we have:
we have an NDB model called "AdGroup", when creating new entities
of this model - we use the same id returned from AdWords (it's an
integer), but we use it as string: AdGroup(id=str(adgroupId))
we have 1,163,871 of these entities in our datastore (that's what
the "Datastore Admin" page tells us - I know it's not entirely
accurate number, but we don't create/delete adgroups very often, so
we can say for sure, that the number is 1.1 million or more).
mapreduce is started (from another pipeline) like this:
yield mapreduce_pipeline.MapreducePipeline(
job_name='AdGroup-process',
mapper_spec='process.adgroup_mapper',
reducer_spec='process.adgroup_reducer',
input_reader_spec='mapreduce.input_readers.DatastoreInputReader',
mapper_params={
'entity_kind': 'model.AdGroup',
'shard_count': 120,
'processing_rate': 500,
'batch_size': 20,
},
)
So, I've tried to run this mapreduce several times today without changing anything in the code and without making changes to the datastore. Every time I ran it, mapper-calls counter had a different value ranging from 450,000 to 550,000.
Correct me if I'm wrong, but considering that I use the very basic DatastoreInputReader - mapper-calls should be equal to the number of entities. So it should be 1.1 million or more.
Note: the reason why I noticed this issue in the first place is because our marketing guys started complaining that "it's been 4 days after we added new adgroups and they still don't show up in your app!".
Right now, I can think of only one workaround - write all keys of all adgroups into a blobstore file (one per line) and then use BlobstoreLineInputReader. The writing to blob part would have to be written in a way that does not utilize DatastoreInputReader, of course. Should I go with this for now, or can you suggest something better?
Note: I have also tried using DatastoreKeyInputReader with the same code - the results were similar - mapper-calls were between 450,000 and 550,000.
So, finally questions. Is it important how you generate ids for your entities? Is it better to use int ids instead of str ids? In general, what can I do to make it easier for mapreduce to find all of my entities mapping them?
PS: I'm still in the process of experimenting with this, I might add more details later.
After further investigation we have found that the error was actually in our code. So, mapreduce actually works as expected (mapper is called for every single datastore entity).
Our code was calling some google services functions that were sometimes failing (the wonderful cryptic ApplicationError messages). Due to these failures, MR tasks were being retried. However, we have set a limit on taskqueue retries. MR did not detect nor report this in any way - MR was still showing "success" in the status page for all shards. That is why we thought that everything is fine with our code and that there is something wrong with the input reader.