i am making a call to
authedClient.placeOrder(sellParams)
with params:
sellParams:any = {
'side': 'sell',
'product_id': 'BTC-USD',
'type': ‘market’,
’size’: 0.012613515
}
this throws error:
Error: HTTP 400 Error: size is too accurate. Smallest unit is 0.00000001
at Request._callback (/srv/node_modules/coinbase-pro/lib/clients/public.js:68:15)
at Request.self.callback (/srv/node_modules/request/request.js:185:22)
at emitTwo (events.js:126:13)
i am not sure why it fails. Please advise
It says that you should only place the size that is the increments of 0.00000001 (the base_increment below). While your size is at precision 9: 0.012613515, it is rejected.
Currently, I cannot find the base_increment in the /products endpoint, but in the status channel:
// Status Message
{
"type": "status",
"products": [
{
"id": "BTC-USD",
"base_currency": "BTC",
"quote_currency": "USD",
"base_min_size": "0.001",
"base_max_size": "70",
"base_increment": "0.00000001", // Here you go
"quote_increment": "0.01",
"display_name": "BTC/USD",
"status": "online",
"status_message": null,
"min_market_funds": "10",
"max_market_funds": "1000000",
"post_only": false,
"limit_only": false,
"cancel_only": false
}
],
...
}
I can add an update this, definitely the order float algebra is a little wonky and differs by exchange. For CB/CB Pro you'll want to get info on your base_increment for sell orders and quote_increment for buy orders, in string format, run a function like this:
def get_increments(ticker, auth_client):
products = auth_client.get_products()
for i in range(len(products)):
if products[i]['id'] == ticker:
base_incr = products[i]['base_increment']
quote_incr = products[i]['quote_increment']
return base_incr, quote_incr
i. Next, you'll want to utilize those increments to round down. I divide by the appropriate increment, use floor division by 1, then multiply by the same increment.
ii. To be safe, I get a decimal count again using string manipulation functions, and run something like round(qty, decimals). This cleans up the occasional lagging string of 9999999 or else 00000001.
iii. If you're looking to trade with 100% equity, use the funds argument in buy (quote asset value), and size argument in sell (base asset value).
Buy code looks something like:
decimals = int(str(quote_incr).find('1'))-int(str(quote_incr).find('.'))
quote_incr = float(quote_incr)
qty = (qty/quote_incr//1)*quote_incr
qty = str(round(qty,decimals))
auth_client.place_market_order(product_id=ticker, side='buy', funds=qty)
While sell code looks something like:
decimals = int(str(base_incr).find('1'))-int(str(base_incr).find('.'))
base_incr = float(base_incr_
qty = (qty/base_incr//1)*base_incr
qty = str(round(qty,decimals))
auth_client.place_market_order(product_id=ticker, side='buy', size=qty)
Related
Here is a post showing how to plot histogram in Google Looker Studio. The solution of the post works well when that data is almost "continuous", but some bins will not be shown when there is a big gap between data. For example, if my raw data is [1,2,3,4,5,6,7,8,9,1000], the bins between 10-999 will not be shown. How can I show those bins with zero y-value in Google Looker Studio?
My data source is a CSV file. A general solution is better.
There is no way to force zero bar entries with the standard graphs in Looker Studio. A solution can be the use of customer viz, such as the VEGA plugin.
Please use then following Vega code:
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"mark": {"type": "bar", "tooltip": true},
"encoding": {
"x": {
"bin": {"binned": true, "step": 1},
"field": "$dimension0"
},
"y": {"aggregate": "sum",
"field":"$metric0"
}
}
}
The step under bin sets the width of a bar.
For building non linear range, there is the need to create an extra column:
CASE
WHEN Age <= 5 THEN FLOOR(age/1) * 1
WHEN Age <= 20 THEN FLOOR(age/5) * 5
WHEN Age <= 100 THEN FLOOR(age/10) * 10
WHEN Age <= 1000 THEN FLOOR(age/100) * 100
ELSE -1
END
I have such schema
schema embeddings {
document embeddings {
field id type int {}
field text_embedding type tensor<double>(d0[960]) {
indexing: attribute | index
attribute {
distance-metric: euclidean
}
}
}
rank-profile distance {
num-threads-per-search:1
inputs {
query(query_embedding) tensor<double>(d0[960])
}
first-phase {
expression: distance(field, text_embedding)
}
}
}
and such query body:
body = {
'yql': 'select * from embeddings where ({approximate:false, targetHits:10} nearestNeighbor(text_embedding, query_embedding));',
"hits":10,
'input': {
'query(query_embedding)': [...],
},
'ranking': {
'profile': 'distance',
},
}
The thing is the output of this query returns different results depending on targetHits parameter. For example, the top-1 distance for targetHits: 10 is 2.847000, and the top-1 distance for targetHits: 200 is 3.028079.
More of that, if I perform the same query using vespa cli:
vespa query -t http://query "select * from embeddings where ([{\"targetHits\":10}] nearestNeighbor(text_embedding, query_embedding));" \
"approximate=false" \
"ranking.profile=distance" \
"ranking.features.query(query_embedding)=[...]"
I'm receiving the third result:
{
"root": {
"id": "toplevel",
"relevance": 1.0,
"fields": {
"totalCount": 10
},
"coverage": {
"coverage": 100,
"documents": 1000000,
"full": true,
"nodes": 1,
"results": 1,
"resultsFull": 1
},
"children": [
{
"id": "id:embeddings:embeddings::926288",
"relevance": 0.8158006540357854,
...
where as we can see top-1 distance is 0.8158
So, how can I perform the exact and not approximate nearest neighbors search, which results do not depend on any parameters?
Vespa sorts results by descending relevance score. When you use the distance rank-feature instead of closeness as the relevance score (your first-phase ranking expression), you end up inverting the order, so that more distant (worse) neighbors are ranked higher. As you increase targetHits you get even worse neighbors.
The correct query syntax for exact search is to set approximate:false:
select * from embeddings where ({approximate:false, targetHits:10} nearestNeighbor(text_embedding, query_embedding));
But you want to use closeness(field, text_embedding) in your first-phase ranking expression.
From https://docs.vespa.ai/en/nearest-neighbor-search.html
The closeness(field, image_embedding) is a rank-feature calculated by the nearestNeighbor query operator. The closeness(field, tensor) rank feature calculates a score in the range [0, 1], where 0 is infinite distance, and 1 is zero distance. This is convenient because Vespa sorts hits by decreasing relevancy score, and one usually want the closest hits to be ranked highest.
The first-phase is part of Vespa’s phased ranking support. In this example the closeness feature is re-used and documents are not re-ordered.
I have a Ruby on Rails application to enter results and create a league table for a football competition.
I'm trying to input some results by creating records in the database through heroku and I get error messages.
The application isn't perfectly designed: to enter the results, I have to create the fixtures and enter the score for each team. Then, independently I have to record each goal scorer, creating a record for each goal which is either associated with an existing player or requires me to firstly create a new player and then create the goal.
When I ran the code below heroku, I got this error:
syntax error, unexpected ':', expecting keyword_end
Maybe I'm missing something simple about lopping through an array within a hash?
Thank you for any advice!
coalition = Team.find_by(name: "Coalition")
moscow_rebels = Team.find_by(name: "Moscow Rebels")
red_star = Team.find_by(name: "Red Star")
unsanctionables = Team.find_by(name: "The Unsanctionables")
cavalry = Team.find_by(name: "Cavalry")
galactics = Team.find_by(name: "The Galactics")
happy_sundays = Team.find_by(name: "Happy Sundays")
hardmen = Team.find_by(name: "Hardmen")
international = Team.find_by(name: "International")
evropa = Venue.find_by(name: "Evropa")
s28 = Season.find_by(number: 28)
start_time = DateTime.new(2020,9,6,11,0,0,'+03:00')
scheduled_matches_1 =
[
{team_1: cavalry, team_1_goals: 1, team_1_scorers: ["Minaev"], team_2_goals: 6, team_2_scorers: ["Kovalev", "Kovalev", "Kovalev", "Thomas", "Thomas", "Grivachev"], team_2: coalition, time: start_time, venue: evropa, season: s28},
{team_1: hardmen, team_1_goals: 4, team_1_scorers: ["Jones", "Jones", "Jones", "Fusi"], team_2_goals: 2, team_2_scorers: ["Kazamula", "Ario"], team_2: galactics, time: start_time + 1.hour, venue: evropa, season: s28},
{team_1: international, team_1_goals: 9, team_1_scorers: ["Kimonnen", "Kimonnen", "Kimonnen", "Burya", "Burya", "Zakharyaev", "Zakharyaev", "Lavruk", "Rihter"], team_2_goals: 0, team_2_scorers: [], team_2: happy_sundays, time: start_time+2.hours, venue: evropa, season: s28}
]
scheduled_matches.each do |match|
new_fixture = Fixture.create(time: match[:time], venue: match[:venue], season: match[:season])
tf1 = TeamFixture.create(team: match[:team_1], fixture: new_fixture)
tf2 = TeamFixture.create(team: match[:team_2], fixture: new_fixture)
ts1 = TeamScore.create(team_fixture: tf1, total_goals: match{:team_1_goals})
ts2 = TeamScore.create(team_fixture: tf2, total_goals: match{:team_2_goals})
match[:team_1_scorers].each do |scorer|
if Player.exists?(team: tf1.team, last_name: scorer)
Goal.create(team_score: ts1, player: Player.find_by(last_name: scorer))
else
new_player = Player.create(team: tf1.team, last_name: scorer)
Goal.create(team_score: ts1, player: new_player)
end
end
match[:team_2_scorers].each do |scorer_2|
if Player.exists?(team: tf2.team, last_name: scorer_2)
Goal.create(team_score: ts2, player: Player.find_by(last_name: scorer_2))
else
new_player = Player.create(team: tf2.team, last_name: scorer_2)
Goal.create(team_score: ts2, player: new_player)
end
end
end
It looks like you are using braces when you meant to use brackets to access the hash. Below is one of the issues, but the same issue is in ts2.
ts1 = TeamScore.create(team_fixture: tf1, total_goals: match{:team_1_goals})
should be match[:team_1_goals]
ts1 = TeamScore.create(team_fixture: tf1, total_goals: match[:team_1_goals])
It may be because you have scheduled_matches_1 at the top and scheduled_matches.each do... further down.
But the real issue here is that your variable names match the data content, rather than being used to hold the content. If a new team joins your league, you have to change the code. Next week, you are going to have to change the hard-coded date value. Your scheduled_matches_1 data structure includes the active record objects returned by the first set of Team.findByName() calls. It would be easier to fetch these objects from the database inside your loops, and just hold the team name as a string in the hash.
There is some duplication too. Consider that each fixture has a home team and an away team. Each team has a name, and an array (possibly empty) of the players who scored. We don't need the number of goals; we can just count the number of players in the 'scorers' array. The other attributes, like the location and season belong to the fixture, not the team. So your hash might be better as
{
"fixtures": [
{
"home": {
"name": "Cavalry",
"scorers": [
"Minaev"
]
},
"away": {
"name": "Coalition",
"scorers": [
"Kovalev",
"Kovalev",
"Kovalev",
"Thomas",
"Thomas",
"Grivachev"
]
},
"venue": "Evropa",
"season": "s28"
}
]
}
because then you can create a reusable method to process each team. And maybe create a new method that returns the player (which it either finds or creates) which can be called by the loop that adds the goals.
Also, as it stands, I'm not sure the code can handle 'own goals', either. Perhaps something for a future iteration :)
I am trying to flatten a JSON file to be able to load it into PostgreSQL all in AWS Glue. I am using PySpark. Using a crawler I crawl the S3 JSON and produce a table. I then use an ETL Glue script to:
read the crawled table
use the 'Relationalize' function to flatten the file
convert the dynamic frame to a dataframe
try to 'explode' the request.data field
Script so far:
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = glue_source_database, table_name = glue_source_table, transformation_ctx = "datasource0")
df0 = Relationalize.apply(frame = datasource0, staging_path = glue_temp_storage, name = dfc_root_table_name, transformation_ctx = "dfc")
df1 = df0.select(dfc_root_table_name)
df2 = df1.toDF()
df2 = df1.select(explode(col('`request.data`')).alias("request_data"))
<then i write df1 to a PostgreSQL database which works fine>
Issues I face:
The 'Relationalize' function works well except the request.data field which becomes a bigint and therefore 'explode' doesn't work.
Explode cannot be done without using 'Relationalize' on the JSON first due to the structure of the data. Specifically the error is: "org.apache.spark.sql.AnalysisException: cannot resolve 'explode(request.data)' due to data type mismatch: input to function explode should be array or map type, not bigint"
If I try to make the dynamic frame a dataframe first then I get this issue: "py4j.protocol.Py4JJavaError: An error occurred while calling o72.jdbc.
: java.lang.IllegalArgumentException: Can't get JDBC type for struct..."
I tried to also upload a classifier so that the data would flatten in the crawl itself but AWS confirmed this wouldn't work.
The JSON format of the original file is as follows, that I an trying to normalise:
- field1
- field2
- {}
- field3
- {}
- field4
- field5
- []
- {}
- field6
- {}
- field7
- field8
- {}
- field9
- {}
- field10
# Flatten nested df
def flatten_df(nested_df):
for col in nested_df.columns:
array_cols = [c[0] for c in nested_df.dtypes if c[1][:5] == 'array']
for col in array_cols:
nested_df =nested_df.withColumn(col, F.explode_outer(nested_df[col]))
nested_cols = [c[0] for c in nested_df.dtypes if c[1][:6] == 'struct']
if len(nested_cols) == 0:
return nested_df
flat_cols = [c[0] for c in nested_df.dtypes if c[1][:6] != 'struct']
flat_df = nested_df.select(flat_cols +
[F.col(nc+'.'+c).alias(nc+'_'+c)
for nc in nested_cols
for c in nested_df.select(nc+'.*').columns])
return flatten_df(flat_df)
df=flatten_df(df)
It will replace all dots with underscore. Note that it uses explode_outer and not explode to include Null value in case array itself is null. This function is available in spark v2.4+ only.
Also remember, exploding array will add more duplicates and overall row size will increase. Flattening struct will increase column size. In short, your original df will explode horizontally and vertically. It may slow down processing data later.
Therefore my recommendation would be to identify feature related data and store only those data in postgresql and original json files in s3.
Once you have rationalized the json column, you don't need to explode it. Relationalize transforms the nested JSON into key-value pairs at the outermost level of the JSON document. The transformed data maintains a list of the original keys from the nested JSON separated by periods.
Example :
Nested json :
{
"player": {
"username": "user1",
"characteristics": {
"race": "Human",
"class": "Warlock",
"subclass": "Dawnblade",
"power": 300,
"playercountry": "USA"
},
"arsenal": {
"kinetic": {
"name": "Sweet Business",
"type": "Auto Rifle",
"power": 300,
"element": "Kinetic"
},
"energy": {
"name": "MIDA Mini-Tool",
"type": "Submachine Gun",
"power": 300,
"element": "Solar"
},
"power": {
"name": "Play of the Game",
"type": "Grenade Launcher",
"power": 300,
"element": "Arc"
}
},
"armor": {
"head": "Eye of Another World",
"arms": "Philomath Gloves",
"chest": "Philomath Robes",
"leg": "Philomath Boots",
"classitem": "Philomath Bond"
},
"location": {
"map": "Titan",
"waypoint": "The Rig"
}
}
}
Flattened out json after rationalize :
{
"player.username": "user1",
"player.characteristics.race": "Human",
"player.characteristics.class": "Warlock",
"player.characteristics.subclass": "Dawnblade",
"player.characteristics.power": 300,
"player.characteristics.playercountry": "USA",
"player.arsenal.kinetic.name": "Sweet Business",
"player.arsenal.kinetic.type": "Auto Rifle",
"player.arsenal.kinetic.power": 300,
"player.arsenal.kinetic.element": "Kinetic",
"player.arsenal.energy.name": "MIDA Mini-Tool",
"player.arsenal.energy.type": "Submachine Gun",
"player.arsenal.energy.power": 300,
"player.arsenal.energy.element": "Solar",
"player.arsenal.power.name": "Play of the Game",
"player.arsenal.power.type": "Grenade Launcher",
"player.arsenal.power.power": 300,
"player.arsenal.power.element": "Arc",
"player.armor.head": "Eye of Another World",
"player.armor.arms": "Philomath Gloves",
"player.armor.chest": "Philomath Robes",
"player.armor.leg": "Philomath Boots",
"player.armor.classitem": "Philomath Bond",
"player.location.map": "Titan",
"player.location.waypoint": "The Rig"
}
Thus in your case, request.data is already a new column flattened out from request column and its type is interpreted as bigint by spark.
Reference : Simplify/querying nested json with the aws glue relationalize transform
So, I'm trying to set up check_json.pl in NagiosXI to monitor some statistics. https://github.com/c-kr/check_json
I'm using the code with the modification I submitted in pull request #32, so line numbers reflect that code.
The json query returns something like this:
[
{
"total_bytes": 123456,
"customer_name": "customer1",
"customer_id": "1",
"indices": [
{
"total_bytes": 12345,
"index": "filename1"
},
{
"total_bytes": 45678,
"index": "filename2"
},
],
"total": "765.43gb"
},
{
"total_bytes": 123456,
"customer_name": "customer2",
"customer_id": "2",
"indices": [
{
"total_bytes": 12345,
"index": "filename1"
},
{
"total_bytes": 45678,
"index": "filename2"
},
],
"total": "765.43gb"
}
]
I'm trying to monitor the sized of specific files. so a check should look something like:
/path/to/check_json.pl -u https://path/to/my/json -a "SOMETHING" -p "SOMETHING"
...where I'm trying to figure out the SOMETHINGs so that I can monitor the total_bytes of filename1 in customer2 where I know the customer_id and index but not their position in the respective arrays.
I can monitor customer1's total bytes by using the string "[0]->{'total_bytes'}" but I need to be able to specify which customer and dig deeper into file name (known) and file size (stat to monitor) AND the working query only gives me the status (OK,WARNING, or CRITICAL). Adding -p all I get are errors....
The error with -p no matter how I've been able to phrase it is always:
Not a HASH reference at ./check_json.pl line 235.
Even when I can get a valid OK from the example "[0]->{'total_bytes'}", using that in -p still gives the same error.
Links pointing to documentation on the format to use would be very helpful. Examples in the README for the script or in the -h output are failing me here. Any ideas?
I really have no idea what your question is. I'm sure I'm not alone, hence the downvotes.
Once you have the decoded json, if you have a customer_id to search for, you can do:
my ($customer_info) = grep {$_->{customer_id} eq $customer_id} #$json_response;
Regarding the error on line 235, this looks odd:
foreach my $key ($np->opts->perfvars eq '*' ? map { "{$_}"} sort keys %$json_response : split(',', $np->opts->perfvars)) {
# ....................................... ^^^^^^^^^^^^^
$perf_value = $json_response->{$key};
if perfvars eq "*", you appear to be looking for $json_reponse->{"{total}"} for example. You might want to validate the user's input:
die "no such key in json data: '$key'\n" unless exists $json_response->{$key};
This entire business of stringifying the hash ref lookups just smells bad.
A better question would look like:
I have this JSON data. How do I get the sum of total_bytes for the customer with id 1?
See https://stackoverflow.com/help/mcve