Is there a way to tell how much an individual training job cost?
I can see the daily / hourly costs in the billing dashboard, which is a good proxy. I looked at the usage report as well, but didn't see a way to get the UsageValues to add up, and din't see tags coming through into the usage report.
When you are calling the DescribeTrainingJob API call, you are getting the following values: TrainingStartTime and TrainingEndTime. You are billed for the time interval between these times.
Now you need to get the next two values to complete the cost calculation, under ResourceConfig (in the same API call output): InstanceType, and InstanceCount.
Lastly, you can query the pricing API for the InstanceType you are using and get the price for the region you are running.
import boto3
pricing_client = boto3.client('pricing', region_name='us-east-1')
filterValue = instanceType + "-Training"
response = pricing_client.get_products(
ServiceCode='AmazonSageMaker',
Filters=[
{
'Type': 'TERM_MATCH',
'Field': 'instanceType',
'Value': filterValue
},
]
)
## TODO: fix this line to take the right region and not the first
python_dict = json.loads(response['PriceList'][0])
pricePerHour = next(iter(next(iter(python_dict['terms']['OnDemand'].values()))["priceDimensions"].values()))["pricePerUnit"]['USD']
return float(pricePerHour)
Related
A sample code of mine:
import ccxt
binance = ccxt.binance({
'enableRateLimit': True,
'apiKey': '****',
'secret': '****',
'options': {'defaultType': 'margin'}
})
binance.create_order('BTC/USDT', 'take_profit_limit', 'buy', 0.1, price = binance.fetch_ticker('BTC/USDT')['last'], params = {'type': 'takeProfit', 'stopPrice' : stop})
where stop > price and I get the following error:
ccxt.base.errors.OrderImmediatelyFillable: binance Stop price would
trigger immediately.
It seems to me that it is attempting to place a stop-loss at the price "stop" rather than a take-profit limit order which is what I want. I see on the documentation for the Binance API that the only extra parameter involved with the take_profit_limit order type is this stopPrice and not a similar "take_profit". I can also set a take-profit order the way I want to manually on the binance website by just setting this trigger price "stop" to be greater than the buying-in price, but I just can't get ccxt to do it.
I'm afraid I couldn't find anything to help in the Almighty Kroiter's examples either, but I may have missed something so I'm open to helpful links as well!
The take_profit_limit order type is meant to trigger a Buy when the price falls to the stop price and then you buy it at the limit price. If you want to buy after the price rises to a particular point, use the STOP_LOSS_LIMIT order type. If you want to buy right away simply use a LIMIT order.
I am a fairly new web developer and would need your help with a project I am currently working on. I have worked in the past on a very simple realtime database example and have little to none experience in firestore or NoSql in general.
I want to create a system which allows end-users to get an email once a week that contains a list of special offers from bars the end-user has subscribed to. The offers change each day of the week. Bar owners can fill out a form in a vue.js web application every week with their weekly special offers.
Every Monday morning a cron job has to look up which end user has subscribed to which bars and then aggregate the data and send it via email.
The question is how would you structure the data so that I can easily compose the email and send it via a cloud function?
My approach would be to have three main collections: RestaurantOwner, EndUser, SpecialOfferings
Please see the graphic for an example process:
BarOwner and EndUser are pretty straight forward. However, the difficult part is how to structure the SpecialOffers in order to be queried the right way.
My idea would be to structure it based on the calendar week and link it to the uid from the barOwner:
specialOffers: {
2019_CW27: {
barUID001: {
mon: {
title: 'Banana Daiquir',
price: 4.99,
},
tue: {
title: 'After Five',
price: 2.99,
},
wed: {
title: 'Cool Colada',
price: 6.99
},
thu: {
title: 'Crantini',
price: 5.99
},
fri: {
title: 'French Martini',
price: 4.99
}
},
barUID002: {
mon: {
title: 'Gin & Tonic',
price: 8.99,
},
tue: {
title: 'Cratini',
price: 4.99,
},
wed: {
title: 'French Martini',
price: 4.99
},
thu: {
title: 'After Five',
price: 3.99
},
fri: {
title: 'Cool Colada',
price: 6.99
}
}
},
2019_CW28: {
barUID01: {~~~},
barUID02: {~~~}
}
}
The disadvantage of this approach is that it creates a deeply nested object when you imagine that there are 52 calendar weeks, f.e 100 signed up bars à 5 special offers per week and I am not sure if I am able to query it the way I need to.
Is this approach reasonable or what would you do differently?
Thank you so much for your help! I highly appreciate it.
I'm assuming the following scenarios:
1) The bar owners make modifications to their offers very often.
2) The bar owners should be the only ones allowed to modify each bar's offers.
If you have these two scenarios, I would recommend a sub-collections approach here.
When to use sub-collections:
1) When there are lot of fields in a document. Cloud Firestore has 20,000 field limit. (If the number of Bars can exceed more than 20,000 fields)
2) When updating the parent collection is a common operation. Firestore only lets you update the document at rate of 1 write/second. (If the SpecialOffers information of each bar is modified very often. If two bar owners modify their offers, only 1 write is successful and the second write operation waits until the first is completed. This can delay the updation offers particularly at the end of a week when almost all the bars update the offers.)
3) When you want to limit the access to particular fields of a document. (If you want to restrict the access to a Bar's Offers to the barOwner alone. You can restrict the access to each document in the Bars sub-collection according to its owner using Firestore Security Rules)
So I would recommend a sub-collection Bars under the main collection SpecialOffers. This way the design becomes scalable and you can add restaurants and super-markets as other similar sub-collections in the future without heavily altering your design.
Another advantage is that sub-collections are basically collections and they don't have a limit for number of documents it can hold. So even if the number of bars registered is above 20,000 which is the limit of number of fields for a fire-store document, your sub-collection wont be having a problem but your document will run out of fields to save the offers for a new bar.
Ultimately the choice depends on your use cases.
Hope this helps.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 10 months ago.
Improve this question
Im going to create a achievement system in Mongodb. But im not sure how i would format/store it in the database.
As of the users should have a progress (on each achievement they would have some progress value stored), im really confused what would be the best way to perform this, and without having an performence issue.
what should i do?, cause i dont know, what i had in mind, was maybe something like:
Should i store each achievement in an unique row in a Achievement collection, and an user array within that row, containing object with userid and achievement progress?
Would i then get an performance issue when its 1000+ achievements, that is beeing checked fairy often?
or should i do something else?
example schema for the option above:
{
name:{
type:String,
default:'Achievement name'
},
users:[
{
userid:{
type:String,
default:' users id here'
},
progress:{
type:Number,
default:0
}
}
]
}
Even though the question is specifically about the database design, I will give a solution for the tracking/awarding logic as well to establish more accurate context for the db design.
I would store the achievements progress separately from the already awarded achievements for cleaner tracking and discovery.
The whole logic is event based and has multiple layers of event handling. This gives you TONS of flexibility on how you track your data and also gives you a pretty good mechanism to track history. Basically, you can look at it as a form of logging.
Of course, your system design and contracts are highly dependent on the information you're gonna be tracking and its complexity. A simple progress field may not suffice for each case(you might want to track something more complex, not a simple number between X and Y). There is also the case of tracking data which updates quite frequently(as distance travelled in games, for example). You didn't give any context on the topic of your achievement system so we're gonna stick with a generic solution. It's just a couple of things that you should take a note about as it will affect the design.
Okay, so, let's start from the top and track the entire flow for a tracked piece of data and its eventual achievement progress. Let's say we're tracking consecutive days of user login and we're gonna award him with an achievement when he reaches [10].
Note that everything below is just a pseudo-code.
So, let's say today is [8th of July, 2017]. For now, our User entity looks like this:
User: {
id: 7;
trackingData: {
lastLogin: 7 of July, 2017 (should be full DateTime object, but using this for brevity),
consecutiveDays: 9
},
achievementProgress: [
{
achievementID: 10,
progress: 9
}
],
achievements: []
}
And our achievements collection contains the following entity:
Achievement: {
id: 10,
name: '10 Consecutive Days',
rewardValue: 10
}
The user tries to login(or visit the site). The application handler takes note of that and after handling the login logic fires an event of type ACTION:
ACTION_EVENT = {
type: ACTION,
name: USER_LOGIN,
payload: {
userID: 7,
date: 8 of July, 2017 (should be full DateTime object, but using this for brevity)
}
}
We have an ActionHandler which listens for events of type ACTION:
ActionHandler.handleEvent(actionEvent) {
subscribersMap = Map<eventName, handlers>;
subscribersMap[actionEvent.name].forEach(subscriber => subscriber.execute(actionEvent.payload));
}
subscribersMap gives us a collection of handlers that should respond to each specific action(this should resolve to USER_LOGIN for us). In our case we can have 1 or 2 that concern themselves with updating the user tracking information of lastLogin and consecutiveDays tracking properties in the user entity. The handlers in our case will update the tracking information and fire new events further down the line.
Once again, for brevity, we're gonna incorporate both into one:
updateLoginHandler: function(payload) {
user = db.getUser(payload.userID);
let eventType;
let eventValue;
if (date - user.trackingData.lastLogin > 1 day) {
user.trackingData = 1;
eventType = 'PROGRESS_RESET';
eventValue = 1;
}
else {
const newValue = user.trackingData.consecutiveDays + 1;
user.trackingData.consecutiveDays = newValue;
eventType = 'PROGRESS_INCREASE';
eventValue = newValue;
}
user.trackingData.lastLogin = payload.date;
/* DISPATCH NEW EVENT OF TYPE ACHIEVEMENT_PROGRESS */
AchievementProgressHandler.dispatch({
type: ACHIEVEMENT_PROGRESS
name: eventType,
payload: {
userID: payload.userID,
achievmentID: 10,
value: eventValue
}
});
}
Here, PROGRESS_RESET have the same contract as the PROGRESS_INCREASE but have a different semantic meaning and I would keep them separate for history/tracking purposes. If you wish, you can combine them into a single PROGRESS_UPDATE event.
Basically, we update the tracked fields that are dependent on the lastLogin date and fire a new ACHIEVEMENT_PROGRESS event which should be handled by a separate handler with the same pattern(AchievementProgressHandler). In our case:
ACHIEVEMENT_PROGRESS_EVENT = {
type: ACHIEVEMENT_PROGRESS,
name: PROGRESS_INCREASE
payload: {
userID: 7,
achievementID: 10,
value: 10
}
}
Then, in AchievementProgressHandler we follow the same pattern:
AchievementProgressHandler: function(event) {
achievementCheckers = Map<achievementID, achievementChecker>;
/* update user.achievementProgress code */
switch(event.name): {
case 'PROGRESS_INCREASE':
achievementCheckers[event.payload.achievementID].execute(event.payload);
break;
case 'PROGRESS_RESET':
...
}
}
achievementCheckers contains a checker function for each specific achievement that decides if the achievement has reached its desired value(a progress of 100%) and should be awarded. This enables us to handle all kinds of complex cases. If you only track a single X out of Y scenario, you can share the function between all achievements.
The handler basically does this:
achievementChecker: function(payload) {
achievementAwardHandler;
achievement = db.getAchievement(payload.achievementID);
if (payload.value >= achievement.rewardValue) {
achievementAwardHandler.dispatch({
type: ACHIEVEMENT_AWARD,
name: ACHIEVEMENT_AWARD,
payload: {
userID: payload.userID,
achievementID: achievementID,
awardedAt: [current date]
}
});
/* Here you can clear the entry from user.achievementProgress as you no longer need it. You can also move this inside the achievementAwardHandler. */
}
}
We once again dispatch an event and use an event handler - achievementAwardHandler. You can skip the event creation step and award the achievement to the user directly but we keep it consistent with the whole history logging flow.
An added benefit here is that you can use the handler to defer the achievement awarding to a specific later time thus effectively batching awarding for multiple users, which serve a couple of purposes including performance enhancement.
Basically, this pseudo code handles the flow from [a user action] to [achievement rewarding] with all intermediate steps included. It's not set in stone, you can modify it as you like but all in all, it gives you a clean separation of concerns, cleaner entities, it's performant, let's you add complex checks and handlers which are easy to reason about while in the same time provide a great history log of the user overall progress.
Regarding the DB schema entities, I would suggest the following:
User: {
id: any;
trackingData: {},
achievementProgress: {} || [],
achievements: []
}
Where:
trackingData is an object that contains everything you're willing
to track about the user. The beauty is that properties here are
independent from achievement data. You can track whatever and eventually use it for achievement purposes.
achievementProgress: a map of <key: achievementID, value: data> or
an array containing the current progress for each achievement.
achievements: an array of awarded achievements.
and Achievement:
Achievement: {
id: any,
name: any,
rewardValue: any (or any other field/fields. You have complete freedom to introduce any kind of tracking with the approach above),
users?: [
{
userID: any,
awardedAt: date
}
]
}
users is a collection of users who have been rewarded the given achievement. This is optional and is here only if you have the use for it and query for this data frequently.
What you might be looking for is a Badge style implementation. Just like Stack Overflow rewards it's users with badges for specific achievements.
Method 1: You can have flags in the user profile for each badge. Since you're doing it in NoSQL database, you just have to set a flag for each badge.
const badgeSchema = new mongoose.Schema({
badgeName: {
type: String,
required: true,
},
badgeDescription: {
type: String,
required: true,
}
});
const userSchema = new mongoose.Schema({
userName: {
type: String,
required: true,
},
badges: {
type: [Object],
required: true,
}
});
If your application architecture is event based, you can trigger awarding badges to users. And that operation is just inserting Badge object with progress in User badges array.
{
badgeId: ObjectId("602797c8242d59d42715ba2c"),
progress: 10
}
Update operation will be to find and update the badges array with progress percentage number
And while displaying user achievements on user interface, you can just loop over badges array to show the badges this user has achieved and their progress with it.
Method 2: Have a separate mongo collection for Badge and User Mapping. Whenever a user achieves a badge you insert a record in that collection. It will be one to one mapping of user _id and badge _id and progress value. But as the table will grow huge you will need to do indexing to efficiently query user and badge mapping.
You will have to do analysis on best approach according to your specific use case.
MongoDB is flexible enough to allow teams develop applications quickly, and involve their model with litter friction as the application needs it. In cases where you need a robust model from day one, theirs is a flexible methodology that can guide you through the process of modeling your data.
The methodology is composed of:
Workload: This stage is about gathering as much information as possible to understand your data. This will allow you formulate assumptions about, you data size the operations that will be performance against it (reads and writes), quantify operations and qualify operations.
You can get this by:
Scenarios
Prototype
Production Logs & Stats (if you are migrating).
Relationships: Identify the relationship between the different entities in your data, quantify those relationships and apply embedding or linking. In general you should prefer embedding by default, but remember that arrays should not grow without bound (6 Rules of Thumb for MongoDB Schema Design: Part 3).
Patterns: Apply schema design patterns. Take a look at Building with Patterns: A Summary, it presents a matrix that highlights the pattern that could be useful for a given use case.
Finally, the goal of this methodology is help you create a model, that can scale and perform well under stress.
If you design the achievement schema like this:
{
name: {
type: String,
default: "Achievement name",
},
userid: {
type: String,
default: " users id here",
},
progress: {
type: Number,
default: 0,
},
}
}
When an achievement is gained you just add another entry
for getting achievements Map-Reduce is a good candidate for running map reduce on the database. you can run them on a less regular basis, using them for offline computation of the data that you want.
based on documentation you can do like the following photo
Ok Im starting out fresh with Firebase. I've read this: https://www.firebase.com/docs/data-structure.html and I've read this: https://www.firebase.com/blog/2013-04-12-denormalizing-is-normal.html
So I'm suitably confused as one seems to contradict the other. You can structure your data hierarchically, but if you want it to be scalable then don't. However that's not the actual problem.
I have the following structure (please correct me if this is wrong) for a blog engine:
"authors" : {
"-JHvwkE8jHuhevZYrj3O" : {
"userUid" : "simplelogin:7",
"email" : "myemail#domain.com"
}
},
"posts" : {
"-JHvwkJ3ZOZAnTenIQFy" : {
"state" : "draft",
"body" : "This is my first post",
"title" : "My first blog",
"authorId" : "-JHvwkE8jHuhevZYrj3O"
}
}
A list of authors and a list of posts. First of all I want to get the Author where the userUid equals my current user's uid. Then I want to get the posts where the authorId is the one provided to the query.
But I have no idea how to do this. Any help would be appreciated! I'm using AngularFire if that makes a difference.
Firebase is a NoSQL data store. It's a JSON hierarchy and does not have SQL queries in the traditional sense (these aren't really compatible with lightning-fast real-time ops; they tend to be slow and expensive). There are plans for some map reduce style functionality (merged views and tools to assist with this) but your primary weapon at present is proper data structure.
First of all, let's tackle the tree hierarchy vs denormalized data. Here's a few things you should denormalize:
lists you want to be able to iterate quickly (a list of user names without having to download every message that user ever wrote or all the other meta info about a user)
large data sets that you view portions of, such as a list of rooms/groups a user belongs to (you should be able to fetch the list of rooms for a given user without downloading all groups/rooms in the system, so put the index one place, the master room data somewhere else)
anything with more than 1,000 records (keep it lean for speed)
children under a path that contain 1..n (i.e. possibly infinite) records (example chat messages from the chat room meta data, that way you can fetch info about the chat room without grabbing all messages)
Here's a few things it may not make sense to denormalize:
data you always fetch en toto and never iterate (if you always use .child(...).on('value', ...) to fetch some record and you display everything in that record, never referring to the parent list, there's no reason to optimize for iterability)
lists shorter than a hundred or so records that you always as a whole (e.g. the list of groups a user belongs to might always be fetched with that user and would average 5-10 items; probably no reason to keep it split apart)
Fetching the author is as simple as just adding the id to the URL:
var userId = 123;
new Firebase('https://INSTANCE.firebaseio.com/users/'+userId);
To fetch a list of posts belonging to a certain user, either maintain an index of that users' posts:
/posts/$post_id/...
/my_posts/$user_id/$post_id/true
var fb = new Firebase('https://INSTANCE.firebaseio.com');
fb.child('/my_posts/'+userId).on('child_added', function(indexSnap) {
fb.child('posts/'+indexSnap.name()).once('value', function(dataSnap) {
console.log('fetched post', indexSnap.name(), dataSnap.val());
});
});
A tool like Firebase.util can assist with normalizing data that has been split for storage until Firebase's views and advanced querying utils are released:
/posts/$post_id/...
/my_posts/$user_id/$post_id/true
var fb = new Firebase('https://INSTANCE.firebaseio.com');
var ref = Firebase.util.intersection( fb.child('my_posts/'+userId), fb.child('posts') );
ref.on('child_added', function(snap) {
console.log('fetched post', snap.name(), snap.val();
});
Or simply store the posts by user id (depending on your use case for how that data is fetched later):
/posts/$user_id/$post_id/...
new Firebase('https://INSTANCE.firebaseio.com/posts/'+userId).on('child_added', function(snap) {
console.log('fetched post', snap.name(), snap.val());
});
I have built classified website using Yii php framework. Now it is getting a lot of traffic. So I want to using caching to optimize the performance of the website.
There are two controllers I want to optimize.
One is the thread list controller: (example) http://www.shichengbbs.com/category/view/id/15
The other one the the thread controller: (example) http://www.shichengbbs.com/info/view/id/67900
What I have done:
the thread list is cached for 3mins.(The other option is update the thread list only when new thread comes)
set the last-modified time HTTP header for the thread view. (expire time is not set, as some user complain that the page appears unchanged after editing)
Partial caching the categories navigation fragment.(It appears on the left side of every page)
Use htaccess to set expire header for img/html/css/js.
Considered database sql caching for the thread list, but not done. As I thought it is the same as 1.
What else can I do to improve the website performance?
I assume you have done the Performance Tuning guide point 1 and 3. It's really helpful.
For number 2 you can use the CHttpCacheFilter
class CategoryController extends Controller {
private $_categoryLastUpdate;
public function filters(){
return array(
array(
'CHttpCacheFilter + view',
'cacheControl' => " max-age=604800, must-revalidate",
'etagSeedExpression' => function() {
return $this->getCategoryLastUpdate();
}
'lastModifiedExpression' => function() {
return $this->getCategoryLastUpdate();
}
)
)
}
public function actionView($id){
$object = Category::model()->findByPk($_GET['id']);
$this->render('view', array('object' => $object));
}
public function getCategoryLastUpdate(){
if (!isset($this->_categoryLastUpdate)){
$obj = Category::model()->findByPk($_GET['id'], array('select' => 'lastUpdate'));
$this->_categoryLastUpdate
}
return $this->_categoryLastUpdate;
}
}
It basically will calculate the ETag and LastUpdate by the category. And to save the query, it will first only calculate the lastUpdate of the Category object.
And for number one, you can always use the CCacheDependency. Just make a field in the thread list object, e.g. lastUpdate. And when a new thread submitted, just update the field and use it for the CCacheDependency.
Since I see you are using a very large pagination, I think you want to read about Four Ways to Optimize Paginated Displays (if you use MySQL for your database and thread search/list).
Try using a Cache Manager with Memcache or APC. For example, http://code.google.com/p/memcache-flag/ . When you edit the list, then you can invalidate the cache item or tag. I suppose it could also just be done with regular APC / Memcache functions if you design is simple (set a key and delete it when it is no longer valid).
Use this to store serialized (or automatically serialized) data instead of retrieving it from mysql.