I've got a lot of applications, each one have her own log. I'm typically in a distributed logs problematic.
To resolve this problematic, I'm using :
Brave / Spring Sleuth to generate trace and span ids
Elastic/Logstash/Kibana to centralize and search in my logs
This way, I'm able to see all the application logs in the order they where written (by their timestamp).
But the fact is that I can't trust the timestamp written in the logs. All the server's clocks are not synchronized.
So I need something which will help me to sort all my logs.
I've thought at generating a number at each span :
app1 : sequenceId = {}
app2 : sequenceId = {}
app3 : sequenceId = {}
app4 : sequenceId = {}
app5 : sequenceId = {}
app6 : sequenceId = {}
app7 : sequenceId = {}
app8 : sequenceId = {}
which would expect something which will order the logs in this order :
app1, app2, app3, app4, app5, app6, app7, app8
Does someone knows a technic which adresses this problematic ?
We have an issue for that in Sleuth (https://github.com/spring-cloud/spring-cloud-sleuth/issues/275). The easiest thing would be to add a tag / entry in the log that would increment with each child span. You can reuse the Sleuth's internals in such a way that you override the DefaultTracer's behaviour to add an additional tag on child span creation. Then in the Slf4jLogger you would reuse that tag to set an additional element in the MDC context
Related
Imagine I have an app1 called 'pricelists' and app2 called 'marketplaces'.
In the marketplaces app, I want to auto create a pricelists.PriceList if it is not yet present. This PriceList is to be used in signals to auto-populate the pricelist depending on a few factors.
Currently, I use something like this in my signals:
price_list, _ = PriceList.objects.get_or_create(
currency='EUR', is_default=False, customer_type='CONS',
remarks='Marketplace')
I don't like this approach since it's repeated a number of times and plainly want the pricelist to be created for sure.
My question. How can I get_or_create a model-object in another app every time django restarts?
Solution
In your app.__init__.py manually define your AppConfig. It doesn't seem to get detected in django 1.10
default_app_config = 'marketplaces.apps.MarketPlacesConfig'
Override your appconfig ready method:
class MarketPlacesConfig(AppConfig):
name = 'marketplaces'
def ready(self):
from pricelists.models import PriceList, PriceListItem
price_list_marketplaces, _ = PriceList.objects.get_or_create(
**settings.MARKETPLACES['price_list']
AppConfig.ready() with django.db.models.signals is the only way I can think of.
At some point in the future I may need to bulk load migration data (i.e. from a CSV). Has anyone had exceptions raised doing the following? Also is there any change in behaviour if the ndb.put_multi() function is used?
from google.appengine.ext import ndb
while True:
if not id:
break
id, name = read_csv_row(readline())
x = X(parent=ndb.Key('Y','static_id')
x.id, x.name = id, name
x.put()
class X(ndb.Model):
id = StringProperty()
name = StringProperty()
class Y(ndb.Model):
pass
def read_csv_row(line):
"""returns tuple"""
From my research and thanks to comments it seems that the code above (where it made into valid code) create problems which would eventually lead to google.appengine.api.datastore_errors.Timeout exceptions being thrown.
See another question:
Datastore write limit tests - trying to break app engine, but it won´t break ;)
The best suggestion I have so far is to use a Task Queue to to rate limit this. More information on:
blog.notdot.net/tag/deferred
I want to scan all records to check if there is not errors inside data.
How can I disable BadValueError to no break scan on lack of required field?
Consider that I can not change StringProperty to not required and such properties can be tenths in real code - so such workaround is not useful?
class A(db.Model):
x = db.StringProperty(required = True)
for instance in A.all():
# check something
if something(instance):
instance.delete()
Can I use some function to read datastore.Entity directly to avoid such problems with not need validation?
The solution I found for this problem was to use a resilient query, it ignores any exception thrown by a query, you can try this:
def resilient_query(query):
query_iter = iter(query)
while True:
next_result = query_iter.next()
#check something
yield next_result
except Exception, e:
next_result.delete()
query = resilient_query(A.query())
If you use ndb, you can load all your models as an ndb.Expando, then modify the values. This doesn't appear to be possible in db because you cannot specify a kind for a Query in db that differs from your model class.
Even though your model is defined in db, you can still use ndb to fix your entities:
# Setup a new ndb connection with ndb.Expando as the default model.
conn = ndb.make_connection(default_model=ndb.Expando)
# Use this connection in our context.
ndb.set_context(ndb.make_context(conn=conn))
# Query for all A kinds
for a in ndb.Query(kind='A'):
if a.x is None:
a.x = 'A more appropriate value.'
# Re-put the broken entity.
a.put()
Also note that this (and other solutions listed) will be subject to whatever time limits you are restricted to (i.e. 60 seconds on an App Engine frontend). If you are dealing with large amounts of data you will most likely want to write a custom map reduce job to do this.
Try setting a default property option to some distinct value that does not exist otherwise.
class A(db.Model):
x = db.StringProperty(required = True, default = <distinct value>)
Then load properties and check for this value.
you can override the _check_initialized(self) method of ndb.Model in your own Model subclass and replace the default logic with your own logic (or skip altogether as needed).
I got a question. I have a db table with settings (id, name).
If I read them from the db
$settings = $this->Setting->find('list');
How can I do this in the AppController or something like that to access from each Controller and Model?
Hope someone can help me.
Thanks
Explanation:
I would assume you're looking for something like below (Obviously you'll want to tweak it per your own application, but - it's the idea).
In the app controller, it
finds the settings from the table
repeats through each and puts each one into a "Configure" variable
Code:
/**
* Read settings from DB and populate them in constants
*/
function fetchSettings(){
$this->loadModel('Setting');
$settings = $this->Setting->findAll();
foreach($settings as $settingsData) {
$value = $settingsData['Setting']['default_value'];
//note: can't check for !empty because some values are 0 (zero)
if(isset($settingsData['Setting']['value'])
&& $settingsData['Setting']['value'] !== null
&& $settingsData['Setting']['value'] !== '') {
$value = $settingsData['Setting']['value'];
}
Configure::write($settingsData['Setting']['key'], $value);
}
}
Then, you can access them anywhere in your app via Configure::read('myVar');
A warning from the CakePHP book about Configure variables. (I think they're fine to use in this case, but - something to keep in mind):
CakePHP’s Configure class can be used to store and retrieve
application or runtime specific values. Be careful, this class allows
you to store anything in it, then use it in any other part of your
code: a sure temptation to break the MVC pattern CakePHP was designed
for. The main goal of Configure class is to keep centralized variables
that can be shared between many objects. Remember to try to live by
“convention over configuration” and you won’t end up breaking the MVC
structure we’ve set in place.
Was wondering if I'm unconsciously using the Put method in my last line of code ( Please have a look). Thanks.
class User(db.Model):
name = db.StringProperty()
total_points = db.IntegerProperty()
points_activity_1 = db.IntegerProperty(default=100)
points_activity_2 = db.IntegerProperty(default=200)
def calculate_total_points(self):
self.total_points = self.points_activity_1 + self.points_activity_2
#initialize a user ( this is obviously a Put method )
User(key_name="key1",name="person1").put()
#get user by keyname
user = User.get_by_key_name("key1")
# QUESTION: is this also a Put method? It worked and updated my user entity's total points.
User.calculate_total_points(user)
While that method will certainly update the copy of the object that is in-memory, I do not see any reason to believe that the change will be persisted to the the datastore. Datastore write operations are costly, so they are not going to happen implicitly.
After running this code, use the datastore viewer to look at the copy of the object in the datastore. I think that you may find that it does not have the changed total_point value.