Is there a way to to get my autoscaled application's VersionID in my init() function without a Context? The only available option seems to be appengine.VersionID(context.Context). Manually scaled instances have /_ah/start called when they start up (giving access to a Context), but there is nothing like this for autoscaled instances.
I am not caring about the generated ID that appengine.VersionID returns with it, just the app.yaml version.
EDIT: A bit of context: I am wanting to deploy versions in the form x-x-x-dev or x-x-x-live and have my database connection depend on the version suffix. This way, when I look in the GCP console, I can be certain which deployed modules/services are using which database. Of course, I setup my DB connection pool in the init(), which has no access to a Context.
I searched and searched with no answers online anywhere, so here it is.
Simply parse the app.yaml file in your init() function. My example here uses a yaml parsing package, but it can be done more lightweight if you need.
import "github.com/ghodss/yaml"
type AppVersion struct {
Version string `json:"version"`
}
func VersionID() (string, error) {
dat, err := ioutil.ReadFile("app.yaml")
if err != nil {
return "", err
}
a := &AppVersion{}
err = yaml.Unmarshal(dat, a)
if err != nil {
return "", err
}
return a.Version, nil
}
Note that this DOES NOT return the generated ID in the form X.Y that appengine.VersionID() does. Only the X part of the version.
As an aside, in the appengine repo on Github, the actual call to appengine.VersionID requires a Context, but internally calls the internal package with nil. So they basically force you to call it with a Context, but it isn't actually used. It's incredibly infuriating.
EDIT: It should be noted that the new Go SDK in gcloud no longer supports version in the app.yaml, as it is now a required parameter at deploy. However, the "legacy" SDK is still supported and maintained, which I am continuing to use as of today (12/24/2018).
Related
Google App Engine now supports Go 1.11 via the new second-generation standard environment. While converting an older standard environment application to the second generation, it wasn't obvious how to combine tracing information from the app engine infrastructure with custom tracing I added to the application using OpenCensus.
Even though I had created a stackdriver exporter and registered traces, I wasn't seeing custom trace information in the stackdriver console attached to inbound requests.
The key is I was missing is understanding how span context is communicated to the serving app. Google leverages the X-Cloud-Trace-Context header to propagate span context within requests sent to your serving instances, and the go.opencensus.io/exporter/stackdriver/propagation library provides an implementation to extract and persist this information within http requests.
Don't forget to create a stackdriver exporter, and register traces to it. The docs for the exporter library show an example of this.
// CreateSpanFromRequest returns a context and span based on the http.Request.
// If no existing spancontext is found, this will start a new span.
// Modifies existing request to contain the generated span's context.
func CreateSpanFromRequest(name string, r *http.Request) (context.Context, *trace.Span) {
var span *trace.Span
ctx := r.Context()
httpFormat := &propagation.HTTPFormat{}
sc, ok := httpFormat.SpanContextFromRequest(r)
if ok {
ctx, span = trace.StartSpanWithRemoteParent(ctx, name, sc)
} else {
ctx, span = trace.StartSpan(ctx, name)
}
// Write the span context into the http.Request. We do this to
// to enable chaining handlers together more easily.
httpFormat.SpanContextToRequest(span.SpanContext(), r)
return ctx, span
}
Using this, I was able to add custom spans to my handlers that would be properly associated with the incoming request information in stackdriver:
func indexHandler(w http.ResponseWriter, r *http.Request) {
_, span := CreateSpanFromRequest("indexHandler", r)
defer span.End()
if r.URL.Path != "/" {
http.NotFound(w, r)
return
}
fmt.Fprint(w, "Hello, World!")
}
My app runs on App Engine Standard and the Go runtime.
I have this trace for my recent request:
There is a big gap between the "urlfetch" span and the "datastore_v3" span, because my app processes some CPU-bound computation for ~1000ms.
I would love to programmatically add my computation as a custom span into the Trace view, and get something like this:
Is there a way to do this in my app written in go? (source here)
It appears it might be possible. From Setting Up Stackdriver Trace for Go:
Alpha
This is an alpha release of the OpenCensus package for Go. These
libraries might be changed in backward-incompatible ways and are not
recommended for production use. They are not subject to any SLA or
deprecation policy.
Stackdriver Trace can be used by Go applications using the
OpenCensus package for Go.
Stackdriver Trace's Go support is provided by OpenCensus, a set
of tracing and application metrics instrumentation libraries that work
with multiple backends. The latest details about OpenCensus for Go,
along with additional documentation and examples, can be found on its
GitHub page.
Support is enabled by default in the flexible environment, however the docs make no mention about the standard environment (if that's your case I'd say just give it a try). From App Engine:
On Google App Engine flexible environment, the Stackdriver Trace API
access scope is enabled by default, and the OpenCensus client
library can be used without needing to provide credentials or a
project ID.
An application code sample is provided on the same page.
I could make it work with the new AppEngine runtime for Go 1.11 (currently in beta) and OpenCensus with the Stackdriver exporter.
In order to attach my custom span to the main Trace of the request, I use this utility func:
// Start a new span "With Remote Parent"
func startSpanfWRT(r *http.Request, msg string, args ...interface{}) (c2 context.Context, endSpan func()) {
caption := fmt.Sprintf(msg, args...)
c := r.Context()
spanContext, ok := (&propagation.HTTPFormat{}).SpanContextFromRequest(r)
if !ok {
return c, func() {}
}
var span *trace.Span
c2, span = trace.StartSpanWithRemoteParent(c, caption, spanContext)
endSpan = func() {
span.End()
}
return c2, endSpan
}
Note that it requires the *http.Request as argument (a context.Context wouldn't be enough here).
Here is the sample app source code.
As a span needs to be started and then later stopped, the start func returns an "end" callback, and a new Context as well.
It is fine to call startSpanfWRT multiple times, and they may overlap. It requires to pass the *http.Request around, which is not super-convenient (usually we only pass Contexts around).
However, after a call to startSpanfWRT, you may add children spans conveniently, just paying attention to the respective Contexts:
c2, childSpan := trace.StartSpan(c, caption)
I am trying to develop an application using SiriKit to get the car door lock status and set the same from Siri. I followed this blog https://www.appcoda.com/sirikit-introduction/ and did all the setup replacing the INStartWorkoutIntent with INGetCarLockStatusIntent.
But when i try to set the vocabulary for carName, the application is getting crashed with following exception,
Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Illegal attempt to provide vocabulary of type INVocabularyStringTypeCarName by an app that does not handle any intents that could use that type of vocabulary'
The source code that i am using to set the vocabulary is,
INPreferences.requestSiriAuthorization { (status) in
}
INVocabulary.shared().setVocabularyStrings(["benz", "bmw", "audi"], of: .carName)
In AppDelegate, i have added the following method,
func application(_ application: UIApplication, continue userActivity: NSUserActivity, restorationHandler: #escaping ([Any]?) -> Void) -> Bool {
guard let intent = userActivity.interaction?.intent as? INGetCarLockStatusIntent else {
print("AppDelegate: Start Workout Intent - FALSE")
return false
}
print("AppDelegate: Start Workout Intent - TRUE")
print(intent)
return true
}
Also created the extension for intent handler and implemented INSetCarLockStatusIntentHandling, INGetCarLockStatusIntentHandling protocols. I am getting this issue when i try to run it in iPhone 10.
Check, if in TARGETS of your project in Build Phases->Embed App Extensions added your Siri Extension. Maybe if you replace the INStartWorkoutIntent with INGetCarLockStatusIntent, old INStartWorkoutIntent remained there.
My crash fix this.
I was facing a similar issue. Make sure your extension's Deployment Target is set to appropriate iOS version. Creating an extension with the latest Xcode (at the moment 10.1) will set the Deployment Target to 12.1 and thus cause crash when run on iOS 10. So you should change it to your desired minimum.
I am currently trying to test a piece of my code that runs a query on the datastore before putting in a new entity to ensure that duplicates are not created. The code I wrote works fine in the context of the app, but the tests I wrote for that methods are failing. It seems that I cannot access data put into the datastore through queries in the context of the testing package.
One possibility might lie in the output from goapp test which reads: Applying all pending transactions and saving the datastore. This line prints out after both the get and put methods are called (I verified this with log statements).
I tried closing the context and creating a new one for the different operations, but unfortunately that didn't help either. Below is a simple test case that Puts in an object and then runs a query on it. Any help would be appreciated.
type Entity struct {
Value string
}
func TestEntityQuery(t *testing.T) {
c, err := aetest.NewContext(nil)
if err != nil {
t.Fatal(err)
}
defer c.Close()
key := datastore.NewIncompleteKey(c, "Entity", nil)
key, err = datastore.Put(c, key, &Entity{Value: "test"})
if err != nil {
t.Fatal(err)
}
q := datastore.NewQuery("Entity").Filter("Value =", "test")
var entities []Entity
keys, err := q.GetAll(c, &entities)
if err != nil {
t.Fatal(err)
}
if len(keys) == 0 {
t.Error("No keys found in query")
}
if len(entities) == 0 {
t.Error("No entities found in query")
}
}
There is nothing wrong with your test code. The issue lies in the Datastore itself. Most queries in the HR Datastore are not "immediately consistent" but eventually consistent. You can read more about this in the Datastore documentation.
So basically what happens is that you put an entity into the Datastore, and the SDK's Datastore "simulates" the latency that you can observe in production, so if you run a query right after that (which is not an ancestor query), the query result will not include the new entity you just saved.
If you put a few seconds sleep between the datastore.Put() and q.GetAll(), you will see the test passes. Try it. In my test it was enough to sleep just 100ms, and the test always passed. But when writing tests for such cases, use the StronglyConsistentDatastore: true option as can be seen in JonhGB's answer.
You would also see the test pass without sleep if you'd use Ancestor queries because they are strongly consistent.
The way to do this is to force the datastore to be strongly consistent by setting up the context like this:
c, err := aetest.NewContext(&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
Now the datastore won't need any sleep to work, which is faster, and better practice in general.
Update: This only works with the old aetest package which was imported via appengine/aetest. It does not work with the newer aetest package which is imported with google.golang.org/appengine/aetest. App Engine has changed from using an appengine.Context to using a context.Context, and consequently the way that the test package now works is quite different.
To compliment #JohnGB's answer in the latest version of aetest, there are more steps to get a context with strong consistency. First create an instance, then create a request from that instance, which you can use to produce a context.
inst, err := aetest.NewInstance(
&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
ctx := appengine.NewContext(req)
Is there a way to dump the generated sql to the Debug log or something? I'm using it in a winforms solution so the mini-profiler idea won't work for me.
I got the same issue and implemented some code after doing some search but having no ready-to-use stuff. There is a package on nuget MiniProfiler.Integrations I would like to share.
Update V2: it supports to work with other database servers, for MySQL it requires to have MiniProfiler.Integrations.MySql
Below are steps to work with SQL Server:
1.Instantiate the connection
var factory = new SqlServerDbConnectionFactory(_connectionString);
using (var connection = ProfiledDbConnectionFactory.New(factory, CustomDbProfiler.Current))
{
// your code
}
2.After all works done, write all commands to a file if you want
File.WriteAllText("SqlScripts.txt", CustomDbProfiler.Current.ProfilerContext.BuildCommands());
Dapper does not currently have an instrumentation point here. This is perhaps due, as you note, to the fact that we (as the authors) use mini-profiler to handle this. However, if it helps, the core parts of mini-profiler are actually designed to be architecture neutral, and I know of other people using it with winforms, wpf, wcf, etc - which would give you access to the profiling / tracing connection wrapper.
In theory, it would be perfectly possible to add some blanket capture-point, but I'm concerned about two things:
(primarily) security: since dapper doesn't have a concept of a context, it would be really really easy for malign code to attach quietly to sniff all sql traffic that goes via dapper; I really don't like the sound of that (this isn't an issue with the "decorator" approach, as the caller owns the connection, hence the logging context)
(secondary) performance: but... in truth, it is hard to say that a simple delegate-check (which would presumably be null in most cases) would have much impact
Of course, the other thing you could do is: steal the connection wrapper code from mini-profiler, and replace the profiler-context stuff with just: Debug.WriteLine etc.
You should consider using SQL profiler located in the menu of SQL Management Studio → Extras → SQL Server Profiler (no Dapper extensions needed - may work with other RDBMS when they got a SQL profiler tool too).
Then, start a new session.
You'll get something like this for example (you see all parameters and the complete SQL string):
exec sp_executesql N'SELECT * FROM Updates WHERE CAST(Product_ID as VARCHAR(50)) = #appId AND (Blocked IS NULL OR Blocked = 0)
AND (Beta IS NULL OR Beta = 0 OR #includeBeta = 1) AND (LangCode IS NULL OR LangCode IN (SELECT * FROM STRING_SPLIT(#langCode, '','')))',N'#appId nvarchar(4000),#includeBeta bit,#langCode nvarchar(4000)',#appId=N'fea5b0a7-1da6-4394-b8c8-05e7cb979161',#includeBeta=0,#langCode=N'de'
Try Dapper.Logging.
You can get it from NuGet. The way it works is you pass your code that creates your actual database connection into a factory that creates wrapped connections. Whenever a wrapped connection is opened or closed or you run a query against it, it will be logged. You can configure the logging message templates and other settings like whether SQL parameters are saved. Elapsed time is also saved.
In my opinion, the only downside is that the documentation is sparse, but I think that's just because it's a new project (as of this writing). I had to dig through the repo for a bit to understand it and to get it configured to my liking, but now it's working great.
From the documentation:
The tool consists of simple decorators for the DbConnection and
DbCommand which track the execution time and write messages to the
ILogger<T>. The ILogger<T> can be handled by any logging framework
(e.g. Serilog). The result is similar to the default EF Core logging
behavior.
The lib declares a helper method for registering the
IDbConnectionFactory in the IoC container. The connection factory is
SQL Provider agnostic. That's why you have to specify the real factory
method:
services.AddDbConnectionFactory(prv => new SqlConnection(conStr));
After registration, the IDbConnectionFactory can be injected into
classes that need a SQL connection.
private readonly IDbConnectionFactory _connectionFactory;
public GetProductsHandler(IDbConnectionFactory connectionFactory)
{
_connectionFactory = connectionFactory;
}
The IDbConnectionFactory.CreateConnection will return a decorated
version that logs the activity.
using (DbConnection db = _connectionFactory.CreateConnection())
{
//...
}
This is not exhaustive and is essentially a bit of hack, but if you have your SQL and you want to initialize your parameters, it's useful for basic debugging. Set up this extension method, then call it anywhere as desired.
public static class DapperExtensions
{
public static string ArgsAsSql(this DynamicParameters args)
{
if (args is null) throw new ArgumentNullException(nameof(args));
var sb = new StringBuilder();
foreach (var name in args.ParameterNames)
{
var pValue = args.Get<dynamic>(name);
var type = pValue.GetType();
if (type == typeof(DateTime))
sb.AppendFormat("DECLARE #{0} DATETIME ='{1}'\n", name, pValue.ToString("yyyy-MM-dd HH:mm:ss.fff"));
else if (type == typeof(bool))
sb.AppendFormat("DECLARE #{0} BIT = {1}\n", name, (bool)pValue ? 1 : 0);
else if (type == typeof(int))
sb.AppendFormat("DECLARE #{0} INT = {1}\n", name, pValue);
else if (type == typeof(List<int>))
sb.AppendFormat("-- REPLACE #{0} IN SQL: ({1})\n", name, string.Join(",", (List<int>)pValue));
else
sb.AppendFormat("DECLARE #{0} NVARCHAR(MAX) = '{1}'\n", name, pValue.ToString());
}
return sb.ToString();
}
}
You can then just use this in the immediate or watch windows to grab the SQL.
Just to add an update here since I see this question still get's quite a few hits - these days I use either Glimpse (seems it's dead now) or Stackify Prefix which both have sql command trace capabilities.
It's not exactly what I was looking for when I asked the original question but solve the same problem.