Publishing to google pub sub asynchronously through goroutine - google-app-engine

I'm trying to push the message to google pub-sub asynchronously through goroutine but I'm facing below error
panic: not an App Engine context
I'm using mux and have an api handler
n = 1 million
func apihandler(w http.ResponseWriter, r *http.Request) {
go createuniquecodes(n)
return "request running in background"
}
func createuniquecodes(n) {
c := make(chan string)
go createuniquecodes(c, n)
for val := range c {
publishtopubsub(val)
}
}
func createuniquecodes(n) {
for i := 0; i < n; i++ {
uniquecode := some random string
// publish to channel and pubsub
c <- uniquecode
}
close(c)
}
func publishuq(msg string) error {
ctx := context.Background()
client, err := pubsub.NewClient(ctx, projectId)
if err != nil {
log.Fatalf("Could not create pubsub Client: %v", err)
}
t := client.Topic(topicName)
result := t.Publish(ctx, &pubsub.Message{
Data: []byte(msg),
})
id, err := result.Get(ctx)
if err != nil {
return err
}
fmt.Printf("Published a message; msg ID: %v\n", id)
return nil
}
Please note that I need to generate 5 million unique codes,
How will I define a context in go routine since I'm doing everything asynchronously

I assume you're using the App Engine standard (not flexible) environment. Please note that a "request handler (apihandler in your case) has a limited amount of time to generate and return a response to a request, typically around 60 seconds. Once the deadline has been reached, the request handler is interrupted".
You're trying to "break out" of the request when calling go createuniquecodes(n) and then ctx := context.Background() down the line is what panics with not an App Engine context. You could technically use NewContext(req *http.Request) to derive a valid context from the original context, but again, you'd only have 60s before your request times out.
Please have a look at TaskQueues, as they " let applications perform work, called tasks, asynchronously outside of a user request."

Related

Sharing a Postgres (or other DBMS) transaction context between processes

A common pattern in monolithic application design is to delegate business logic to a dedicated service, passing in an open transaction as, for example, a javax.persistence.EntityTransaction instance in Java, or an sql.Transaction in Go.
Go example:
// business.go
type BusinessLogicService interface {
DoSomething(tx *sql.Transaction)
}
type businessLogicService struct {
}
func (s *BusinessLogicService) DoSomething(tx *sql.Transaction) {
tx.ExecuteContext(.....)
}
func NewBusinessLogicService() {
return &businessLogicService{}
}
// server.go
ctx := context.Background()
tx, err := db.BeginTx(ctx)
if err != nil {
log.Fatal(err)
}
bls := business.NewBusinessLogicService()
bls.DoSomething(tx)
tx.Commit()
Could the same effect be achieved in an architecture where each of these components are implemented in a different language/runtime? In such an application, Postgres is responsible for doing the 'bookkeeping' in relation to the DB transaction. It seems to me that it should be possible to pass a similar 'handle' for the transaction to another process to read its state and append operations.
For example the equivalent business logic is provided as a gRPC service with the following definition:
message TransactionInfo {
string transaction_id = 1;
}
message DoSomethingRequest {
TransactionInfo transaction_info = 1;
}
message DoSomethingResponse {
}
service BusinessLogicService {
rpc DoSomething(DoSomethingRequest) returns (DoSomethingResponse)
}
The server process BEGINs the transaction and passes a reference to this BusinessLogicService.
ctx := context.Background()
tx, err := db.BeginTx(ctx)
if err != nil {
log.Fatal(err)
}
conn, err := grpc.Dial(*serverAddr, opts...)
if err != nil {
...
}
defer conn.Close()
bls := pb.NewBusinessLogicClient()
/// SOMEHOW PASS THE TX OBJECT TO THE REMOTE SERVICE
txObj := &pb.TransactionInfo{....???????????.....}
result := bls.DoSomething(txObj)
tx.Commit()
Is this possible with Postgres or another DBMS?

How to set timeout for long running queries in gorm

Is there a way I can set gorm to time out after a configurable period of time when running a long query? I am using mssql. I have looked through the documentation and haven't discovered a way yet.
This code seems to work for me and is pretty clean. Just use transactions I guess.
//Gorm query below
query = query.Where(whereClause)
//Set up Timeout
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var x sql.TxOptions
db := query.BeginTx(ctx, &x)
defer db.Commit()
// Execute the query
if err := db.Find(*results).Error; err != nil {
return err
}
By using WithContext from *gorm.DB you can pass a Timeout Context to Gorm:
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var products []Product
db.WithContext(ctx).Find(&products)
gorm at the moment seems is not implementing any query that accepts as a parameter a context.Context, as e.g. QueryRowContext is doing.
You can create a workaround and use a Context to make "expirable" your query.
type QueryResponse struct {
MyResult *MyResult
Error error
}
func queryHelper(ctx context.Context) <- chan *QueryResponse {
chResult := make(chan *QueryResponse, 1)
go func() {
//your query here
//...
//blah blah check stuff do whatever you want
//err is an error that comes from the query code
if err != nil {
chResult <- &QueryResponse{nil, err}
return
}
chResult <- &QueryResponse{queryResponse, nil}
} ()
return chResult
}
func MyQueryFunction(ctx context.Context) (*MyResult, error) {
select {
case <-ctx.Done():
return nil, fmt.Errorf("context timeout, query out of time")
case res := <-queryHelper(ctx):
return res.MyResult, res.Error
}
}
And then in your upper function, whatever is it you can create a context and pass it to the MyQueryFunction. If the query exceeds the time you have set an error is raised and you should (must) check it.
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
res, err := MyQueryFunction(ctx)
if err != nil {
fmt.Printf("err %v", err)
} else {
fmt.Printf("res %v", res)
}
Of course, it is an example, is not considering a lot of use cases and a proper implementation inside the framework must be preferred.

Go gRPC []bytes encoding/decoding

I'm developing a set of functions (based on KNative Serving + Eventing) and I'm having a lot of problems getting the decoded data on the receiving-end if the stream.
So, I'm implementing a couple of functions that, will use the Google APIs (via google.golang.org/api/slides/v1) to grab a Google Slides Presentation (a struct from the library), encode it as a []byte and send it over the network with protobuf/gRPC.
This appears to be working correctly, however, when I try to decode it back into a Presentation I'm getting an error. Printing out the Decode call returns only EOF.
Here's the proto definition:
syntax = "proto3";
package api;
message ParserRequest {
bytes Presentation = 1;
}
message ParserResponse {
int32 Status = 1;
bytes Document = 2;
}
service ParserService {
// ParsePresentation parse the Google Slides presentation it into the SDR representation
rpc ParsePresentation(ParserRequest) returns (ParserResponse) {}
}
The sending function is similar to:
presentation, err := svc.Presentations.Get(docID).Do()
if err != nil {
log.Fatalf("Unable to retrieve data from document: %+v", err)
}
if presentation != nil {
log.Printf("Calling Parser...")
address := "PORT:IP"
conn, err := grpc.DialContext(ctx, address, grpc.WithInsecure())
if err != nil {
log.Printf("Dial Error! %+v", err)
return nil, fmt.Errorf("could not connect shipping service: %+v", err)
}
defer conn.Close()
log.Printf("Marshalling data...")
// data, err := presentation.MarshalJSON()
// data, err := json.Marshal(presentation)
var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
err = enc.Encode(&presentation)
if err != nil {
log.Println("Encode Error: %+v", err)
log.Fatal("encode error:", err)
}
data := buf.Bytes()
log.Println(data) //[255 211 255 129 3 1 1 12 ...]
cli, err := parser.NewParserServiceClient(conn).ParsePresentation(ctx, &parser.ParserRequest{Presentation: data})
if err != nil {
log.Printf("Cli call Error! %+v", err)
return nil, fmt.Errorf("failed to get parser service: %+v", err)
}
log.Printf("Result: %d", cli.Status)
}
On the other end, I should now decode the data array and "translate" back into a Presentation struct, doing so via:
func (c *parserService) ParsePresentation(ctx context.Context, in *pb.ParserRequest) (*pb.ParserResponse, error) {
log.Printf("ParserService.ParsePresentation was called!")
if in.Presentation == nil {
log.Fatalf("Missing parameter Google Slides document.")
}
sdrDocument, err := gslides_parser.ParsePresentationBytes(in.Presentation)
if err != nil {
log.Fatalf("Unable to parse the Google Slides presentation: %+v", err)
}
presentation, err := json.Marshal(sdrDocument)
return &pb.ParserResponse{Status: 200, Document: presentation}, nil
}
Than, when it gets to the gslides_parser.ParsePresentationBytes(in.Presentation) will then be decoded:
func ParsePresentationBytes(presentationParam []byte) (sdr.Document, error) {
var (
document sdr.Document
presentation slides.Presentation
)
log.Printf("PresentationBytes gob decoder")
log.Println(presentationParam) // Output: [255 211 255 129 3 1 1 12 ...]
// err := json.Unmarshal(presentationParam, &presentation)
buf := bytes.NewBuffer(presentationParam)
log.Printf("New decoder...")
dec := gob.NewDecoder(buf)
log.Printf("Decode...")
log.Println(&presentation) // Output: &{[] [] <nil> <nil> [] {0 map[]} [] []}
err := dec.Decode(&presentation)
log.Printf("Done decoding...")
if err != nil {
// Never gets here
document = sdr.Document{}
(...)
} else {
log.Printf("PresentationBytes Error!")
}
return document, nil
}
So, why can't I decode the information? I don't see anything terribly wrong with this code, but I'm also a golang newb so I may have some error that is eluding me.
Isn't gob the appropriate way of dealing with this? I tried simply marshalling/unmarshalling but that also produces errors.

Use Google Cloud Storage API with Go AppEngine

I've written a test code to list buckets from Google Cloud Storage through the Cloud Storage API, however I'm getting the error below when I run the code:
googleapi: Error 403: Forbidden, forbiddenFinished
I've checked the permissions and the appengine service account has access to the buckets and both the appengine app and cloud storage bucket are on the same project.
This is my sample code:
package src
import (
"fmt"
"net/http"
"golang.org/x/oauth2/google"
storage "google.golang.org/api/storage/v1"
appengine "google.golang.org/appengine"
)
func init() {
http.HandleFunc("/", index)
}
func ListBuckets(r *http.Request, projectID string) ([]*storage.Bucket, error) {
ctx := appengine.NewContext(r)
client, err := google.DefaultClient(ctx, storage.DevstorageReadOnlyScope)
if err != nil {
return nil, err
}
service, err := storage.New(client)
if err != nil {
return nil, err
}
buckets, err := service.Buckets.List(projectID).Do()
if err != nil {
return nil, err
}
return buckets.Items, nil
}
func index(w http.ResponseWriter, r *http.Request) {
r.Header.Set("x-goog-project-id", "theIdProvidedByTheAPI")
bucket, err := ListBuckets(r,"myProject")
if err != nil {
fmt.Fprint(w,err.Error())
}
for i:=range bucket {
fmt.Fprint(w,bucket[i].Name)
}
fmt.Fprint(w,"\n","Finished.")
}
And this is the yaml file:
application: myProject
version: alpha-001
runtime: go
api_version: go1
handlers:
- url: /
script: _go_app
The error message is nor helpful as it does not provide much useful information. I just can't figure out what I'm missing.
I've had issues with the google.DefaultClient method in the past. Here's a more explicit strategy for configuring the storage client object that might be of use for you:
httpClient = &http.Client{
Transport: &oauth2.Transport{
Source: google.AppEngineTokenSource(ctx, scopes...),
Base: &urlfetch.Transport{Context: ctx},
},
}
service, err := storage.New(httpClient)
if err != nil {
return nil, err
}
However, I'm not familiar with the forbiddenFinished error message, which may indicate that the issue lies elsewhere.
Additionally, if you don't have a specific reason for using the autogenerated google.golang.org/api/storage/v1 library, I'd recommend using the higher level interface, which can be found at cloud.google.com/go/storage. Here's the go doc for it:
https://godoc.org/cloud.google.com/go/storage

gob: interface is only registered on Encode but not on Decode

I'm working on an appengine app using the datastore. I'm attempting to gob
encode an interface and store it into the datastore. But when I try to load from
the datastore, I get the error:
gob: name not registered for interface: "main27155.strand"
The peculiar thing is that the load() method starts working after having
called the save() method. It no longer returns an error, and everything saved
in the datastore is loaded as expected. But when I restart the intance, the
load() method stops working again.
The load and save methods I mention refer to the methods defined by the
datastore.PropertyLoadSaver interface
From the looks of it, it seems like a problem with registering the
type/interfaces with gob, but I have exactly the same gob.Register() calls in
both the load() and save() methods.
I even tried removing the gob.Register() calls from both load and save methods
and adding it to init(). The exact same behavior was observed.
How can I load my datastore on a cold start?
type bio struct {¬
Id string¬
Hp int¬
godie chan bool //should be buffered¬
dam chan int¬
Genetics dna¬
}¬
type dna interface {
decode() mRNA
Get(int) trait
Set(int, trait)
Duplicate() dna
Len() int
}
type trait interface {
mutate() trait
}
// implements dna{}
type strand []trait
// implements trait{}
type tdecoration string
type color struct {
None bool // If true, colors are not shown in theme
Bg bool // If true, color is a background color
R int // 0-255
G int
B int
}
.
func start(w http.ResponseWriter, r *http.Request) error {
c := appengine.NewContext(r)
var bs []bio
if _, err := datastore.NewQuery("bio").GetAll(c, &bs); err != nil {
log.Println("bs is len: ", len(bs))
return err
}
...
return nil
}
func stop(w http.ResponseWriter, r *http.Request) error {
c := appengine.NewContext(r)
log.Println("Saving top 20 colors")
var k []*datastore.Key
var bs []*bio
stat := getStats()
for i, b := range stat.Leaderboard {
k = append(k, datastore.NewKey(c, "bio", b.Id, 0, nil))
bv := b
bs = append(bs, &bv)
// At most 20 bios survive across reboots
if i > 178 {
break
}
}
// Assemble slice of keys for deletion
dk, err := datastore.NewQuery("bio").KeysOnly().GetAll(c, nil)
if err != nil {
return errors.New(fmt.Sprintf("Query error: %s", err.Error()))
}
fn := func(c appengine.Context) error {
// Delete all old entries
err := datastore.DeleteMulti(c, dk)
if err != nil {
return errors.New(fmt.Sprintf("Delete error: %s", err.Error()))
}
// save the elite in the datastore
_, err = datastore.PutMulti(c, k, bs)
if err != nil {
return err
}
return nil
}
return datastore.RunInTransaction(c, fn, &datastore.TransactionOptions{XG: true})
}
// satisfy datastore PropertyLoadSaver interface ===============================
func (b *bio) Load(c <-chan datastore.Property) error {
gob.Register(&color{})
gob.Register(new(tdecoration))
var str strand
gob.Register(str)
tmp := struct {
Id string
Hp int
Gengob []byte
}{}
if err := datastore.LoadStruct(&tmp, c); err != nil {
return err
}
b.Id = tmp.Id
b.Hp = tmp.Hp
return gob.NewDecoder(strings.NewReader(string(tmp.Gengob))).Decode(&(b.Genetics))
}
func (b *bio) Save(c chan<- datastore.Property) error {
defer close(c)
gob.Register(&color{})
gob.Register(new(tdecoration))
var str strand
gob.Register(str)
var buf bytes.Buffer
gen := b.Genetics
if err := gob.NewEncoder(&buf).Encode(&gen); err != nil {
log.Println(err)
return err
}
dp := []datastore.Property{
{Name: "Id", Value: b.Id},
{Name: "Hp", Value: int64(b.Hp)},
{Name: "Gengob", Value: buf.Bytes(), NoIndex: true},
}
for _, p := range dp {
c <- p
}
return nil
}
Additional info: This behavior was not present before I stuffed the datastore
calls in stop() into datastore.RunInTransaction()
Register all types an in init() functions using RegisterName(). Delete all existing data from the store and you should be good to go.
App Engine generates a mangled name for the main package every time the application is built. The name generated by Register() includes this mangled package name. Any gobs encoded with the mangled name will only be readable using the same build of the app. If you cause the application to be rebuilt by modifying the code, then the app will not be able to decode gobs stored previously.

Resources