I have an app that tries to store small images (less than 50kB) in memcache, but each call to memcache.Set() results in an error memcache: server error.
I'm on the shared memcache class, so I understand there is no service guarantee, but currently I have no service at all.
Is it a temporary outage? Am I just unlucky?
Here is the piece of code creating an item and calling memcache. ctx is the appengine context for the request. memkey is my key (a string). img_data is a string with my data.
This code works well in the local dev environment.
cache_item = &memcache.Item{
Key: memkey,
Value: bytes.NewBufferString(img_data).Bytes(),
}
err = memcache.Set(ctx, cache_item)
if err != nil {
ctx.Infof("Could not store image in memcache: %s", err)
}
If it's still happening, file a bug, but I suspect it was just a transient problem.
Incidentally, your Value initialiser is unnecessarily complex. This would work the same:
cache_item = &memcache.Item{
Key: memkey,
Value: []byte(img_data),
}
Related
In all tutorials on using datastore with appengine standard go environment suggest that I should obtain appengine context from the http request and use this context when manipulating the datastore. Golang appengine datastore introduction
This is unfortunate because it makes it hard for me to use dependency injection of data services into my controllers (handlers). I can't create one instance of aservice while initializing the controller, but I have to pass requests/contexts to every data related operation.
func createHandler(dbService db.DBService) http.HandleFunc
return func (req http.Request, resp http.Response) {
entity := bindEntity(req)
/*
I undertand that I can pass whole request to the dbService
to at least remove dependency on datastore from this handler
but it doesn't seem right to pass request everywhere
*/
ctx := appengine.NewContext(req)
dbService.StoreEntity(ctx, entity)
}
}
Is there a way of getting the appengine context from somewhere else except for request? Or is there some common known design pattern how to separate the responsibility of handling request and manipulating data in app engine?
dbService := db.CreateServcie(somehowObtainedContext)
func createHandler(dbService db.DBService) http.HandleFunc
return func (req http.Request, resp http.Response) {
entity := bindEntity(req)
dbService.StoreEntity(entity)
}
}
This would make the code clearer and testing easier.
For example, suppose you embed the following Javascript code in Vue.js or React.js.
var elasticsearch = require ('elasticsearch');
var esclient = new elasticsearch.Client ({
host: 'Elasticsearch host name of Elascticsearch Cloud's(URL?')
});
esclient.search ({
index: 'your index',
body: {
query: {
match: {message: 'search keyword'}
},
aggs: {
your_states: {
terms: {
field: 'your field',
size: 10
}
}
}
}
}
).then (function (response) {
var hits = response.hits.hits;
}
);
When aiming at a search engine of an application like stackoverflow,
if only GET from the public is OK by using the ROLE setting of the cloud of Elasticseach,
Even though I did not prepare an API server, I thought that the same thing could be realized with the above client side code,
Is it a security problem? (Such as whether it is dangerous for the host name to fall on the client side)
If there is no problem, the search engine response will be faster and the cost of implementation will be reduced,
I wondered why many people would not do it. (Because sample code like this can not be seen on the net much)
Thank you.
It is NOT a good idea.
If any client with a bit of programming knowledge finds our your ElasticSearch IP address, you are screwed, he could basically delete all the data without you even noticing.
I have no understanding about XPack Security, but if you are not using that you are absolutely forced to hide ES behind an API.
Then you also have to secure you ES domain to allow access only from the API server and block the rest of the world.
I have the following piece of code with Restlet in Google AppEngine from an Android client.
ClientResource clientResource = new ClientResource(RESTLET_TEST_URL);
ProductResource productResource = clientResource.wrap(ProductResource.class);
productResource.store(mProduct);
Status status = clientResource.getResponse().getStatus();
Toast.makeText(this, "Status: "+ status.getDescription(), Toast.LENGTH_SHORT).show();
clientResource.release();
The .store() method is analogous to a PUT request. The weird thing is, this works fine when I connect to the development server but on the actual AppEngine site, nothing happens. I just get Status: OK indicating that the request went through.
I can't troubleshoot cause I can only do that in the Dev Server and that is working fine.
Any ideas on what the problem may be or how to approach this ?
For reference, the code at the server end is :
if (product != null ) {
if (new DataStore().putToDataStore(product) ) {
log.warning("Product written to datastore");
} else {
log.warning("Product not found in datastore");
}
}
This is just a simple write to the datastore using Objectify.
Turns out this is a known issue. See here
The solution is to use clientResource.setEntityBuffering(true);. However, please note that this method is only available in the Release Candidate for Android Client and not in the stable release.
I want to know the relation between velocity and IS. If a request is satisfied by velocity, then will it going to use worker process. Or what happen I’m confused. ?
Also I want to store some data like country, state and city for auto suggest in velocity. This database could be on 3 gb. Now how velocity will work. And how IS will work. Is this going to effect IS. Basically my requirements is that I want to save all country, state and city data in velocity and don’t want to hit database and don’t want to make IS busy. What is the solution?
Please help
Velocity was the codename for Microsoft's AppFabric distributed caching technology. Very similar to memcache, it is used for caching objects across multiple computers.
This has no real bearing on how IIS processes requests. All requests are satisfied by IIS, AppFabric is a mechanism for storing data, not processing requests.
In answer to your second question; You can use AppFabric is a first-call check for data. If the data does not exist in the cache, call the database to populate the cache, and then return the data.
var factory = DataCacheFactory();
var cache = factory.GetCache("AutoSuggest");
List<Region> regions = cache.Get("Regions") as List<Region>;
if (regions == null) {
regions = // Get regions from database.
cache.Add("Regions", regions);
}
return regions;
Checking the cache first enables the app to get a faster response, as the database is only hit on the first instance (ideally), and the result data is pushed back into the cache.
You could wrap this up a bit more:
public T Get<T>(string cacheName, string keyName, Func<T> itemFactory)
where T : class
{
var cache = dataFactory.GetCache(cacheName);
T value = cache.Get(keyName) as T;
if (value == null) {
value = itemFactory();
cache.Add(keyName, value);
}
return value;
}
That way you can change your lookup calls to something similar to:
var regions = Get<List<Region>>("AutoSuggest", "Regions", () => GetRegions());
I'm writing an app on the Google App Engine, and I have been able to write code that runs locally and puts data into the local datastore. However, when I deploy the code to the google server, no data is put into the datastore. My primary indicators that nothing is getting put into the datastore are, one, when I go to the Developer Console, I'm told there are no entries, and two, when I run the page that gets data, nothing is returned.
Do I need to define new Kinds in the app.yaml or developer console before they will work on the production datastore?
Here is my put and get methods from the data store. Again, works in the testing environment and does not work in production, everything compiles just fine.
package tweetdata
import (
"net/url"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/log"
"github.com/ChimeraCoder/anaconda"
)
const linkTweetKind string = "LinkTweet"
const tweetKey string = "Tweets"
const tweetKeyID string = "default_tweetstore"
//LinkTweet contains the address extracted from a tweet and the original tweet
type LinkTweet struct {
Address *url.URL
Tweet anaconda.Tweet
}
//StoreTweet is a struct used for storing a tweet in the datastore
type StoreTweet struct {
Address string
Text string
TweetID int64
CreatedTime time.Time
Retweets int
Favorites int
}
//TweetScore is a struct that shows the relative score of an address based on
// it's populatrity
type TweetScore struct {
Address *url.URL
score int
lastActive time.Time
}
//WriteLinkTweet writes a given Tweet to the datastore
func WriteLinkTweet(tweet LinkTweet, c context.Context) error {
log.Infof(c, "Putting Tweet into datastore: %v", tweet.Tweet.Id)
key := datastore.NewIncompleteKey(c, linkTweetKind, getTweetKey(c))
created, _ := tweet.Tweet.CreatedAtTime()
store := &StoreTweet{Address: tweet.Address.String(),
Text: tweet.Tweet.Text,
TweetID: tweet.Tweet.Id,
CreatedTime: created,
Retweets: tweet.Tweet.RetweetCount,
Favorites: tweet.Tweet.FavoriteCount,
}
err := datastore.RunInTransaction(c, func(c context.Context) error {
_, err := datastore.Put(c, key, store)
log.Errorf(c, "Failed to write LinkTweet to datastore. %v", err.Error())
return err
}, nil)
return err
}
//GetAllNewTweets queries the datastore and gets all tweets created since the last
// time given
func GetAllNewTweets(since time.Time, c context.Context) []StoreTweet {
q := datastore.NewQuery(linkTweetKind).Ancestor(getTweetKey(c)).Filter("CreatedTime >=", since)
out := make([]StoreTweet, 0, 15)
q.GetAll(c, &out)
return out
}
// guestbookKey returns the key used for all guestbook entries.
func getTweetKey(c context.Context) *datastore.Key {
// The string "default_guestbook" here could be varied to have multiple guestbooks.
return datastore.NewKey(c, tweetKey, tweetKeyID, 0, nil)
}
You're seeing the (often initially confusing) effect of "eventual consistency". Entities that are written outside of a transaction take a while to be visible to queries.
https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ explains the problem. https://cloud.google.com/appengine/docs/go/datastore/transactions is specific to go.
Are you using datastore:,noindex on any fields in the StoreTweet struct? In order to run any query against any fields, the datastore must first index the fields you'll be querying. So not indexing those fields will cause the query to return nil without any errors. Does your dev server automatically create an index.yaml file in your project after running your queries for the first time? The production datastore uses this file to decide which fields should be indexed, all fields not indexed won't be able to be queried. From the information you've provided, this is all I can think of that would be causing your queries to return empty.
In order to debug this further, I would browse your production datastore and make sure your data is structured properly and the values are set to what you are expecting them to be.