My Problem:
I am attempting to fill a datastore model in GAE that contains an ndb.Structured Property() using a 'POST' request.
This question has been asked recently but not answered (How to “POST” ndb.StructuredProperty?)
I have the following two models:
class Check(EndpointsModel):
this = ndb.StringProperty()
that = ndb.StringProperty()
class CheckMessage(EndpointsModel):
check = ndb.StructuredProperty(Check)
I'm trying to post this data:
{
check:
{
"this":"test",
"that":"test"
}
}
with the following API request:
#CheckMessage.method(name='check_insert',path='check/insert',http_method='POST')
def check_insert(self,request):
print(request)
Upon posting from the client I receive the following error:
AttributeError: 'Check' object has no attribute '_Message__decoded_fields'
The Issue:
From my very high-level understanding of the endpoints-proto-datastore module it seems that when the json is being decoded and saved onto the incoming message (utils.py line 431) it's not checking for structured/localstructured properties and saving their keys as well which is all fine and dandy until FromValue (ndb/model.py line 115) checks for instances of structured properties and attempts to recursively convert the structured property from the protorpc message into a model entity (which needs the _Message__decoded_fields).
Sasxa (see the link above) had found a nifty little workaround to this issue by using an EndpointsAliasProperty converted to a ProtoRPC message class to bypass endpoints-proto-datastore's automatic conversion of the structuredproperty into its associated model entity, however this workaround had some side effects that made what I was trying to do difficult.
The Question:
Does anyone know how to correctly fill a datastore model containing a StructuredProperty using a 'POST' request, and are there any working examples of this available?
Related
Is it possible with Flask Restplus to create a model for a multipart/form-data request so that I can use it to validate the input with #api.expect?
I have this complex data structure for which I've created a api.namespace().model that has to be received together with a file. However when I tried to document the endpoint I noticed that this doesn't seem to be supported by Flask Restplus.
I've tried to find something along the lines of
parser = ns.parser()
parser.add_argument("jsonModel", type=Model, location="form")
parser.add_argument("file", type=FileStorage, location="files")
and
formModel = ns.model("myForm", {"jsonModel": fields.Nested(myModel), "file": fields.File})
But neither methods seem to support this kind of behavior.
I've got a Django app with Tastypie, and mainly BackBone client side. One of my models has a few ImageFields. Here is a similar setup to help me explain the issue.
settings.py
MEDIA_URL = "/media/"
models.py
class Foo(models.model):
bar = models.ImageField()
baz = models.CharField()
api.py
class FooResource(ModelResource):
class Meta:
queryset=models.Foo.objects.all()
resource_name = "foo"
authorization = Authorization()
When I make a GET request to the API, it appends the MEDIA_URL to the file names to return the URI where a bar can be accessed. However, when I change the value of baz on a row, and then make a PUT request with that, it also changes the value for a bar to the URI. This means that the next time I GET the row, it appends the MEDIA_URL again, breaking the system and appending it for each successive GET and PUT. I end up with values for bar in the DB that look like.
/media/media/media/bar.jpg
I think I should fix this by overriding a method in my ModelResource, so that when there is a PUT request, it recognizes that it's getting either a URI or a real file, and alters its behavior in some way.
Is this the correct fix? Could you provide some implementation details of a fix?
I found the answer. Tastypie is well designed, similarly to Django. Unfortunately I was not familiar with the terminology so when I read the docs I didn't understand. You can easily modify behavior of the API at many levels. Here is my new API definition, which fixed the issue.
api.py
class FooResource(ModelResource):
class Meta:
queryset=models.Foo.objects.all()
resource_name = "foo"
authorization = Authorization()
def hydrate_bar(bundle):
bundle["bar"] = bundle["bar"].strip(MEDIA_URL)
return bundle
I should add that this only works for me because I exclusively POST my image files individually with a post_detail method which doesn't call this method. If I was to POST or PUT image files as part of the entire row, I expect this might raise an error if that isn't considered.
I want to build my endpoint, which accept JSON array of below format:
[
{
"test":"Math",
"result":"Pass"
},
{
"test":"Science",
"result":"FirstClass"
}
]
It will be a POST call with the above JSON to my endpoint.
I tried it with servlet too but did not get the required result, and also tried to with list and inserting in a new class and posting to that class. Thanks in advance.
Is that an accurate representation of the JSON object which is being sent over? Because one does not simply send a a POST request with a JSON object of their param object to a cloud endpoint. See here for a thorough guide to Endpoint API interaction from a javascript perspective - notice how the client library exposes an object "gapi" through which calls are made. If you're sending this JSON from an iOS or Android app, there are similar client libraries which can be generated for you by a cloud endpoints build tool.
After much frustration, I resorted to reading the docs more carefully. In that quest, I found an important note in the doc:
https://cloud.google.com/endpoints/docs/frameworks/java/parameter-and-return-types
"Any type except a parameter or injected type is considered an entity type. ... Entity types cannot be annotated with #Named"
With all examples showing named parameters, I was stumped as the docs don't explain further, but then found a solution. It ends up that if you do not have named parameters, everything is just passed in as a LinkedHashMap. Usually, you can do any work you need to with just that data structure, but if you HAVE to have it in JSON, you can convert it. Here are some examples:
#ApiMethod(name = "endpointIterfaceName.createItems", httpMethod = "post", path = "test/items")
public WhateverReturnType createItems(LinkedHashMap<String, Object> itemsMap) {
// Do Stuff with map values
return whateverReturnValue;
}
With this, you need to be sure that you post your data with the Content-Type of json (i.e. Content-Type:application/json; charset=UTF-8). So, for example, when testing, with a jquery ajax call you would need to set dataType to "json" or with Postman, you would select "Raw" then JSON (application/json).
If you really want to convert this to a JSON object in Java because for whatever reason you can not use a hash map, you can do the following in your method:
// Use gson library to convert the map to a string
Gson gson = new Gson();
String mapAsJsonString = gson.toJson(itemsMap);
// create a JSON object from the new string representation
JSONObject obj = new JSONObject(mapAsJsonString);
As a side note, if this is passed as Content-Type:text then the whole body will be in the map as the first key of the map. You could do some inadvisable things here and just get that key and avoid converting the map to a string and then to a json object, but, like I said, that is inadvisable. :)
I recently started using bottle and GAE blobstore and while I can upload the files to the blobstore I cannot seem to find a way to download them from the store.
I followed the examples from the documentation but was only successful on the uploading part. I cannot integrate the example in my app since I'm using a different framework from webapp/2.
How would I go about creating an upload handler and download handler so that I can get the key of the uploaded blob and store it in my data model and use it later in the download handler?
I tried using the BlobInfo.all() to create a query the blobstore but I'm not able to get the key name field value of the entity.
This is my first interaction with the blobstore so I wouldn't mind advice on a better approach to the problem.
For serving a blob I would recommend you to look at the source code of the BlobstoreDownloadHandler. It should be easy to port it to bottle, since there's nothing very specific about the framework.
Here is an example on how to use BlobInfo.all():
for info in blobstore.BlobInfo.all():
self.response.out.write('Name:%s Key: %s Size:%s Creation:%s ContentType:%s<br>' % (info.filename, info.key(), info.size, info.creation, info.content_type))
for downloads you only really need to generate a response that includes the header "X-AppEngine-BlobKey:[your blob_key]" along with everything else you need like a Content-Disposition header if desired. or if it's an image you should probably just use the high performance image serving api, generate a url and redirect to it.... done
for uploads, besides writing a handler for appengine to call once the upload is safely in blobstore (that's in the docs)
You need a way to find the blob info in the incoming request. I have no idea what the request looks like in bottle. The Blobstoreuploadhandler has a get_uploads method and there's really no reason it needs to be an instance method as far as I can tell. So here's an example generic implementation of it that expects a webob request. For bottle you would need to write something similar that is compatible with bottles request object.
def get_uploads(request, field_name=None):
"""Get uploads for this request.
Args:
field_name: Only select uploads that were sent as a specific field.
populate_post: Add the non blob fields to request.POST
Returns:
A list of BlobInfo records corresponding to each upload.
Empty list if there are no blob-info records for field_name.
stolen from the SDK since they only provide a way to get to this
crap through their crappy webapp framework
"""
if not getattr(request, "__uploads", None):
request.__uploads = {}
for key, value in request.params.items():
if isinstance(value, cgi.FieldStorage):
if 'blob-key' in value.type_options:
request.__uploads.setdefault(key, []).append(
blobstore.parse_blob_info(value))
if field_name:
try:
return list(request.__uploads[field_name])
except KeyError:
return []
else:
results = []
for uploads in request.__uploads.itervalues():
results += uploads
return results
For anyone looking for this answer in future, to do this you need bottle (d'oh!) and defnull's multipart module.
Since creating upload URLs is generally simple enough and as per GAE docs, I'll just cover the upload handler.
from bottle import request
from multipart import parse_options_header
from google.appengine.ext.blobstore import BlobInfo
def get_blob_info(field_name):
try:
field = request.files[field_name]
except KeyError:
# Maybe form isn't multipart or file wasn't uploaded, or some such error
return None
blob_data = parse_options_header(field.content_type)[1]
try:
return BlobInfo.get(blob_data['blob-key'])
except KeyError:
# Malformed request? Wrong field name?
return None
Sorry if there are any errors in the code, it's off the top of my head.
I'm having trouble setting a parameter when kicking off a mapreduce via start_map so I can access it in done_callback. Numerous things I've read imply that it's possible, but somehow I've not got the earth-moon-stars properly aligned. Ultimately, what I'm trying to accomplish is to delete the temporary blob I created for the mapreduce job.
Here's how I kick it off:
mrID = control.start_map(
"Find friends",
"findfriendshandler.findFriendHandler",
"mapreduce.input_readers.BlobstoreLineInputReader",
{"blob_keys": blobKey},
shard_count=7,
mapreduce_parameters={'done_callback': '/fnfrdone','blobKey': blobKey})
In done_callback, the context object isn't available:
class FindFriendsDoneHandler(webapp.RequestHandler):
def post(self):
ctx = context.get()
if ctx is not None:
params = ctx.mapreduce_spec.mapper.params
try:
blobKey = params['blobKey']
logging.info(['BLOBKEY ' + blobKey])
except KeyError:
logging.info('blobKey key not found in params')
else:
logging.info('context.get did not work') #THIS IS WHAT GETS OUTPUT
Thanks!
EDIT: It seems like there may be more than one MR library, so I wanted to include my various imports:
from mapreduce import control
from mapreduce import operation as op
from mapreduce import context
from mapreduce import model
Below is the code I used in my done_callback handler to retrieve my blobKey user parameter:
class FindFriendsDoneHandler(webapp.RequestHandler):
mrID = self.request.headers['Mapreduce-Id']
try:
mapreduceState = MapreduceState.get_by_key_name(mrID)
mrSpec = mapreduceState.mapreduce_spec
jsonSpec = mrSpec.to_json()
jsonParams = jsonSpec['params']
blobKey = jsonParams['blobKey']
blobInfo = BlobInfo.get(blobKey)
blobInfo.delete()
logging.info('Temp blob deleted successfully for mapreduce:' + mrID)
except:
logging.warning('Unable to delete temp blob for mapreduce:' + mrID)
This uses the mapreduce ID passed into done callback via the header to retrieve the mapreduce state model object from the mapreduce state table. The model stores any user params sent via start_map in a mapreduce_spec property which is in json format.
Note that MR, itself, actually stores the blob_key elsewhere in mapreduce_spec.
Thanks again to #Nick for pointing me to the model.py source file.
I'd love to hear if there's a simpler way to get at MR user params...
Context is only available to mappers/reducers - it's largely concerned with things that don't make sense outside the context of one. As you can see from the source, however, the "Mapreduce-Id" header is set, from which you can get the ID of the mapreduce job.
You shouldn't have to do your own cleanup, though - mapreduce has a handler that does it for you.