Is it possible with Flask Restplus to create a model for a multipart/form-data request so that I can use it to validate the input with #api.expect?
I have this complex data structure for which I've created a api.namespace().model that has to be received together with a file. However when I tried to document the endpoint I noticed that this doesn't seem to be supported by Flask Restplus.
I've tried to find something along the lines of
parser = ns.parser()
parser.add_argument("jsonModel", type=Model, location="form")
parser.add_argument("file", type=FileStorage, location="files")
and
formModel = ns.model("myForm", {"jsonModel": fields.Nested(myModel), "file": fields.File})
But neither methods seem to support this kind of behavior.
Related
I don't see a solution to this using the available api documentation.
It is also not available on the web console.
Is it possible to get the file url using the Watson Discovery Service?
If you need to store the original source/file URL, you can include it as a field within your documents in the Discovery service, then you will be able to query that field back out when needed.
I also struggled with this request but ultimately got it working using Python bindings into Watson Discovery. The online documentation and API reference is very poor; here's what I used to get it working:
(Assume you have a Watson Discovery service and have a created collection):
# Programmatic upload and retrieval of documents and metadata with Watson Discovery
from watson_developer_cloud import DiscoveryV1
import os
import json
discovery = DiscoveryV1(
version='2017-11-07',
iam_apikey='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
url='https://gateway-syd.watsonplatform.net/discovery/api'
)
environments = discovery.list_environments().get_result()
print(json.dumps(environments, indent=2))
This gives you your environment ID. Now append to your code:
collections = discovery.list_collections('{environment-id}').get_result()
print(json.dumps(collections, indent=2))
This will show you the collection ID for uploading documents into programmatically. You should have a document to upload (in my case, an MS Word document), and its accompanying URL from your own source document system. I'll use a trivial fictitious example.
NOTE: the documentation DOES NOT tell you to append , 'rb' to the end of the open statement, but it is required when uploading a Word document, as in my example below. Raw text / HTML documents can be uploaded without the 'rb' parameter.
url = {"source_url":"http://mysite/dis030.docx"}
with open(os.path.join(os.getcwd(), '{path to your document folder with trailing / }', 'dis030.docx'), 'rb') as fileinfo:
add_doc = discovery.add_document('{environment-id}', '{collections-id}', metadata=json.dumps(url), file=fileinfo).get_result()
print(json.dumps(add_doc, indent=2))
print(add_doc["document_id"])
Note the setting up of the metadata as a JSON dictionary, and then encoding it using json.dumps within the parameters. So far I've only wanted to store the original source URL but you could extend this with other parameters as your own use case requires.
This call to Discovery gives you the document ID.
You can now query the collection and extract the metadata using something like a Discovery query:
my_query = discovery.query('{environment-id}', '{collection-id}', natural_language_query="chlorine safety")
print(json.dumps(my_query.result["results"][0]["metadata"], indent=2))
Note - I'm extracting just the stored metadata here from within the overall returned results - if you instead just had:
print(my_query) you'll get the full response from Discovery ... but ... there's a lot to go through to identify just your own custom metadata.
My Problem:
I am attempting to fill a datastore model in GAE that contains an ndb.Structured Property() using a 'POST' request.
This question has been asked recently but not answered (How to “POST” ndb.StructuredProperty?)
I have the following two models:
class Check(EndpointsModel):
this = ndb.StringProperty()
that = ndb.StringProperty()
class CheckMessage(EndpointsModel):
check = ndb.StructuredProperty(Check)
I'm trying to post this data:
{
check:
{
"this":"test",
"that":"test"
}
}
with the following API request:
#CheckMessage.method(name='check_insert',path='check/insert',http_method='POST')
def check_insert(self,request):
print(request)
Upon posting from the client I receive the following error:
AttributeError: 'Check' object has no attribute '_Message__decoded_fields'
The Issue:
From my very high-level understanding of the endpoints-proto-datastore module it seems that when the json is being decoded and saved onto the incoming message (utils.py line 431) it's not checking for structured/localstructured properties and saving their keys as well which is all fine and dandy until FromValue (ndb/model.py line 115) checks for instances of structured properties and attempts to recursively convert the structured property from the protorpc message into a model entity (which needs the _Message__decoded_fields).
Sasxa (see the link above) had found a nifty little workaround to this issue by using an EndpointsAliasProperty converted to a ProtoRPC message class to bypass endpoints-proto-datastore's automatic conversion of the structuredproperty into its associated model entity, however this workaround had some side effects that made what I was trying to do difficult.
The Question:
Does anyone know how to correctly fill a datastore model containing a StructuredProperty using a 'POST' request, and are there any working examples of this available?
I want to parse Swagger data from the JSON I get from {service}/swagger/docs/v1 into dynamically generated .NET class.
The problem I am facing is that different APIs can have different number of parameters and operations. How do I dynamically parse Swagger JSON data for different services?
My end result should be list of all APIs and it's operations in a variable on which I can perform search easily.
Did you ever find an answer for this? Today I wanted to do the same thing, so I used the AutoRest open source project from MSFT, https://github.com/Azure/autorest. While it looks like it's designed for generating client code (code to consume the API documented by your swagger document), at some point on the way producing this code it had to of done exactly what you asked in your question - parse the Swagger file and understand the operations, inputs and outputs the API supports.
In fact we can get at this information - AutoRest publically exposes this information.
So use nuget to install AutoRest. Then add a reference to AutoRest.core and AutoRest.Model.Swagger. So far I've just simply gone for:
using Microsoft.Rest.Generator;
using Microsoft.Rest.Generator.Utilities;
using System.IO;
...
var settings = new Settings();
settings.Modeler = "Swagger";
var mfs = new MemoryFileSystem();
mfs.WriteFile("AutoRest.json", File.ReadAllText("AutoRest.json"));
mfs.WriteFile("Swagger.json", File.ReadAllText("Swagger.json"));
settings.FileSystem = mfs;
var b = System.IO.File.Exists("AutoRest.json");
settings.Input = "Swagger.json";
Modeler modeler = Microsoft.Rest.Generator.Extensibility.ExtensionsLoader.GetModeler(settings);
Microsoft.Rest.Generator.ClientModel.ServiceClient serviceClient;
try
{
serviceClient = modeler.Build();
}
catch (Exception exception)
{
throw new Exception(String.Format("Something nasty hit the fan: {0}", exception.Message));
}
The swagger document you want to parse is called Swagger.json and is in your bin directory. The AutoRest.json file you can grab from their GitHub (https://github.com/Azure/autorest/tree/master/AutoRest/AutoRest.Core.Tests/Resource). I'm not 100% sure how it's used, but it seems it's needed to inform the tool about what is supports. Both JSON files need to be in your bin.
The serviceClient object is what you want. It will contain information about the methods, model types, method groups
Let me know if this works. You can try it with their resource files. I used their ExtensionLoaderTests for reference when I was playing around(https://github.com/Azure/autorest/blob/master/AutoRest/AutoRest.Core.Tests/ExtensionsLoaderTests.cs).
(Also thank you to the Denis, an author of AutoRest)
If still a question you can use Swagger Parser library:
https://github.com/swagger-api/swagger-parser
as simple as:
// parse a swagger description from the petstore and get the result
SwaggerParseResult result = new OpenAPIParser().readLocation("https://petstore3.swagger.io/api/v3/openapi.json", null, null);
I want to build my endpoint, which accept JSON array of below format:
[
{
"test":"Math",
"result":"Pass"
},
{
"test":"Science",
"result":"FirstClass"
}
]
It will be a POST call with the above JSON to my endpoint.
I tried it with servlet too but did not get the required result, and also tried to with list and inserting in a new class and posting to that class. Thanks in advance.
Is that an accurate representation of the JSON object which is being sent over? Because one does not simply send a a POST request with a JSON object of their param object to a cloud endpoint. See here for a thorough guide to Endpoint API interaction from a javascript perspective - notice how the client library exposes an object "gapi" through which calls are made. If you're sending this JSON from an iOS or Android app, there are similar client libraries which can be generated for you by a cloud endpoints build tool.
After much frustration, I resorted to reading the docs more carefully. In that quest, I found an important note in the doc:
https://cloud.google.com/endpoints/docs/frameworks/java/parameter-and-return-types
"Any type except a parameter or injected type is considered an entity type. ... Entity types cannot be annotated with #Named"
With all examples showing named parameters, I was stumped as the docs don't explain further, but then found a solution. It ends up that if you do not have named parameters, everything is just passed in as a LinkedHashMap. Usually, you can do any work you need to with just that data structure, but if you HAVE to have it in JSON, you can convert it. Here are some examples:
#ApiMethod(name = "endpointIterfaceName.createItems", httpMethod = "post", path = "test/items")
public WhateverReturnType createItems(LinkedHashMap<String, Object> itemsMap) {
// Do Stuff with map values
return whateverReturnValue;
}
With this, you need to be sure that you post your data with the Content-Type of json (i.e. Content-Type:application/json; charset=UTF-8). So, for example, when testing, with a jquery ajax call you would need to set dataType to "json" or with Postman, you would select "Raw" then JSON (application/json).
If you really want to convert this to a JSON object in Java because for whatever reason you can not use a hash map, you can do the following in your method:
// Use gson library to convert the map to a string
Gson gson = new Gson();
String mapAsJsonString = gson.toJson(itemsMap);
// create a JSON object from the new string representation
JSONObject obj = new JSONObject(mapAsJsonString);
As a side note, if this is passed as Content-Type:text then the whole body will be in the map as the first key of the map. You could do some inadvisable things here and just get that key and avoid converting the map to a string and then to a json object, but, like I said, that is inadvisable. :)
I recently started using bottle and GAE blobstore and while I can upload the files to the blobstore I cannot seem to find a way to download them from the store.
I followed the examples from the documentation but was only successful on the uploading part. I cannot integrate the example in my app since I'm using a different framework from webapp/2.
How would I go about creating an upload handler and download handler so that I can get the key of the uploaded blob and store it in my data model and use it later in the download handler?
I tried using the BlobInfo.all() to create a query the blobstore but I'm not able to get the key name field value of the entity.
This is my first interaction with the blobstore so I wouldn't mind advice on a better approach to the problem.
For serving a blob I would recommend you to look at the source code of the BlobstoreDownloadHandler. It should be easy to port it to bottle, since there's nothing very specific about the framework.
Here is an example on how to use BlobInfo.all():
for info in blobstore.BlobInfo.all():
self.response.out.write('Name:%s Key: %s Size:%s Creation:%s ContentType:%s<br>' % (info.filename, info.key(), info.size, info.creation, info.content_type))
for downloads you only really need to generate a response that includes the header "X-AppEngine-BlobKey:[your blob_key]" along with everything else you need like a Content-Disposition header if desired. or if it's an image you should probably just use the high performance image serving api, generate a url and redirect to it.... done
for uploads, besides writing a handler for appengine to call once the upload is safely in blobstore (that's in the docs)
You need a way to find the blob info in the incoming request. I have no idea what the request looks like in bottle. The Blobstoreuploadhandler has a get_uploads method and there's really no reason it needs to be an instance method as far as I can tell. So here's an example generic implementation of it that expects a webob request. For bottle you would need to write something similar that is compatible with bottles request object.
def get_uploads(request, field_name=None):
"""Get uploads for this request.
Args:
field_name: Only select uploads that were sent as a specific field.
populate_post: Add the non blob fields to request.POST
Returns:
A list of BlobInfo records corresponding to each upload.
Empty list if there are no blob-info records for field_name.
stolen from the SDK since they only provide a way to get to this
crap through their crappy webapp framework
"""
if not getattr(request, "__uploads", None):
request.__uploads = {}
for key, value in request.params.items():
if isinstance(value, cgi.FieldStorage):
if 'blob-key' in value.type_options:
request.__uploads.setdefault(key, []).append(
blobstore.parse_blob_info(value))
if field_name:
try:
return list(request.__uploads[field_name])
except KeyError:
return []
else:
results = []
for uploads in request.__uploads.itervalues():
results += uploads
return results
For anyone looking for this answer in future, to do this you need bottle (d'oh!) and defnull's multipart module.
Since creating upload URLs is generally simple enough and as per GAE docs, I'll just cover the upload handler.
from bottle import request
from multipart import parse_options_header
from google.appengine.ext.blobstore import BlobInfo
def get_blob_info(field_name):
try:
field = request.files[field_name]
except KeyError:
# Maybe form isn't multipart or file wasn't uploaded, or some such error
return None
blob_data = parse_options_header(field.content_type)[1]
try:
return BlobInfo.get(blob_data['blob-key'])
except KeyError:
# Malformed request? Wrong field name?
return None
Sorry if there are any errors in the code, it's off the top of my head.