Python Google appengine 'Attachment' object does not support indexing - google-app-engine

Since sometime after 3pm EST on January 9th I am getting
TypeError: 'Attachment' object does not support indexing errors when trying to access the data portion of an email attachment:
attach = mail_message.attachments.pop()
encodedAttachment = attach[1]
The format of the emails I am processing has not changed in that time, and this code worked flawlessly up until then

The latest version (1.8.9) has introduced an Attachment class that is returned now instead of the (filename content) tuple that was returned previously. The class does implement __iter__, so unpacking works exactly the same:
filename, content = attachment
But it doesn't implement __getitem__, so accessing via index as you're doing will cause the error you're seeing. It's possible that creating an issue will get the code changed to be completely backwards-compatible, but the practical thing would be to change your code.

Related

Parsing Swagger JSON data and storing it in .net class

I want to parse Swagger data from the JSON I get from {service}/swagger/docs/v1 into dynamically generated .NET class.
The problem I am facing is that different APIs can have different number of parameters and operations. How do I dynamically parse Swagger JSON data for different services?
My end result should be list of all APIs and it's operations in a variable on which I can perform search easily.
Did you ever find an answer for this? Today I wanted to do the same thing, so I used the AutoRest open source project from MSFT, https://github.com/Azure/autorest. While it looks like it's designed for generating client code (code to consume the API documented by your swagger document), at some point on the way producing this code it had to of done exactly what you asked in your question - parse the Swagger file and understand the operations, inputs and outputs the API supports.
In fact we can get at this information - AutoRest publically exposes this information.
So use nuget to install AutoRest. Then add a reference to AutoRest.core and AutoRest.Model.Swagger. So far I've just simply gone for:
using Microsoft.Rest.Generator;
using Microsoft.Rest.Generator.Utilities;
using System.IO;
...
var settings = new Settings();
settings.Modeler = "Swagger";
var mfs = new MemoryFileSystem();
mfs.WriteFile("AutoRest.json", File.ReadAllText("AutoRest.json"));
mfs.WriteFile("Swagger.json", File.ReadAllText("Swagger.json"));
settings.FileSystem = mfs;
var b = System.IO.File.Exists("AutoRest.json");
settings.Input = "Swagger.json";
Modeler modeler = Microsoft.Rest.Generator.Extensibility.ExtensionsLoader.GetModeler(settings);
Microsoft.Rest.Generator.ClientModel.ServiceClient serviceClient;
try
{
serviceClient = modeler.Build();
}
catch (Exception exception)
{
throw new Exception(String.Format("Something nasty hit the fan: {0}", exception.Message));
}
The swagger document you want to parse is called Swagger.json and is in your bin directory. The AutoRest.json file you can grab from their GitHub (https://github.com/Azure/autorest/tree/master/AutoRest/AutoRest.Core.Tests/Resource). I'm not 100% sure how it's used, but it seems it's needed to inform the tool about what is supports. Both JSON files need to be in your bin.
The serviceClient object is what you want. It will contain information about the methods, model types, method groups
Let me know if this works. You can try it with their resource files. I used their ExtensionLoaderTests for reference when I was playing around(https://github.com/Azure/autorest/blob/master/AutoRest/AutoRest.Core.Tests/ExtensionsLoaderTests.cs).
(Also thank you to the Denis, an author of AutoRest)
If still a question you can use Swagger Parser library:
https://github.com/swagger-api/swagger-parser
as simple as:
// parse a swagger description from the petstore and get the result
SwaggerParseResult result = new OpenAPIParser().readLocation("https://petstore3.swagger.io/api/v3/openapi.json", null, null);

How to view JSON logs of a managed VM in the Log Viewer?

I'm trying to get JSON formatted logs on a Compute Engine VM instance to appear in the Log Viewer of the Google Developer Console. According to this documentation it should be possible to do so:
Applications using App Engine Managed VMs should write custom log
files to the VM's log directory at /var/log/app_engine/custom_logs.
These files are automatically collected and made available in the Logs
Viewer.
Custom log files must have the suffix .log or .log.json. If the suffix
is .log.json, the logs must be in JSON format with one JSON object per
line. If the suffix is .log, log entries are treated as plain text.
This doesn't seem to be working for me: logs ending with .log are visible in the Log Viewer, but displayed as plain text. Logs ending with .log.json aren't visible at all.
It also contradicts another recent article that states that file names must end in .log and its contents are treated as plain text.
As far as I can tell Google uses fluentd to index the log files into the Log Viewer. In the GitHub repository I cannot find any evidence that .log.json files are being indexed.
Does anyone know how to get this working? Or is the documentation out-of-date and has this feature been removed for some reason?
Here is one way to generate JSON logs for the Managed VMs logviewer:
The desired JSON format
The goal is to create a single line JSON object for each log line containing:
{
"message": "Error occurred!.",
"severity": "ERROR",
"timestamp": {
"seconds": 1437712034000,
"nanos": 905
}
}
(information sourced from Google: https://code.google.com/p/googleappengine/issues/detail?id=11678#c5)
Using python-json-logger
See: https://github.com/madzak/python-json-logger
def get_timestamp_dict(when=None):
"""Converts a datetime.datetime to integer milliseconds since the epoch.
Requires special handling to preserve microseconds.
Args:
when:
A datetime.datetime instance. If None, the timestamp for 'now'
will be used.
Returns:
Integer time since the epoch in milliseconds. If the supplied 'when' is
None, the return value will be None.
"""
if when is None:
when = datetime.datetime.utcnow()
ms_since_epoch = float(time.mktime(when.utctimetuple()) * 1000.0)
return {
'seconds': int(ms_since_epoch),
'nanos': int(when.microsecond / 1000.0),
}
def setup_json_logger(suffix=''):
try:
from pythonjsonlogger import jsonlogger
class GoogleJsonFormatter(jsonlogger.JsonFormatter):
FORMAT_STRING = "{message}"
def add_fields(self, log_record, record, message_dict):
super(GoogleJsonFormatter, self).add_fields(log_record,
record,
message_dict)
log_record['severity'] = record.levelname
log_record['timestamp'] = get_timestamp_dict()
log_record['message'] = self.FORMAT_STRING.format(
message=record.message,
filename=record.filename,
)
formatter = GoogleJsonFormatter()
log_path = '/var/log/app_engine/custom_logs/worker'+suffix+'.log.json'
make_sure_path_exists(log_path)
file_handler = logging.FileHandler(log_path)
file_handler.setFormatter(formatter)
logging.getLogger().addHandler(file_handler)
except OSError:
logging.warn("Custom log path not found for production logging")
except ImportError:
logging.warn("JSON Formatting not available")
To use, simply call setup_json_logger - you may also want to change the name of worker for your log.
I am currently working on a NodeJS app running on a managed VM and I am also trying to get my logs to be printed on the Google Developper Console. I created my log files in the ‘/var/log/app_engine’ directory as described in the documentation. Unfortunately this doesn’t seem to be working for me, even for the ‘.log’ files.
Could you describe where your logs are created ? Also, is your managed VM configured as "Managed by Google" or "Managed by User" ? Thanks!

go-endpoint Invalid date format and method not found

Hy, I have some problems with the Go endpoints and Dart client library.
I use the Go library https://github.com/crhym3/go-endpoints and the dart generator https://github.com/dart-lang/discovery_api_dart_client_generator
The easy examples works fine. But they show never how to use time.Time.
In my project, I have a struct with a field:
Created time.Time `json:"created"`
The output in the explorer looks like this:
"created": "2014-12-08T20:42:54.299127593Z",
When i use it in the dart client library, I get the error
FormatException: Invalid date format 2014-12-08T20:53:56.346129718Z
Should I really format every time fields in the go app (Format Timestamp in outgoing JSON in Golang?)?
My research come to that the dart accept something:
t.Format(time.RFC3339) >> 2014-12-08T20:53:56Z
Second problem, if comment out the Created field or leave it blank. I get a other error:
The null object does not have a method 'map'.
NoSuchMethodError: method not found: 'map' Receiver: null Arguments:
[Closure: (dynamic) => dynamic]
But I can't figure it out which object is null. I'm not sure if I'm using the Dart client correct
import 'package:http/browser_client.dart' as http;
...
var nameValue = querySelector('#name').value;
var json = {'name':nameValue};
LaylistApi api = new LaylistApi(new http.BrowserClient());
api.create(new NewLayListReq.fromJson(json)).then((LayList l) {
print(l);
}).catchError((e) {
querySelector('#err-message').innerHtml=e.toString();
});
Does anyone know of a larger project on github with Go endpoint and Dart?
Thanks for any advice
UPDATE[2014-12-11]:
I fixed the
NoSuchMethodError
with the correct discovery url https://constant-wonder-789.appspot.com/_ah/api/discovery/v1/apis/greeting/v1/rest
The problem with the time FormatExcetion still open, but I'm one step further. If i create a new item, it doesn' work. But if I load the items from the datastore and send it back, this works.
I guess this can be fixed with implementing Marshaler interface, thanks Alex. I will update my source soon.
See my example:
http://constant-wonder-789.appspot.com/
The full source code:
https://github.com/cloosli/greeting-example

Deploying from a development org to a new unrelated org without a package

I've been struggling with the deployment of our force.com application code that has, up to now, been developed in a single development org I'll call DevOrg.
What I want to do is transfer the DevOrg objects, code, pages, etc from DevOrg to a sandbox in a new org (NewOrg) that I have created. NewOrg and its sandbox are devoid of any code/objects related to the application. I've used the migration tool to retrieve the code from DevOrg and have fashioned the package.xml in various ways to get the code to deploy the NewOrg sandbox, but I keep getting tons of errors, for example:
[sf:deploy] SomeClass: line 1014, column 95: Dependent class is invalid and needs recompilation:
[sf:deploy] SomceClass2: line 475, column 37: Dependent class is invalid and needs recompilation:
I went into the Eclipse IDE and checked out the project from DevOrg, exported the project and imported it to a project pointed to NewOrg's sandbox. I thought perhaps this would give me hints about a working package.xml since I can work in the DevOrg project in the Eclipse IDE without any problems/errors.
But instead I found the same errors in the NewOrg sandbox project when loaded within the IDE. So I tried making adding a space to the dependent class and saving it to the NewOrg sandbox and I received an sObject error like this:
Save error:
sObject type 'SomeClass__c' is not supported.
If you are attempting to use a custom object, be sure to append the '__c' after the entity name.
Please reference your WSDL or the describe call for the appropriate names.
Thinking this must be because the objects haven't been deployed to NewOrg's sandbox, I attempted to the deploy only the objects in the project. That too failed. This time with errors like this:
Save error:
Appointment_Type_Form__c.Form__c : referenceTo value of 'Form_Template__c'
does not resolve to a valid sObject type Appointment_Type_Form__c.object
Save error:
Form__c : Cannot set ControlledByParent on a CustomObject without a
MasterDetail relationship field Form__c.object
Save error:
Form__c.All : In field: columns - no CustomField named
Form__c.Form_Template__c found Form__c.object
Save error:
Form__c.Form_Template__c : referenceTo value of 'Form_Template__c'
does not resolve to a valid sObject type Form__c.object
Save error:
Form_Template__c : formoverride does not exist or is not
a valid override for action Edit. Form_Template__c.object
Save error:
GWAPI_Log__c : log_detail does not exist or is not
a valid override for action View. GWAPI_Log__c.object
Save error:
Idea.Ideas_Last_7_Days : Could not resolve list
view column: IDEA.IDEA_THEME Idea.object
Save error:
Message__c.Form_in_Question__c : referenceTo value of 'Form_Template__c'
does not resolve to a valid sObject type Message__c.object
Save error:
News__c : customarticle does not exist or is not a valid override
for action Edit. News__c.object
Save error:
Question__c : Cannot set ControlledByParent on a CustomObject without
a MasterDetail relationship field Question__c.object
Save error:
Question__c.All : In field: columns - no CustomField named
Question__c.Form_Template__c found Question__c.object
Save error:
Question__c.Form_Template__c : referenceTo value of 'Form_Template__c'
does not resolve to a valid sObject type Question__c.object
Save error:
Section__c : Cannot set ControlledByParent on a CustomObject without
a MasterDetail relationship field Section__c.object
Save error:
Section__c.Form_Template__c : referenceTo value of 'Form_Template__c'
does not resolve to a valid sObject type Section__c.object
Save error:
Security_Question_Answer__c.Question_Text__c : Field Is_Patient__c
does not exist. Check spelling. Security_Question_Answer__c.object
Save error:
sObject type 'Portal_Session__c' is not supported. If you are
attempting to use a custom object, be sure to append the '__c'
after the entity name. Please reference your WSDL or the describe call for the appropriate names.
And so I have to have any luck with the new deployment in either the IDE or via the Migration Tool (MT).
I'm looking for advice on how to approach these issues when deploying to a brand new org using either IDE or MT. The errors imply, I think, something about the ordering of the deployment but as I understand from the Migration Tool Guide, Development Lifecycle Guide, etc the dependencies and ordering can be determined by the platform when deploying.
Have others run into similar issues and found that the solution was strictly in the package.xml? I'm curious because in one instance the package.xml works fine (when deploying to the original org) but in the other case (a new org) it doesn't seem sufficient.
The troubles with a virgin sandboxes is that usually you can't deploy it without a few iterations.
I used the following workflow:
deploy the custom objects which are on relationships with standard objects
deploy standard sObjects
deploy the rest of sObjects
deploy triggers and apex code
deploy VF pages/VF email templates
deploy the rest of metadata
after that I used only migration tool for deployment.
I'd like to suggest you perform deployment of sObjects by one at a time for most significant objects. Also before the deployment please check that all settings of standard objects conform to DevOrg's settings
Also you can try to change API version of metadata.

chan->cdr no data after upgrade from Asterisk 1.4.21

I have a legacy Asterisk application in C which does authentication of users, routing and billing using MySQL. I have kept it with Asterisk 1.4.21 because none of the CDR data is returned in newer versions of Asterisk.
Apparently there have been some changes in 1.4.22 https://issues.asterisk.org/jira/browse/ASTERISK-13064 that have completely changed the way CDR-s are handled. Unfortunately no helpful information was given on how to properly migrate existing code.
They have changed the order of execution, the 'h' extension is called and the CDR data is reset.
My code:
ast_log(LOG_NOTICE,"Dialing string: '%s'\n", dialstr);
app = pbx_findapp("Dial");
if (app)
res = pbx_exec(chan, app, dialstr);
ast_log(LOG_NOTICE,"Return from pbx_exec '%i', Disposition: '%s'\n", res, ast_cdr_disp2str(chan->cdr->disposition));
Other parts of the code handle chan->cdr->billsec etc, but it always gives 0 values.
After a successful call I always get this log from CLI:
Return from pbx_exec '-1', Disposition: 'NO ANSWER' while the same code works fine on 1.4.21
One solution I heard is to use ast_reset() before Dial but I am not sure how to implement it.
Any help on how to adapt this application?
You can just get DIALSTATUS variable,that is enought for you application and will be supported in future releases.
pbx_builtin_getvar_helper(chan, "DIALSTATUS");

Resources