How to write more than ~1,500 characters to a fusion table cell via the SQL API - google-app-engine

I have an AppEngine app using the Google API Python Client to access the Fusion Tables API over OAuth. When trying to run UPDATE commands, the API client is putting my whole SQL statement into the query string in the URL.
So when I write a SQL statement like this...
UPDATE <table ID> SET <column> = 'some really long piece of text...' WHERE ROWID = '1'
...I get an API call like this:
POST https://www.googleapis.com/fusiontables/v1/query?sql=UPDATE+<table ID>+SET+<column>+%3D+%27some+really+long+piece+of+text...%27+WHERE+ROWID+%3D+%271%27&alt=json
All this works fine for most things. But I'm encountering errors when writing more than ~1,500 characters (depending on how many of those are special characters I have to escape) to that cell. The answer to another question says the limit to the number of characters in a cell is 1,000,000. I'm assuming this may be because the URL is getting just way too long (for something in the pipeline from AppEngine to the Fusion Tables API servers), maybe kind of like the issue addressed in this question.
With other APIs, I'm used to sending parameters in form data for POST requests not the query string, which keeps the URL a manageable size. But the Fusion Tables API docs seem to suggest that the query string is the proper place and that nothing should be sent in the request body. The API client seems to be dutifully following this pattern (and in fact using that as the default behavior for ALL Google APIs??).
So my question is threefold:
Does anyone know if the URL can get too long as I suspect is happening?
Is the query string really the only place to send the SQL statement or will if I find a way to include it in the request body will the API accept that?
If the query string really is the only way and it really can get too long, is there another way to post large strings to fusion table cells?

So I tried it. The answer to number 2 is that you CAN put the query in the request body as form data and the API will take it (contrary to what the docs suggest). My request to Fusion Tables from App Engine now looks like this:
import httplib2
import urllib
value = 'Some really long string...'
# http is an instance of httplib2.Http
http.request('https://www.googleapis.com/fusiontables/v1/query?alt=json',
method='POST',
body=urllib.urlencode({
'sql': unicode('UPDATE <table ID>' +
' SET <column>=\'' + value + '\'' +
' WHERE ROWID = \'1\'').encode('utf-8')
}),
headers={'Content-Type': 'application/x-www-form-urlencoded'})
Note that I believe the Content-Type header is necessary, but I haven't tried it without it.
I also left some unicode and UTF-8 encoding stuff in there because chances are, for anyone who needs to support a few thousand characters, a few of those characters might be non-ascii and urllib.urlencode doesn't like non-ascii characters...
I'd still appreciate an answer to numbers 1 and 3 if anyone has any more information, but this seems to work for me for now. I'm curious as to why using form data in the request body wasn't the default approach from the beginning for the Fusion Tables team...

Related

Sending Data in JSON schema using AT Commands

I am working on MQTT connection establishment to the server.
I need to send the data to the server in JSON schema format using AT Commands.
The module used in N58 Neoway module. Using AT Commands connection got established and for publishing data or subscribing data to/from the server, it should happen in JSON format.
The AT Command used is:
AT+MQTTPUB=1,1,<"topic_name">,<"data">
I need to send the JSON schema in the place of data.
Looking for any suggestions/help.
The source code is based on C.
The problem in sending JSON through AT commands is that it contains double quotes ", that are unfortunately interpreted according to AT commands ETSI specification as the beginning of a string parameter. So, what happens in many modules is that it is impossible so send a JSON string as a parameter.
Some modems vendors solve this issue by starting an online mode in which data can be sent rawly.
N58 uses a different strategy instead, that consists in escaping the special characters. In the AT command guide it is called data link escape.
Though the guide could be better (there's not explicit explanation of data link escape), we can infer it from the examples (see for example the one in AT+UDPSEND): in order to escape " character, just write \" as you would do in a C string. Example:
AT+MQTTPUB=1,1,"topic_name","{\"menu\":{\"id\":\"1\",\"value\":\"2\"}}"

"?" character in MSSQL DB getting replaced with (capital A with grave accennt) when displayed by ASP script

I'm attempting to provide support for a legacy ASP/MSSQL web application - I wasn't involved in the development of the software (the company that built it no longer exists) & I'm not the admin of the server where it's hosted, I just manage the hosting for the owners of the site via a reseller account. I'm also not an ASP developer (more a PHP guy), and am not that familiar with it beyond the basics - updating DB connection strings after server migrations, etc.
The issue is that the site in question stores the content of individuals pages in an MSSQL database, and much of the content includes links. Almost all of the internal links on the site are format like "main.asp?123" (with "123" being the ID of a database row). The problem is, starting sometime in the last 8 months or so*, something caused the links in the DB content to show up as "main.aspÀ123" instead - in other words, the "?" character is being replaced by the "À" character (capital A with grave accent). Which, of course, breaks all of those links. Note that Stackoverflow won't allow me to include that character in the post title, because it seems to think that it indicates I'm posting in Spanish...?
(*unfortunately I don't know the timing beyond that, the site owners didn't know when the issue started occurring, so all I have to go by is an archive.org snapshot from last October, where it was working)
I attempted to manually change the "?" character in one of the relevant DB records to "?" (the HTML entity for the question mark), but that didn't make any difference. I also checked the character encoding of the HTML code used to display the content, but that doesn't seem to be the cause either - the same ASP files contain hard-coded links to some of the same pages (formatted exactly the same way), and those work correctly: the "?" doesn't get replaced.
I've also connected to the database directly with the MSSQL Management Studio Express application, but couldn't find any charset/character encoding options for either the database or the table.
And I've tried contacting the hosting provider, but they (M247 UK, in case anyone is curious) have been laughably unhelpful. The responses from them have been along the lines of "durrrrrr, we checked a totally different link that wasn't actually the one that you clearly described AND highlighted in a screenshot, and it works when we check the wrong link, so the problem must be resolved, right?" Suffice it to say, I wouldn't recommend them - used to be a customer of RedFox hosting, and the quality of customer has dropped off substantially since M247 bought them.
Any suggestions? If this were PHP/MySQL, I'd probably start by creating a small test script that did nothing but fetch one of the relevant records and display it's contents, to narrow down the issue - but I'm not familiar enough with ASP to do that here, at least not without a fair amount of googl'ing (and most of the info I can find is specific to ASP.net instead).
Edit: the thread suggested as a solution appears to be for character encoding issues when writing to MSSQL, not reading from it - and I've tried the solutions suggested in that thread, none make any difference.
Looks like you're converting from UNICODE to ASCII somewhere along the line...
Have a look at this to get a quick demo of what happens. In particular, pay attention to the ascii derived from inr, versus the ascii derived from unicode...
SELECT
t.n,
ascii_char = CHAR(t.n),
unicode_char = NCHAR(t.n),
unicode_to_ascii = CONVERT(varchar(10), NCHAR(t.n))
FROM (
SELECT TOP (1024)
n = ROW_NUMBER() OVER (ORDER BY ao.object_id)
FROM
sys.all_objects ao
) t
WHERE 1 = 1
--AND CONVERT(varchar(10), NCHAR(t.n)) ='À'
;
I found a workaround that appears to do the trick: I was previously trying to replace the ? in the code with &#63 (took out the ; so that it will show the code rather than the output), which didn't work. BUT it seems to work if I use &quest instead.
One thing to note, it seemed that I was originally incorrect in thinking that the issue was only affecting content being read/displayed from the MSSQL DB. Rather, it looks like the same problem was also occurring with static content being "echo'd" by code in the ASP scripts (I'm more of a PHP guy, not sure the correct term is for ASP's equivalent to echo is). Though the links that were hardcoded as static (rather HTML being dynamically output by ASP) were unaffected. Though chancing the ? to &quest worked for those ones too (hardest part was tracking down the file I needed to edit).

Google App Engine JDO query filter error with multiple String methods [duplicate]

I am working on a GAE Django Project where I have to implementing the search functionality, I have written a query and it fetches the data according to the search keyword.
portfolio = Portfolio.all().filter('full_name >=',key).filter('full_name <',unicode(key) + u'\ufffd')
The issue with this query is, that it is case sensitive.
Is there any way through which I can make it to work, without depending upon the case of the keyword?
Please suggest.
Thanks in advance.
You need to store normalized versions of your data at write time, then use the same normalization to search.
Store the data either all uppercase or all lowercase, optionally removing punctuation and changing all whitespace to a single space and maybe converting non-ASCII characters to some reasonable ASCII representation (which is, of course, trickier than it sounds.)
An alternative solution to this problem - where the datasets are small - is to filter the results in python after you have called them from the datastore:
for each_item in list_of_results:
if each_item.name.lower().rfind(your_search_term) != -1:
#Your results action

How pull a select html formatted chunk of a google spreadsheet using a URL

I have a tree farm.
I have a Google spreadsheet that has my inventory in the form that I took it.
I have pivot table that summarizes that sheet.
How can I run a query from the Jack Pine description page on my website that pulls the appropriate blob off the pivot table on the spreadsheet?
Here's what I've done so far:
Create a new spreadsheet that does an importrange() from the individual sheet with my pivot table.
Share to the world, published to the web. Using another browser where I am not logged in with my google ID I can see the file, and it is view only.
https://docs.google.com/spreadsheets/d/13pXb7Kek010B6s8Ez3h6yX4qF92MgvV4uMk71dJhe3o/edit#gid=0
I'm basing this on this article: [https://blog.ouseful.info/2009/05/18/using-google-spreadsheets-as-a-databace-with-the-google-visualisation-api-query-language/][1]
Now, in a query (split line for reading convenience)
https://spreadsheets.google.com/d/
13pXb7Kek010B6s8Ez3h6yX4qF92MgvV4uMk71dJhe3o/tq?
tqx=out.html&tq=select+*+where+B+contains+%27Pine,%20Jack%27
And I get the following message:
google.visualization.Query.setResponse({
"version":"0.6","status":"error","errors
[{"reason":"access_denied","message":"Access
denied","detailed_message":"Access denied"}]});
Obviously I'm missing something here. How do I troubleshoot this?
Google has changed something. This answer no longer works
Added Sunday.
The following now will fetch the entire sheet:
https://docs.google.com/spreadsheets/d/
13pXb7Kek010B6s8Ez3h6yX4qF92MgvV4uMk71dJhe3o/
edit?tqx=out.html&tq=select+A,B,C,+where+A+starts+with+%27Pine%27#gid=0
But while it fetches, the select statement returns the entire sheet, or rather the query is ignored.
(I originally had %20's for all the +'s, but Google rewrote them, or my browser does.)
This method
https://docs.google.com/spreadsheets/d/
13pXb7Kek010B6s8Ez3h6yX4qF92MgvV4uMk71dJhe3o/
gviz/tq?tq=select%20A,B,C%20where%20A%20contains%20'Pine'#gid=0
returns a file json.txt. I don't read JSON, but sliding over the brackets and punctuation the content is there.
Note the difference around gviz/tq...
Google rewrites the URL removing tq? from it.
I cannot leave the tqx=out.html in place. I get no JSON file and a 'file unavailable error.'
Turns out what I need is tqx=out:html Colon, not period.
Found the information in a table labeled "Request Format" in the document
https://developers.google.com/chart/interactive/docs/dev/implementing_data_source

python encoding characters in jinja2

It's a similar one with one of my other questions. I try to solve all the side effects of the first one.
I have stored few non-ascii characters on my database. If I make few "encoding-decoding" stuffs, I managed to work with the database queries. But I have another problem.
If I use the
self.response.out.write(mystring)
in one of my entities ( looks like this -> u'\u0395\u03c0\u03b9\u03c3\u03c4\u03ae\u03bc\u03b5\u03c2')
I can see it without any problem. But, I have a javascript which create a graph and needs a list with those strings. If I pass the list to the javascript like it is from the database, the javascript doesn't work at all. If I use the
tag2 = tag.encode("utf-8")
for every entity on the list and then pass the new list, I see all the non-ascii characters like this one -> ÎÏιÏÏήμεÏ

Resources