I have a Django application writing to sqlite3 database, which is immediately accessed by Python scripts on the same machine.
I want the python scripts to be able to read the last entry that the Django application has submitted.
Scenario
User logs in Django website from Raspberry Pi (RPi)
User enters desired data.
Data submitted to database.
User pushes push button connected to RPi, which prompts database query.
Separate python scripts perform calculations based on user input.
I am sure that the Django app writes to the database before I am trying to access this data from the python scripts. I am using sqiltebrowser to check this.
Database:
tact
-----------------
45 <- old value retrieved
60 <- new value not retrieved
60 <- new value not retrieved
Code:
conn = sqlite3.connect('Line3_Data.db')
conn.commit()
c = conn.cursor()
c.execute("SELECT tact FROM LineOEE03 ORDER BY tact desc limit 1")
current_tact = c.fetchone()
print(current_tact) #prints 45
I am aware that conn.commit() should get my session up to date, but why is this not the case here?
Related
My Google App Engine application (Python3, standard environment) serves requests from users: if there is no wanted record in the database, then create it.
Here is the problem about database overwriting:
When one user (via browser) sends a request to database, the running GAE instance may temporarily fail to respond to the request and then it creates a new process to respond this request. It results that two instances respond to the same request. Both instances make a query to database almost in the same time, and each of them finds there is no wanted record and thus creates a new record. It results as two repeated records.
Another scenery is that for certain reason, the user's browser sends twice requests with time difference less than 0.01 second, which are processed by two instances at the server side and thus repeated records are created.
I am wondering how to temporarily lock the database by one instance to prevent the database overwriting from another instance.
I have considered the following schemes but have no idea whether it is efficient or not.
For python 2, Google App Engine provides "memcache", which can be used to mark the status of query for the purpose of database locking. But for python3, it seems that one has to setup a Redis server to rapidly exchange database status among different instances. So, how about the efficiency of database locking by using Redis?
The usage of session module of Flask. The session module can be used to share data (in most cases, the login status of users) among different requests and thus different instances. I am wondering the speed to exchange the data between different instances.
Appended information (1)
I followed the advice to use transaction, but it did not work.
Below is the code I used to verify the transaction.
The reason of failure may be that the transaction only works for CURRENT client. For multiple requests at the same time, the server side of GAE will create different processes or instances to respond to the requests, and each process or instance will have its own independent client.
#staticmethod
def get_test(test_key_id, unique_user_id, course_key_id, make_new=False):
client = ndb.Client()
with client.context():
from google.cloud import datastore
from datetime import datetime
client2 = datastore.Client()
print("transaction started at: ", datetime.utcnow())
with client2.transaction():
print("query started at: ", datetime.utcnow())
my_test = MyTest.query(MyTest.test_key_id==test_key_id, MyTest.unique_user_id==unique_user_id).get()
import time
time.sleep(5)
if make_new and not my_test:
print("data to create started at: ", datetime.utcnow())
my_test = MyTest(test_key_id=test_key_id, unique_user_id=unique_user_id, course_key_id=course_key_id, status="")
my_test.put()
print("data to created at: ", datetime.utcnow())
print("transaction ended at: ", datetime.utcnow())
return my_test
Appended information (2)
Here is new information about usage of memcache (Python 3)
I have tried the follow code to lock the database by using memcache, but it still failed to avoid overwriting.
#user_student.route("/run_test/<test_key_id>/<user_key_id>/")
def run_test(test_key_id, user_key_id=0):
from google.appengine.api import memcache
import time
cache_key_id = test_key_id+"_"+user_key_id
print("cache_key_id", cache_key_id)
counter = 0
client = memcache.Client()
while True: # Retry loop
result = client.gets(cache_key_id)
if result is None or result == "":
client.cas(cache_key_id, "LOCKED")
print("memcache added new value: counter = ", counter)
break
time.sleep(0.01)
counter+=1
if counter>500:
print("failed after 500 tries.")
break
my_test = MyTest.get_test(int(test_key_id), current_user.unique_user_id, current_user.course_key_id, make_new=True)
client.cas(cache_key_id, "")
memcache.delete(cache_key_id)
If the problem is duplication but not overwriting, maybe you should specify data id when creating new entries, but not let GAE generate a random one for you. Then the application will write to the same entry twice, instead of creating two entries. The data id can be anything unique, such as a session id, a timestamp, etc.
The problem of transaction is, it prevents you modifying the same entry in parallel, but it does not stop you creating two new entries in parallel.
I used memcache in the following way (using get/set ) and succeeded in locking the database writing.
It seems that gets/cas does not work well. In a test, I set the valve by cas() but then it failed to read value by gets() later.
Memcache API: https://cloud.google.com/appengine/docs/standard/python3/reference/services/bundled/google/appengine/api/memcache
#user_student.route("/run_test/<test_key_id>/<user_key_id>/")
def run_test(test_key_id, user_key_id=0):
from google.appengine.api import memcache
import time
cache_key_id = test_key_id+"_"+user_key_id
print("cache_key_id", cache_key_id)
counter = 0
client = memcache.Client()
while True: # Retry loop
result = client.get(cache_key_id)
if result is None or result == "":
client.set(cache_key_id, "LOCKED")
print("memcache added new value: counter = ", counter)
break
time.sleep(0.01)
counter+=1
if counter>500:
return "failed after 500 tries of memcache checking."
my_test = MyTest.get_test(int(test_key_id), current_user.unique_user_id, current_user.course_key_id, make_new=True)
client.delete(cache_key_id)
...
Transactions:
https://developers.google.com/appengine/docs/python/datastore/transactions
When two or more transactions simultaneously attempt to modify entities in one or more common entity groups, only the first transaction to commit its changes can succeed; all the others will fail on commit.
You should be updating your values inside a transaction. App Engine's transactions will prevent two updates from overwriting each other as long as your read and write are within a single transaction. Be sure to pay attention to the discussion about entity groups.
You have two options:
Implement your own logic for transaction failures (how many times to
retry, etc.)
Instead of writing to the datastore directly, create a task to modify
an entity. Run a transaction inside a task. If it fails, the App
Engine will retry this task until it succeeds.
I have a single table with contact info and about 25 checkboxes. I have a form to input the contact info, and almost all the checkboxes are to specify which power tool was purchased, with the last two checkboxes specifying whether marketing should come via US Mail or Email.
My end-game is to have a CSV file containing the name, address, and email of a subset of all the records. Which records go in is simple: if I check "chainsaw" and "email", then everyone who didn't buy a chainsaw AND want email communication is filtered out. If I check "chainsaw" and "weedwacker" and "US mail" I want all records who bought EITHER a chainsaw OR a weedwacker OR both AND want US mail communications.
I envision using a form with checkboxes where checking any box filters the records to only include those records where that checkbox = yes in the table (for the power tools), and the mail or email filters it further, and then to generate a query that contains those records, then export as a CSV.
My Google-ing skills are good, my VBA coding skills are nonexistent (I'm an HTML/CSS/Joomla guy but got handed this mini project), and the results of searching Google and stackoverflow have yielded nothing that would guide me in the right direction.
I know the table structure is not optimal, but it is manageable for the person I'm making it for who would struggle with more than one table in Access. And it's based off of some archaic database software and they want the structure to stay the same.
You will need to build a sql statement in your code. It can start with:
sql = "select * from MyTable where ID > 0" 'the "Where ID > 0" is here so the remaining sql additions can all say AND
then have a series of if statments that will add to the sql statement:
if chkChainsaw = -1 then
sql = sql & " AND Chainsaw = 1"
end if
if chkweedwacker = -1 then
sql = sql & " AND weedwacker= 1"
end if
When you are done building your sql statement, you can build a recordset and export a csv file from that.
dim rs as recordset
set rs = currentdb.openrecordset(sql, dbopendynaset)
do while not rs.eof
'export data
rs.movenext
loop
Hello ppl I am trying to work with databases and I am new to Matlab.
I want to manipulate databeses created in MS Access but I don't know(I hope find a way to enter data from GUI (this GUI created using matlab ) and save in database)
I've designed the user interface in MATLAB, and create a database in MS Access
The problem I do not know how I connect between the database and MATLAB
I find some code to how connect between it.
dbpath = ['C:\Users\Esra\Documents\Esra.accdb'];
conurl = [['jdbc:odbc:Driver={Microsoft Access Driver (*.mdb, *.accdb)};DSN='';DBQ='] dbpath];
con = database('','','','sun.jdbc.odbc.JdbcOdbcDriver', conurl);
I hope find good code or book about this .
final , i don not know if it is the correct place to my question or not , if not ,please put my question in correct place
You need to run SQL queries on the database; you can do this with database.fetch (and a few other friends).
The example query from the docs:
conn = database('dbtoolboxdemo','','');
setdbprefs('DataReturnFormat','cellarray')
results = fetch(conn, 'select productdescription from producttable')
% Not in the example in the docs: this syntax, which I prefer, is also supported
results = conn.fetch('select productdescription from producttable');
Note that you will also need to know how to write SQL. For that, there are plenty of resources online - you just have to search for them.
I have 1.6 Million entities in a Google App Engine app that I would like to download. I tried using the built in bulkloader mechanism but found that it is terribly slow. While I can only download ~30 entities/second via the bulkloader, I can do ~500 entities/second by querying the datastore via a backend. A backend is necessary to circumvent the 60 second request limit. In addition, datastore queries can only live for up to 30 seconds so you need to break up your fetches across multiple queries using query cursors.
The code on the server side fetches an 1000 entities and returns a query cursor:
cursor = request.get('cursor')
devices = Pushdev.all()
if (cursor and cursor!=''):
devices.with_cursor(cursor)
next1000 = devices.fetch(1000)
for d in next1000:
t = int(time.mktime(d.created.timetuple()))
response.out.write('%s/%s/%d\n'%(d.name,d.alias,t))
response.out.write(devices.cursor())
On the client side, I have a loop that invokes the handler on the server with a null cursor to begin with and then starts to pass the cursor received by the previous invocation. It terminates when it gets an empty result.
PROBLEM: I am only able to fetch a fraction - ~20% of the entities using this method. I get a response with empty data even though the full set of entities has not been traversed. Why does this method not fetch everything comprehensively?
I couldn't find anything to confirm or deny this in the docs, but my guess is that all() has a non-deterministic ordering such that eventually one of your fetch(1000)'s will hit the "last element" and devices.cursor() will return nothing.
Try this:
devices = Pushdev.all().order('__key__')
me and my partners have this piece of code where we extract tweets in R and put it in a database, what we like to know is how to loop this piece of code, so that it loops periodically. Preferably every 30 minutes.
Here's our code:
#Load twitter package for R
library(twitteR)
#load MySQL package for R
library(RMySQL)
#Load authentication files for twitter
load(file="twitter_authentication.Rdata")
registerTwitterOAuth(cred)
#Search twitter for tweets e.g. #efteling
efteling <- searchTwitter("#efteling", n=100)
#Store the tweets into a dataframe
dataFrameEfteling <- do.call("rbind", lapply(efteling, as.data.frame))
#Setup up the connection to the database()
doConnect <- dbConnect(MySQL(), user="root", password="", dbname="portfolio", host="127.0.0.1")
dbWriteTable(doConnect, "tweetsEfteling", dataFrameEfteling)
eftelingResult <- dbSendQuery(doConnect, "select text from tweetsEfteling")
showResultEfteling <- fetch(eftelingResult, n=20)
Do you have access to crontab? If so, you can set it to run the script however frequently you like.
Here is a little information on crontab.
If your server is running linux, you can just type in
crontab -e
to pull up your personal crontab file. After that, you schedule your command.
For every 30 mins, you would use this command.
*/30 * * * * /path/to/script
Save and exit.
Have you considered using Twitter's streaming API vs REST? This would likely accomplish the same thing if you leave the connection open for an extended period of time. Plus it would cut down on API pulls. Try the streamR package.
If you still want to set it on a timer—http://statistics.ats.ucla.edu/stat/r/faq/timing_code.htm looks useful.