My database entry has a UUID with the value (extracted using the Microsoft SQL Server Management Studio)
CDF86F27-AFF4-2E47-BABB-2F46B079E98B
After this is loaded into my Scala application, the toString method yields this value
276ff8cd-f4af-472e-babb-2f46b079e98b
How does this happen? And how can I programmatically create a UUID instance, when I have only the bare string CDF86F27-AFF4-2E47-BABB-2F46B079E98B at hand?
Relevant Slick code (former: table definition, latter: database access object)
class ChannelTable(tag: Tag) extends Table[ChannelTuple](tag, "Channel") {
def id = column[UUID]("Id", O.PrimaryKey)
def channelId = column[Int]("Channel_Id", O.NotNull)
def timer = column[UUID]("Timer_Id", O.NotNull)
def from = column[Timestamp]("FromTime", O.NotNull)
def to = column[Timestamp]("ToTime", O.NotNull)
def mon = column[Boolean]("Mon", O.NotNull)
def tues = column[Boolean]("Tues", O.NotNull)
def wed = column[Boolean]("Wed", O.NotNull)
def thu = column[Boolean]("Thu", O.NotNull)
def fri = column[Boolean]("Fri", O.NotNull)
def sat = column[Boolean]("Sat", O.NotNull)
def sun = column[Boolean]("Sun", O.NotNull)
def * = (id, channelId, timer, from, to, mon, tues, wed, thu, fri, sat, sun)
}
object ChannelDAO extends EntityDAO[Channel, ChannelTuple] {
private val entities = TableQuery[ChannelTable]
[...]
override def get(id: UUID)(implicit session: Session): Option[Channel] = {
val y = for {
a <- entities if a.id === id
} yield (a)
if (y.list.length > 1) throw new NonUniqueResultException
y.firstOption
}
[...]
}
Slick converts a UUID to a uniqueidentifier differently then SQL Server.
endianness is unspecified in .NET / SQL Server.
Make sure to use big endian encoding for UUIDs to be consistent with the JVM if you are getting the UUID from MSSQL.
Check out this SO post. Looks like you will need to create a method to translate the UUID.
How to read a .NET Guid into a Java UUID
Works fine for me:
> val uuidString = "CDF86F27-AFF4-2E47-BABB-2F46B079E98B"
uuidString: String = CDF86F27-AFF4-2E47-BABB-2F46B079E98B
> java.util.UUID.fromString(uuidString)
res2: java.util.UUID = cdf86f27-aff4-2e47-babb-2f46b079e98b
Related
Getting an error in executing the code. I have a datastore entity which has a property of type Date. An example date property value stored in an entity for a particular row is 2016-01-03 (19:00:00.000) EDT
The code i am executing is filtering the entity values based on date greater than 2016-01-01. Any idea what is wrong with the code
Error
ValueError: Unknown protobuf attr type <type 'datetime.date'>
Code
import pandas as pd
import numpy as np
from datetime import datetime
from google.cloud import datastore
from flask import Flask,Blueprint
app = Flask(__name__)
computation_cron= Blueprint('cron.stock_data_transformation', __name__)
#computation_cron.route('/cron/stock_data_transformation')
def cron():
ds = datastore.Client(project="earningspredictor-173913")
query = ds.query(kind='StockPrice')
query.add_filter('date', '>', datetime.strptime("2016-01-01", '%Y-%m-%d').date())
dataframe_data = []
temp_dict = {}
for q in query.fetch():
temp_dict["stock_code"] = q["stock_code"]
temp_dict["date"] = q["date"]
temp_dict["ex_dividend"] = q["ex_dividend"]
temp_dict["split_ratio"] = q["split_ratio"]
temp_dict["adj_open"] = q["adj_open"]
temp_dict["adj_high"] = q["adj_high"]
temp_dict["adj_low"] = q["adj_low"]
temp_dict["adj_close"] = q["adj_close"]
temp_dict["adj_volume"] = q["adj_volume"]
dataframe_data.append(temp_dict)
sph = pd.DataFrame(data=dataframe_data,columns=temp_dict.keys())
# print sph.to_string()
query = ds.query(kind='EarningsSurprise')
query.add_filter('act_rpt_date', '>', datetime.strptime("2016-01-01", '%Y-%m-%d').date())
dataframe_data = []
temp_dict = {}
for q in query.fetch():
temp_dict["stock_code"] = q["stock_code"]
temp_dict["eps_amount_diff"] = q["eps_amount_diff"]
temp_dict["eps_actual"] = q["eps_actual"]
temp_dict["act_rpt_date"] = q["act_rpt_date"]
temp_dict["act_rpt_code"] = q["act_rpt_code"]
temp_dict["eps_percent_diff"] = q["eps_percent_diff"]
dataframe_data.append(temp_dict)
es = pd.DataFrame(data=dataframe_data,columns=temp_dict.keys())
You seem to be using the generic google-cloud-datastore client library, not the NDB Client Library.
For google-cloud-datastore all date and/or time properties have the same format. From Date and time:
JSON
field name: timestampValue
type: string (RFC 3339 formatted, with milliseconds, for instance 2013-05-14T00:01:00.234Z)
Protocol buffer
field name: timestamp_value
type: Timestamp
Sort order: Chronological
Notes: When stored in Cloud Datastore, precise only to microseconds; any additional precision is rounded down.
So when setting/comparing such properties try to use strings formatted as specified (or integers for protobuf Timestamp?), not directly objects from the datetime modules (which work with the NDB library). The same might be true for queries as well.
Note: this is based on documentation only, I didn't use the generic library myself.
I'm trying to access the table from a database. First I checked the connection of the database and it worked. Then I checked the close db and it worked. After I try to select couple of items from the table and print them using eachraw command.
I'm getting below error:
Wed Oct 05 07:26:01 EDT 2016:INFO:No signature of method: groovy.sql.Sql.eachrow() is applicable for argument types: (java.lang.String, Script26$_run_closure1) values: [SELECT Type,Description FROM A.dbo.Type, ...]
Possible solutions: eachRow(java.lang.String, groovy.lang.Closure), eachRow(groovy.lang.GString, groovy.lang.Closure), eachRow(java.lang.String, groovy.lang.Closure, groovy.lang.Closure), eachRow(java.lang.String, java.util.List, groovy.lang.Closure), eachRow(java.lang.String, java.util.Map, groovy.lang.Closure), eachRow(java.util.Map, java.lang.String, groovy.lang.Closure)
This is my code:
import groovy.sql.Sql
try{
def dbURL="jdbc:sqlserver://1.1.2.1:1433;databaseName=A"
def dbUsername="sa"
def dbPassword="password"
def dbDriver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
def db = Sql.newInstance(dbURL,dbUsername,dbPassword,dbDriver)
def q1 = "SELECT Type,Description FROM A.dbo.Type"
db.eachrow(q1){row ->
log.info "${it.toString().Description}"
}
}catch (Exception e){
log.info "Some db error"
log.info e.getMessage()
}finally{
db.close()
}
The method is eachRow, no eachrow, and if you specify a row variable name inside the closure, you have to use 'row' instead of 'it'.
So you code should look like this:
db.eachRow(q1){row ->
log.info "${row.description}"
}
I'm using google app engine and need to have the keys of an entity between 1000 and 2^31. I'm considering 2 ways of doing this:
1) keep a counter of the created keys as detailed here https://cloud.google.com/appengine/articles/sharding_counters. But this requires several datastore read/writes for every key and I'm not sure it is guaranteed to be consistent.
2) generate a random int in my range and check if that key is already in the database. To make it cheap, i'd like a keys_only query, but i can't find a way to do this except saving the key also as a separate field:
MyEntity.query(MyEntity.key_field==new_random_number).fetch(keys_only=True)
Is there a better way to achieve this?
How many writes per second are you expecting in production? Both of your proposals are good, but for our application I decided to go with a sharded counter approach. You can also set the id of an entity before you put it to avoid the query altogether:
MyModel(id="foo")
then you can look it up:
MyModel.get_by_id("foo")
Id doesn't have to be a string, it can be a number also:
MyModel(id=123)
If you decide to go with the sharded counter, here's our production-level code which is darn close what you read in that article ;o) Memcache adds the level of consistency we needed to be able to get the right count.
class GeneralShardedCounterConfig(ndb.Model):
SHARD_KEY_TEMPLATE = 'gen-count-{}-{:d}'
num_shards = ndb.IntegerProperty(default=200)
#classmethod
def all_keys(cls, name):
config = cls.get_or_insert(name)
shard_key_strings = [GeneralShardedCounterConfig.SHARD_KEY_TEMPLATE.format(name, index)
for index in range(config.num_shards)]
return [ndb.Key(GeneralShardedCounter, shard_key_string)
for shard_key_string in shard_key_strings]
class GeneralShardedCounter(BaseModel):
count = ndb.IntegerProperty(default=0)
#classmethod
def get_count(cls, name):
total = memcache.get(name)
if total is None:
total = 0
all_keys = GeneralShardedCounterConfig.all_keys(name)
for counter in ndb.get_multi(all_keys):
if counter is not None:
total += counter.count
memcache.set(name, total, constants.SHORT_MEMCACHE_TTL)
return total
#classmethod
#ndb.transactional(retries=5)
def increase_shards(cls, name, num_shards):
config = GeneralShardedCounterConfig.get_or_insert(name)
if config.num_shards < num_shards:
config.num_shards = num_shards
config.put()
#classmethod
#ndb.transactional(xg=True)
def _increment(cls, name, num_shards):
index = random.randint(0, num_shards - 1)
shard_key_string = GeneralShardedCounterConfig.SHARD_KEY_TEMPLATE.format(name, index)
counter = cls.get_by_id(shard_key_string)
if counter is None:
counter = cls(id=shard_key_string)
counter.count += 1
counter.put()
# Memcache increment does nothing if the name is not a key in memcache
memcache.incr(name)
#classmethod
def increment(cls, name):
config = GeneralShardedCounterConfig.get_or_insert(name)
cls._increment(name, config.num_shards)
#classmethod
def _add(cls, name, value, num_shards):
index = random.randint(0, num_shards - 1)
shard_key_string = GeneralShardedCounterConfig.SHARD_KEY_TEMPLATE.format(name, index)
counter = cls.get_by_id(shard_key_string)
if counter is None:
counter = cls(id=shard_key_string)
counter.count += value
counter.put()
# Memcache increment does nothing if the name is not a key in memcache
memcache.incr(name, value)
#classmethod
def add(cls, name, value):
config = GeneralShardedCounterConfig.get_or_insert(name)
cls._add(name, value, config.num_shards)
Example of get_or_insert. Insert 7 unique keys
import webapp2
from google.appengine.ext import ndb
from datetime import datetime
import random
import logging
class Examples(ndb.Model):
data = ndb.StringProperty()
modified = ndb.DateTimeProperty(auto_now=True)
created = ndb.DateTimeProperty() # NOT auto_now_add HERE !!
class MainHandler(webapp2.RequestHandler):
def get(self):
count = 0
while count < 7:
random_key = str(random.randrange(1, 9))
dt_created = datetime.now()
example = Examples.get_or_insert(random_key, created=dt_created, data='some data for ' + random_key)
if example.created != dt_created:
logging.warning('Random key %s not unique' % random_key)
continue
count += 1
self.response.write('Keys inserted')
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)
I got following problem in sqlalchemy. I made three different tables in sqlite, the first has no realtion, the second has a relation to the first and the third a relation to the second. So when I want to insert things in the first and the second table everything works fine. When I want to insert datas in the third table I'm getting troubles when I'm going to do it like it's discribed in the tutorial. Here is my code for the three tables:
First table:
# Save_Data_Type.py
from sqlalchemy import ForeignKey
from sqlalchemy import Column, Float, Integer, String
from base import Base
from sqlalchemy.orm import relationship, backref
########################################################################
class Save_Data_Type(Base):
__tablename__ = "save_datas_type"
save_datas_type_id = Column(Integer, primary_key=True)
type_memory = Column(String)
comment = Column(String)
dummy1 = Column(String)
dummy2 = Column(String)
#----------------------------------------------------------------------
def __init__(self, type_memory, comment, dummy1, dummy2):
""""""
self.type_memory = type_memory
self.comment = comment
self.dummy1 = dummy1
self.dummy2 = dummy2
Second Table
# Save_Data.py
from sqlalchemy import ForeignKey
from sqlalchemy import Column, Float, Integer, String
from base import Base
from sqlalchemy.orm import relationship, backref
########################################################################
class Save_Data(Base):
""""""
__tablename__ = "save_datas"
save_datas_id = Column(Integer, primary_key=True)
save_datas_type_id = Column(Integer,ForeignKey("save_datas_type.save_datas_type_id"))
save_datas_type = relationship("Save_Data_Type", backref=backref("save_datas", order_by=save_datas_id))
value = Column(Float)
comment = Column(String)
dummy1 = Column(String)
#----------------------------------------------------------------------
def __init__(self, value, comment, dummy1):
""""""
self.value = value
self.comment = comment
self.dummy1 = dummy1
Third Table
# Station.py
from sqlalchemy import ForeignKey
from sqlalchemy import Column, Integer, Boolean, String
from base import Base
from sqlalchemy.orm import relationship, backref
########################################################################
class Station(Base):
""""""
__tablename__ = "stations"
stations_id = Column(Integer, primary_key=True)
name = Column(String)
password = Column(String)
save_datas_id = Column(Integer,ForeignKey("save_datas.save_datas_id"))
save_datas = relationship("Save_Data", backref=backref("stations", order_by=stations_id))
dummy1 = Column(String)
dummy2 = Column(String)
dummy3 = Column(String)
#----------------------------------------------------------------------
def __init__(self, name, password, dummy1, dummy2, dummy3):
""""""
self.name = name
self.password = password
self.dummy1 = dummy1
self.dummy2 = dummy2
self.dummy3 = dummy3
base.py
# base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
So if I'm going to insert the datas like that:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
import base
from Save_Data_Type import Save_Data_Type
from Save_Data import Save_Data
from Station import Station
engine = create_engine('sqlite:///Database.db', echo=True)
base.Base.metadata.create_all(engine, checkfirst=True)
# create a Session
Session = sessionmaker(bind=engine)
session = Session()
jack = Save_Data_Type("Ring Memory",'In seconds',None,None)
jack.save_datas = [Save_Data(1980,'Sometimes more, sometimes less',None)]
jack.save_datas.stations = [Station('name1','123456',None,None,None)]
session.add(jack)
session.commit()
Nothing is writing in the the third table of the database. How is the usual way to build this relationship?
Thanks in advance,
Johannes
Apart from some typos in the code you posted (see comment from zzzeek), it looks like the problem will be solved by replacing the code:
jack.save_datas.stations = [Station('name1','123456',None,None,None)]
with the one below:
jack.save_datas[0].stations = [Station('name1','123456',None,None,None)]
The reason is the following: in your case you assign/create a stations attribute to a class member. Not only this does not do the right job, it might actualy harm your ORM model.
In the second case you correctly assign the Station instance to the first save_datas.
The more clear code would look like this:
jack = Save_Data_Type("Ring Memory",'In seconds',None,None)
save_data1 = Save_Data(1980,'Sometimes more, sometimes less',None)
jack.save_datas.append(save_data1)
# or: jack.save_datas = [save_data1]
# or (my favourite): save_data1.Save_Data_Type = jack
save_data1.stations = [Station('name1','123456',None,None,None)]
# or... similar to the relationship above
I'm trying to filter database entries by their datetime:
models.py:
from django.db import models
class ContestEvent(models.Model):
year = models.DateTimeField()
month = models.DateTimeField()
In my view I define a function calender:
def calendar(request, pYear, pMonth):
"""
Show calendar of events for specified month and year
"""
lYear = int(pYear)
lMonth = int(pMonth)
lCalendarFromMonth = datetime(lYear, lMonth, 1)
lCalendarToMonth = datetime(lYear, lMonth, monthrange(lYear, lMonth)[1])
my_workouts = ContestEvent.objects.filter(id__year=lCalendarFromMonth, id__month=lCalendarToMonth)
lCalendar = html_calendar.WorkoutCalendar(my_workouts).formatmonth(lYear, lMonth)
Then I got the following error:
Request Method: GET
Request URL: http://127.0.0.1:8000/htmlcalendar/
Exception Type: TypeError
Exception Value:
int() argument must be a string or a number, not 'datetime.datetime'
Exception Location: C:\Python25\lib\site-packages\django\db\models\fields\__init__.py in get_db_prep_lookup, line 225
The error is the same, when I define my Model with year = models.IntegerField() instead of year = models.DateTimeField()
What is wrong here. I'm a Django Beginner.
Thank you in advance !!
I would say if you want the month and year then use IntegerField, and when you do that you no longer need to create datetime objects to pass to your filter. Try something like this:
class ContestEvent(models.Model):
year = models.IntegerField()
month = models.IntegerField()
def calendar(request, pYear, pMonth):
"""
Show calendar of events for specified month and year
"""
lYear = int(pYear)
lMonth = int(pMonth)
my_workouts = ContestEvent.objects.filter(year=lYear, month=lMonth)