I have downloaded data using manage.py dumpdata some_app.SomeModel > some_app_SomeModel.json for several models and am trying to programmatically load the files like so :
with open('some_app_SomeModel.json', encoding='utf-8') as data_file:
json_data = json.loads(data_file.read())
for model_data in json_data:
model_data = model_data['fields']
SomeModel.objects.create(**model_data)
Unfortunately this does not work for foreign key fields : the id value is not turned into the corresponding model instance and I get ValueError: Cannot assign "1": "SomeModel.fk_field" must be a "FK_Model" instance. where "1" is the json value of the fk field.
How to simply fix this ?
If fk_field, then you can assign to fk_field_id, hence you can rewrite this to:
with open('some_app_SomeModel.json', encoding='utf-8') as data_file:
json_data = json.loads(data_file.read())
for model_data in json_data:
model_data = model_data['fields']
model_data['fk_field_id'] = model_data.pop('fk_field')
SomeModel.objects.create(**model_data)
Note that your items should contain a primary key, otherwise it is not said that these will link to the object with the primary key in the JSON file.
Related
I'm trying (and failing) to figure out how to use pyvmomi to retrieve a custom attribute named "owner" for all VMs in a vCenter.
I use code like this to retrieve VM name, power state, and uuid:
summary = vm.summary
print(summary.config.name, " ", summary.runtime.powerState, " summary.config.uuid)
But, I cannot figure out how to retrieve the custom attribute named "owner" for all VMs in a vCenter.
Thanks in advance
CustomAttribute is stored separately in customValue field. Each customValue have its name and key, the "Owner" in your case is the name, you need to get its key first.
here is a sample:
conn=connect.SmartConnect(host='***', user='***', pwd='***', port=443)
content = conn.RetrieveContent()
cfm = content.customFieldsManager
required_field = ["Owner"]
my_customField = {}
for my_field in cfm.field:
if my_field.name in required_field:
my_customField[my_field.key]=my_field.name
The key and its display name is in my_customField dict, you can get customValue by it
for opts in vm.customValue:
if opts.key in my_customField:
output[my_customField[opts.key]]=opts.value
and in output dict you have what you want.
Realmswift Data Model
class User {
let id = RealmOptional<Int>()
dynamic var name = ""
let albums = List<Album>()
override static func primaryKey() -> String {
return "id"
}
}
class Album: Object {
dynamic albumName = ""
let imageIDs = List<ImageID>()
}
class ImageID: Object {
let imageId = RealmOptional<Int>()
}
JSON data
{
"10001": {
"id" : 10001,
"name": "John",
"album": {
"albums": [
{
"albumName": "Summer1999",
"imageIds": [11223, 11224, 11227]
},
{
"albumName": "Xmas1999",
"imageIds": [22991, 22997]
},
{
"albumName": "NewYear2000",
"imageIds": [5556, 776, 83224, 87543]
}
]
}
}
}
I have the above json data and I m using SwiftyJSON to parse the data then write into realm. Everything is working great except for checking and updating of data (for example imageIds on json file have changed).
Question: How do i compare the JSON arrays and RealmSwift List to determine it any updates need to be written into the database?
You can take advantage of your primary key here. As Realm Swift documentation states:
Creating and Updating Objects With Primary Keys: If your model class
includes a primary key, you can have Realm intelligently update or add
objects based off of their primary key values using
Realm().add(_:update:).
So (I assume that you get the JSON from some kind of a request (REST etc.) and then parse it with SwiftyJSON to create a 'User' object) you can treat the new 'User' object as regular new 'User' and try to add it to Realm as usual, but 'update' parameter must be 'true'. If there already was a user with the id of the 'User' object you are trying to add, it will just update the existing 'User' i.e. changing its modified values from the new 'User' created by parsing new JSON data. This might look something like this:
//Parse JSON and create a 'User'
let newUserFromJSON = parseAndCreateUserFromJSON(JSONData)
let realm = try! Realm()
do {
try realm.write {
realm.add(newUserFromJSON, update: true)
}
} catch let error as NSError {
print("error writing to realm: \(error.description) & \(error)")
} catch {
print("error writing to realm: UNKNOWN ERROR")
}
I'm afraid there's probably no easy answer to this. There's no mechanism in Realm to compare the contents of a Realm Object to an external object to see if their data matches. You would need to iterate through each object in the Realm object and manually compare it.
This wouldn't be too much code to write (Since you can get a list of all of the Realm file's properties via the objectSchema property of Realm objects, and then use key-value coding to pull them out in a single for loop), but would still be a fair amount of overhead to perform the compare.
That being said, if what you're wanting to look at is just certain properties that might change (i.e. like you said, just the imageIDs property), then you could easily just check the values you need.
What bcamur has suggested is definitely the quickest (And usually preferred for JSON handing) solution here. As long as you've set your primary key properly, you can call Realm.add(_:, update:) with update set to true to update the object.
Please keep in mind this doesn't merge the new data with what was already in Realm; it'll completely overwrite the old object with the new values, which if it sounds like your ID numbers are changing, would be the best course of action.
i want to create a registration form that will be in batch with a continuation button, getting the id of the entry will help me to call the save method.
I want to immediately get the primary key of a new Entry Created using BreezeJS, Pls i need help on this.
Thanks
Not entirely sure I understand your question, but it sounds like you want to get the id of a newly saved record immediately after the save. If so then the answer below applies.
When the save promise resolves it returns both the list of saved entities as well as a keyMappings array for any entities whose ids changed as a result of the save. i.e. a mapping from temporary to real ids. i.e. (Documented here: http://www.breezejs.com/sites/all/apidocs/classes/EntityManager.html#method_saveChanges)
myEntityManager.saveChanges().then(function (saveResult) {
// entities is an array of entities that were just saved.
var entitites = saveResult.entities;
var keyMappings = saveResult.keyMappings;
keyMappings.forEach(function(km) {
var tempId = km.tempValue;
var newId = km.realValue;
});
});
On the other hand if you have an entity and you just want its 'key' you can use the EntityAspect.getKey method. (see http://www.breezejs.com/sites/all/apidocs/classes/EntityAspect.html#method_getKey)
// assume order is an order entity attached to an EntityManager.
var entityKey = order.entityAspect.getKey();
I am creating the bulkloader.yaml automatically from my existing schema and have trouble downloading my data due the repeated=True of my KeyProperty.
class User(ndb.Model):
firstname = ndb.StringProperty()
friends = ndb.KeyProperty(kind='User', repeated=True)
The automatic created bulkloader looks like this:
- kind: User
connector: csv
connector_options:
# TODO: Add connector options here--these are specific to each connector.
property_map:
- property: __key__
external_name: key
export_transform: transform.key_id_or_name_as_string
- property: firstname
external_name: firstname
# Type: String Stats: 2 properties of this type in this kind.
- property: friends
external_name: friends
# Type: Key Stats: 2 properties of this type in this kind.
import_transform: transform.create_foreign_key('User')
export_transform: transform.key_id_or_name_as_string
This is the error message I am getting:
google.appengine.ext.bulkload.bulkloader_errors.ErrorOnTransform: Error on transform. Property: friends External Name: friends. Code: transform.key_id_or_name_as_string Details: 'list' object has no attribute 'to_path'
What can I do please?
Possible Solution:
After Tony's tip I came up with this:
- property: friends
external_name: friends
# Type: Key Stats: 2 properties of this type in this kind.
import_transform: myfriends.stringToValue(';')
export_transform: myfriends.valueToString(';')
myfriends.py
def valueToString(delimiter):
def key_list_to_string(value):
keyStringList = []
if value == '' or value is None or value == []:
return None
for val in value:
keyStringList.append(transform.key_id_or_name_as_string(val))
return delimiter.join(keyStringList)
return key_list_to_string
And this works! The encoding is in Unicode though: UTF-8. Make sure to open the file in LibreOffice as such or you would see garbled content.
The biggest challenge is import. This is what I came up with without any luck:
def stringToValue(delimiter):
def string_to_key_list(value):
keyvalueList = []
if value == '' or value is None or value == []:
return None
for val in value.split(';'):
keyvalueList.append(transform.create_foreign_key('User'))
return keyvalueList
return string_to_key_list
I get the error message:
BadValueError: Unsupported type for property friends: <type 'function'>
According to Datastore viewer, I need to create something like this:
[datastore_types.Key.from_path(u'User', u'kave#gmail.com', _app=u's~myapp1')]
Update 2:
Tony you are to be a real expert in Bulkloader. Thanks for your help. Your solution worked!
I have moved my other question to a new thread.
But one crucial problem that appears is that, when I create new users I can see my friends field shown as <missing> and it works fine.
Now when I use your solution to upload the data, I see for those users without any friend entries a <null> entry. Unfortunately this seems to break the model since friends can't be null.
Changing the model to reflect this, seems to be ignored.
friends = ndb.KeyProperty(kind='User', repeated=True, required=False)
How can I fix this please?
update:
digging further into it:
when the status <missing> is shown in the data viewer, in code it shows friends = []
However when I upload the data via csv I get a <null>, which translates to friends = [None]. I know this, because I exported the data into my local data storage and could follow it in code. Strangely enough if I empty the list del user.friends[:], it works as expected. There must be a beter way to set it while uploading via csv though...
Final Solution
This turns out to be a bug that hasn't been resolved since over one year.
In a nutshell, even though there is no value in csv, because a list is expected, gae makes a list with a None inside. This is game breaking, since retrieval of such a model ends up in an instant crash.
Adding a post_import_function, which deletes the lists with a None inside.
In my case:
def post_import(input_dict, instance, bulkload_state_copy):
if instance["friends"] is None:
del instance["friends"]
return instance
Finally everything works as expected.
When you are using repeated properties and exporting to a CSV, you should be doing some formatting to concatenate the list into a CSV understood format. Please check the example here on import/export of list of dates and hope it can help you.
EDIT : Adding suggestion for import transform from an earlier comment to this answer
For import, please try something like:
`from google.appengine.api import datastore
def stringToValue(delimiter):
def string_to_key_list(value):
keyvalueList = []
if value == '' or value is None or value == []: return None
for val in value.split(';'):
keyvalueList.append(datastore.Key.from_path('User', val))
return keyvalueList
return string_to_key_list`
if you have id instead of name , add like val = int(val)
I just have a hunch about this. But if feels like I'm doing it the wrong way. What I want to do is to have a db.StringProperty() as a unique identifier. I have a simple db.Model, with property name and file. If I add another entry with the same "name" as one already in the db.Model I want to update this.
As of know I look it up with:
template = Templates.all().filter('name = ', name)
Check if it's one entry already:
if template.count() > 0:
Then add it or update it. But from what I've read .count() is every expensive in CPU usage.
Is there away to set the "name" property to be unique and the datastore will automatic update it or another better way to do this?
..fredrik
You can't make a property unique in the App Engine datastore. What you can do instead is to specify a key name for your model, which is guaranteed to be unique - see the docs for details.
I was having the same problem and came up with the following answer as the simplest one :
class Car(db.Model):
name = db.StringProperty(required=True)
def __init__(self,*args, **kwargs):
super(Car, self).__init__(*args, **kwargs)
loadingAnExistingCar = ("key" in kwargs.keys() or "key_name" in kwargs.keys())
if not loadingAnExistingCar:
self.__makeSureTheCarsNameIsUnique(kwargs['name'])
def __makeSureTheCarsNameIsUnique(self, name):
existingCarWithTheSameName = Car.GetByName(name)
if existingCarWithTheSameName:
raise UniqueConstraintValidationException("Car should be unique by name")
#staticmethod
def GetByName(name):
return Car.all().filter("name", name).get()
It's important to not that I first check if we are loading an existing entity first.
For the complete solution : http://nicholaslemay.blogspot.com/2010/07/app-engine-unique-constraint.html
You can just try to get your entity and edit it, and if not found create a new one:
template = Templates.gql('WHERE name = :1', name)
if template is None:
template = Templates()
# do your thing to set the entity's properties
template.put()
That way it will insert a new entry when it wasn't found, and if it was found it will update the existing entry with the changes you made (see documentation here).
An alternative solution is to create a model to store the unique values, and store it transationally using a combination of Model.property_name.value as key. Only if that value is created you save your actual model. This solution is described (with code) here:
http://squeeville.com/2009/01/30/add-a-unique-constraint-to-google-app-engine/
I agree with Nick. But, if you do ever want to check for model/entity existence based on a property, the get() method is handy:
template = Templates.all().filter('name = ', name).get()
if template is None:
# doesn't exist
else:
# exists
I wrote some code to do this. The idea for it is to be pretty easy to use. So you can do this:
if register_property_value('User', 'username', 'sexy_bbw_vixen'):
return 'Successfully registered sexy_bbw_vixen as your username!'
else:
return 'The username sexy_bbw_vixen is already in use.'
This is the code. There are a lot of comments, but its actually only a few lines:
# This entity type is a registry. It doesn't hold any data, but
# each entity is keyed to an Entity_type-Property_name-Property-value
# this allows for a transaction to 'register' a property value. It returns
# 'False' if the property value is already in use, and thus cannot be used
# again. Or 'True' if the property value was not in use and was successfully
# 'registered'
class M_Property_Value_Register(db.Expando):
pass
# This is the transaction. It returns 'False' if the value is already
# in use, or 'True' if the property value was successfully registered.
def _register_property_value_txn(in_key_name):
entity = M_Property_Value_Register.get_by_key_name(in_key_name)
if entity is not None:
return False
entity = M_Property_Value_Register(key_name=in_key_name)
entity.put()
return True
# This is the function that is called by your code, it constructs a key value
# from your Model-Property-Property-value trio and then runs a transaction
# that attempts to register the new property value. It returns 'True' if the
# value was successfully registered. Or 'False' if the value was already in use.
def register_property_value(model_name, property_name, property_value):
key_name = model_name + '_' + property_name + '_' + property_value
return db.run_in_transaction(_register_property_value_txn, key_name )