I have a persistent class stored in a GAE datastore. I have removed one of the attributes from the class. The new records in this table show a value <none> for the removed attribute. But is there a way I can completely drop this column off the table?
Thanks.
Added the following 'migration' code according to moraes' suggestion, but it isn't achieving the desired result:
PersistenceManager pm = PMF.get().getPersistenceManager();
try {
Query q = pm.newQuery(UserLogin.class);
Collection<UserLogin> list = (Collection<UserLogin>) q.execute();
Iterator<UserLogin> iter = list.iterator();
while (iter.hasNext()) {
UserLogin obj = (UserLogin) iter.next();
obj.setLoginDate(obj.getLoginDate());
}
pm.makePersistentAll(list);
} finally {
pm.close();
}
I found the answer to this problem in this Article:
http://code.google.com/appengine/articles/update_schema.html
"Removing Deleted Properties from the Datastore
If you remove a property from your model, you will find that existing entities still have the property. It will still be shown in the admin console and will still be present in the datastore. To really clean out the old data, you need to cycle through your entities and remove the data from each one.
Make sure you have removed the properties from the model definition.
If your model class inherits from db.Model, temporarily switch it to inherit from db.Expando. (db.Model instances can't be modified dynamically, which is what we need to do in the next step.)
Cycle through existing entities (like described above). For each entity, use delattr to delete the obsolete property and then save the entity.
If your model originally inherited from db.Model, don't forget to change it back after updating all the data."
And here is an example with code:
http://sandrylogan.wordpress.com/2010/12/08/delattr/
If you are using ndb (and you probably should), you can easily delete properties by deleting them from entity._properties:
for entity in MyModel.query():
if 'old_property' in entity._values:
del entity._properties['old_property']
del entity._values['old_property']
entity.put()
Or you could make it faster by using an asynchronous query map:
#ndb.tasklet
def cleanup(entity):
if 'old_property' in entity._values:
del entity._properties['old_property']
del entity._values['old_property']
yield entity.put_async()
MyModel.query().map(cleanup)
There is no concept of "table" in datastore. Each entity can have arbitrary properties that don't follow a common schema. The only "schema" is in your model code, and existing records don't change automatically when you change your models.
So, to delete the property from existing records, you need to iterate over all records and re-save them without the property.
The datastore viewer gets its list of columns from the datastore stats, which are updated on a regular basis. If you've removed that column from every entity that had it, wait a day or two and the datastore viewer will stop showing it.
Related
Here is the issue I am facing all the time since I started to learn CakePHP 3
What is this concept of entity a real world example would help alot.
public function add()
{
// why do we have to create new entity / what is the role of entity here.
$comment = $this->Comments->newEntity();
if ($this->request->is('post','put')) {
// why do we have to use this line after posting / what is the role of this line.
$comment = $this->Comments->patchEntity($comment,$this->request->data);
if ($this->Comments->save($comment)) {
$this->Flash->success('comment submitted successfully.');
} else {
$this->Flash->error('Sorry, comment could not be updated.');
}
}
return $this->redirect($this->referer());
}
Let me open the book for you:
While Table Objects represent and provide access to a collection of
objects, entities represent individual rows or domain objects in your
application. Entities contain persistent properties and methods to
manipulate and access the data they contain.
-
why do we have to create new entity / what is the role of entity here.
Almost everything, if not all, in Cake3 works with entities, what an entity is is explained above. You need to create a new entity so that the FormHelper can work with it, AFAIR it can still work with an array if configured to do so as well but the entity should be used.
The reason entities exist is to abstract the data. Some people think entities are the representation of a DB row - that's wrong. As the book says, they can be a row but don't have to represent a row because the 3.0 ORM can work with other resources as well. In theory you can have a CSV data source that returns an entity per line.
I suggest you to read the entity code in the CakePHP core to get a deeper understanding of what else entities provide, just saying they're "just" a set of properties is to short thought.
why do we have to use this line after posting / what is the role of this line.
The post data is merged into the previously created entity, that's it. Use the API if you have basic questions like that. See the API entry for patchEntity().
In simple word, Entity is a set of one record of table and their relational table, on that you can perform operation without touch of database and encapsulate property of entity (fields of table) as you want.
Advantages of Entity.
Modifying result sets outside of the database (for formatting or otherwise)
Needing to represent both the table and row in the same class.
Data validation was a fucking nightmare.
Inconsistent API in terms of both how we handled things internally as well as what (and how) we returned stuff.
Other random stuff as you want.
You can do run-time modification of result sets. Just add a method to your entity to return results in the way you want. This also means you can use composition for managing entities (yaya traits)
Validation is beautiful. We can validate data before it gets into an object and then validate the object state in a separate step.
It is easier for developers to understand what they are dealing with. You either have an object or an array of objects. An object can be linked to data which can also include other objects, but you no longer have to guess at what the array key will be, nor whether its nested funkily.
We can iterate on the interface for tables and entities separately. We couldn't easily change internals for the old Model class because of the implications on both, whereas now we can (in theory) change one without mucking about in the other.
It looks prettier simple.
Try this:
if ($this->request->is('post','put')) {
$data = $this->request->getData();
$comment = $this->Comments->newEntity();
$comment = $this->Comments->patchEntity($comment, $data);
$status = $this->Comments->save($comment);
if ($status) {
$this->Flash->success('comment submitted successfully.');
} else {
$this->Flash->error('Sorry, comment could not be updated.');
}
}
return $this->redirect($this->referer());
}
My advice is never use Post and Put in the same function. Just for good pratice. Put works fine when you make a update using id like a parameter.
I'd like to know whether default values are applied at the time of instance creation, instance update, or only when attempting to read the value. I'd like to understand how this applies to queries.
For example, suppose I initially create a model w/o default values
class Foo (db.Model):
bar = db.BooleanProperty()
Then I create and put a few instances of Foo.
Then later I update the model
class Foo (db.Model):
bar = db.BooleanProperty(default=False)
And then I have a query
foos = Foo.all().filter('bar =', False)
Will the result include the instances of Foo that were created prior to adding the default value to the model definition?
And if I instead query as
foos = Foo.all().filter('bar !=', True)
Does this include the instances of Foo that were created prior to adding the default value to the model definition? Is it (in the case of a Boolean Property) and wrt default values any different to the previous query?
I looked for docs on this, but found none, sorry. Hopefully I'm just missing them. If possible, please provide links to GAE docs.
Update:
I found this in the docs
Entities lacking a property named in the query are ignored
Entities of the same kind need not have the same properties. To be eligible as a query result, an entity must possess a value (possibly null) for every property named in the query's filters and sort orders. If not, the entity is omitted from the indexes used to execute the query and consequently will not be included in the query's results.
Note: It is not possible to query for entities that are specifically lacking a given property. One alternative is to define a fixed (modeled) property with a default value of None, then filter for entities with None as the value of that property.
https://cloud.google.com/appengine/docs/python/datastore/queries
Assuming that default values are applied at time of entity creation or update, I believe that means that neither of the above queries would return entities that were last put prior to the default value being added to the model. Is that right?
thanks
It really depends. If you actually put a "bar" in your foo prior to having the default value or not.
1- In the case you never set bar and just "put" foos, then yes, anything you put prior to adding the default value will have nothing as their "bar".
2- In the case you actually programaticaly setup bars for your foos as you create them, then your query will return everything.
The way you phrased your question makes me believe you had it setup with # 1, so no, anything you put in your datastore before the default value wouldn't be returned. You will need to go through your whole datastore and update your foos.
While not in the official documentation, this has been validated in comments by Alex Martelli (a Googler)
The default values are assigned only on when left untouched on entity creation. Updates will keep the original value.
This default value doesn't affect queries in any way, indexes are written when you save the entity, so you would need to write the entity again with the proper value.
So lets say we have Project and Task entities in kinvey, each entity is stored in it's own collection (projects and tasks). When saving this entities I defined 'relations' so I can access the tasks related to a project easily.
// Save the Project, and save its Task as separate entity.
var promise = $kinvey.DataStore.save('project', project, {
relations : { task: 'tasks' }
});
That was easy and it's in the docs. But when deleting the task like this:
someService.deleteApplication = function(task){
return $kinvey.DataStore.destroy('tasks', task._id);
};
It does delete the item in tasks collection but an item in projects collection still holds a reference to the deleted task (In this case it says the project entity has 3 tasks (1 more than what it actually has) ).
how to overcome this ? is it in the docs?
Thanks!
It does not seem super clear from the docs but it looks like you need to update the relations in project yourself. Fetch the project using DataStore.get. Then remove the that task from the task array, then update the project using DataStore.update.
If you need to find the projects that reference that task, you can construct a query using the relation - this is from the Relational Data, Fetching, Queries area.
Kinvey does not support queries that peer into a related object’s
properties. However, you can construct queries to retrieve all
entities that have a relationship to a specific entity. This is done
by querying on the _id field of the reference.
I'm just starting to learn about NoSQL/Document storage this morning. I am used to EntityFramework/SQLServer.
My questions is the following: If I have a bunch of "documents" stored and somewhere down the line I add a property to my class that is needed by my app, how do I back-populate the already existing records?
If you change the model after the fact then you have a few options.
If you have a default value for the additional field and can wait until the next time that entity is saved for the database then you can simply add the new property and set the value to the defaultv value in the constructor.
You can use a IDocumentConversionListener (http://ayende.com/blog/66563/ravendb-migrations-rolling-updates)
You can also use https://github.com/khalidabuhakmeh/RavenMigrations which I have never used but it seems like it would do what you want.
I have a WPF Application with Entity Data Model.
I would like to set IsDeleted flag to true when entity is deleted instead of deleting from database.
I read many thread here and alot of threads are suggesting to use Modification Function Mapping with the use of conditional mapping.
I used conditional mapping for filtering IsDeleted = true rows and works great.
However, While I'd like to use Modification Function Mapping for just Delete Function, the program give me error, trying to find Function for Insert and Update as well on SaveChanges() method.
I only specified for Delete Function. Is there a way to use only Delete Modification Function and not all Insert, Update, Delete?
If not, is there any other ways that can achieve this? P.S. I read about using Instead Of Delete trigger. However, I would like to avoid it, if possible, since we are developing the program for a mobile application and triggers might slow down the application.
Thank you in advance!
We implemented a soft/logical delete in our Entity Framework 4.1 solution.
I partially described it in this answer:
Partial Answer
First, we added an IsDeleted column to all the tables in the DB that require a soft, or logical delete. In the template, I added inheritance to the IEnforceLogicalDelete interface
public interface IEnforceLogicalDelete
{
global::System.Boolean IsDeleted{ get; set; }
}
In the template where the entities are being defined we added this line:
if(entity.Properties.Where(p => p.DeclaringType == entity && (p.Name == "IsDeleted")).ToList().Count == 1){#>, IEnforceLogicalDelete<#}
This adds the interface after the entity inherits from EntityObject.
Once this is set up, it is time to either create your own save method on the Context, which is what we did, or in the Context.OnSavingChanges event.
As I said above we created our own method (SaveChangesWithHistory) where we also logged all changes (inserts, updates, and deletes) into a history table.
To do the logical delete you need to capture all of the state entries that have an EntityState of Deleted, change their state to Modified, and set your flag. Implementing from the interface makes it easy to do a full delete on any entity that needs it.
foreach (ObjectStateEntry entry in this.ObjectStateManager.GetObjectStateEntries(EntityState.Modified | EntityState.Deleted | EntityState.Added))
{
if (entry.State == EntityState.Deleted)
{
if (entry.Entity is IEnforceLogicalDelete)
{
IEnforceLogicalDelete delete = entry.Entity as IEnforceLogicalDelete;
entry.ChangeState(EntityState.Modified);
entry.ApplyOriginalValues(delete);
delete.IsDeleted = true;
}
}
}