I know the automatic setting is to have any models you define in models.py become database tables.
I am trying to define models that won't be tables. They need to store dynamic data (that we get and configure from APIs), every time a user searches for something. This data needs to be assembled, and then when the user is finished, discarded.
previously I was using database tables for this. It allowed me to do things like "Trips.objects.all" in any view, and pass that to any template, since it all came from one data source. I've heard you can just not "save" the model instantiation, and then it doesn't save to the database, but I need to access this data (that I've assembled in one view), in multiple other views, to manipulate it and display it . . . if i don't save i can't access it, if i do save, then its in a database (which would have concurrency issues with multiple users)
I don't really want to pass around a dictionary/list, and I'm not even sure how i was do that if I had to.
ideas?
Thanks!
Another option may be to use:
class Meta:
managed = False
to prevent Django from creating a database table.
https://docs.djangoproject.com/en/2.2/ref/models/options/#managed
Just sounds like a regular Class to me.
You can put it into models.py if you like, just don't subclass it on django.db.models.Model. Or you can put it in any python file imported into the scope of whereever you want to use it.
Perhaps use the middleware to instantiate it when request comes in and discard when request is finished. One access strategy might be to attach it to the request object itself but ymmv.
Unlike SQLAlchemy, django's ORM does not support querying on the model without a database backend.
Your choices are limited to using a SQLite in-memory database, or to use third party applications like dqms which provide a pure in-memory backend for django's ORM.
Use Django's cache framework to store data and share it between views.
Try to use database or file based sessions.
You need Caching, which will store your data in Memory and will be seperate application.
With Django, you can use various caching backend such as memcache, database-backend, redis etc.
Since you want some basic query and sorting capability, I would recommend Redis. Redis has high performance (not higher than memcache), supports datastructures (string/hash/lists/sets/sorted-set).
Redis will not replace the database, but will fit good as Key-Value Database Model, where you have to prepare the key to efficiently query the data, since Redis supports querying on keys only.
For example, user 'john.doe' data is: key1 = val1
The key would be - john.doe:data:key1
Now I can query all the data for for this user as - redis.keys("john.doe:data:*")
Redis Commands are available at http://redis.io/commands
Django Redis Cache Backend : https://github.com/sebleier/django-redis-cache/
I make my bed to MongoDB or any other nosql; persisting and deleting data is incredibly fast, you can use django-norel(mongodb) for that.
http://django-mongodb.org/
Related
I am building a website which will contain profiles in a stackoverflow fashion (generally speaking), where I first need to have a Postgres DB for all the relational data and then I need to have tags as a feature, for example a user should have their own descritptive tags such as "php,wordpress" etc.
I am thinking to store all the taglist in a non-relational database such as Mongo or Couch and then pass the related queries to fetch them into the profile.
But how would the profile database (postgres) or the model (MVC) know what tags to fetch for each profile?
I am not asking for code, only architecturally-wise, how would that be possible, from a top-down, higher perspective.
Use from CQRS and if you use the best architecture for your project, The DDD (Domain Driven Design) concept is very useful
For my new project I'm looking forward to use JSON data as a text file rather then fetching data from database. My concept is to save a JSON file on the server whenever admin creates a new entry in the database.
As there is no issue of security, will this approach will make user access to data faster or shall I go with the usual database queries.
JSON is typically used as a way to format the data for the purpose of transporting it somewhere. Databases are typically used for storing data.
What you've described may be perfectly sensible, but you really need to say a little bit more about your project before the community can comment on your approach.
What's the pattern of access? Is it always read-only for the user, editable only by site administrator for example?
You shouldn't worry about performance early on. Worry more about ease of development, maintenance and reliability, you can always optimise afterwards.
You may want to look at http://www.mongodb.org/. MongoDB is a document-centric store that uses JSON as its storage format.
JSON in combination with Jquery is a great fast web page smooth updating option but ultimately it still will come down to the same database query.
Just make sure your query is efficient. Use a stored proc.
JSON is just the way the data is sent from the server (Web controller in MVC or code behind in standind c#) to the client (JQuery or JavaScript)
Ultimately the database will be queried the same way.
You should stick with the classic method (database), because you'll face many problems with concurrency and with having too many files to handle.
I think you should go with usual database query.
If you use JSON file you'll have to sync JSON files with the DB (That's mean an extra work is need) and face I/O problems (if your site super busy).
Can someone explain to me what the fundamental difference is between Core Data (apparently, a "data store") and a database like SQLite or MySQL?
I am working on writing an iPhone app, and needed a table of static data to display. I thought core data would be a good choice for this, so I got everything set up and functioning as far as the database (i'm sorry - data STORE) went, and then went to try to import my data (it was in an excel file which I exported to CSV). I was thinking it should be a straight forward process like I have done in SQLite and other databases many times, but as it turned out after much research, the only "official" way to do this was to write a parser specifically for my data.
When I asked about this on the Apple Developer forums, the response I got was basically "What kind of idiot are you to think that you could possibly import data directly without having to write code to do it? Core data isn't a database- it's a data STORE!!" For the life of me, though, I can't see the distinction. In every way I have looked at it, core data behaves EXACTLY like a database, with a fancy way of accessing it and enough abstraction that it can use a variety of file formats for actually storing the data. In fact, I was eventually able to import my data using a simple SQLite .import command, so I really don't understand why the concept was so foreign to the responders to my original question.
So what am I missing here? What is so fundamentally different about a data store from a database that makes the concept of simple data importing completely alien to those who know the technology?
Core Data is not simply a means of persisting/storing data to and from disk as is SQL. Core Data's true function is to provide the complete model layer for the Model-View-Controller app design that the Apple API uses. As such Core Data is primarily an object-graph manager with persistence options tack onto the side.
An object-graph is a collection of live objects in memory. In Core Data, these are the managed objects. They are called "managed" objects because the managed object context observes the objects constantly making sure they are in the states and relationships that the data model says they should be in.
Core Data does provide persistence option but exactly what that option is for any particular implementation is largely hidden. You can even use the same data model and managed objects with different persistence methods, sometime in the same app.
The key difference with SQL is that SQL writes the actual data to disk whereas Core Data serializes live objects. When you look at a sqlite store in Core Data you are looking at objects that have been taken apart and "freeze dried". Obviously, "freeze drying" objects requires a rather specific data format in the sqlite store so the Core Data store uses its own custom schema that is largely the same regardless of details of the store.
That is why you can't just swap in any old SQL file and expect Core Data to import it. The SQL file is rows, tables and columns of data and not a specialized tables, columns and rows use to reconstitute freeze dried objects.
Since Core Data is first and foremost an object-graph manager, the only supported and reliable means of importing data is to create the object-graph. In the case of an SQL file, that means reading the SQL data using the SQL api and then generating managed objects from that data and then saving them to a persistent store.
That part is more work but you save time integrating the data into the rest of the app, upgrading the data and gains in reliability and maintainability.
A dictionary definition gives me:
Databases are data stores, but a data store isn't always a database.
The feature you expected isn't available in some databases either (but most are).
A data store can for example store non-relational data.
They should have just pointed you at the Wikipedia article on Core Data.
According to that article, "It allows data organised by the relational entity-attribute model to be serialised into XML, binary, or SQLite stores. The data can be manipulated using higher level objects representing entities and their relationships. Core Data manages the serialised version, providing object lifecycle and object graph management, including persistence. Core Data interfaces directly with SQLite, insulating the developer from the underlying SQL."
I guess it's the fact that "Core Data manages the serialised version" that means you can't import data directly. That is, you probably can't import data directly into SQLite in such a way that Core Data can manage it, although you probably can import data directly into SQLite in some way.
Core Data is not a data store, a data store is one part of Core Data. Core Data is closer related to an Object Relational Mapping (ORM) tool. Core Data actually has the option of using SQLite for it's datastore, but you can also choose XML files, proprietary format, or write your own datastore.
Not sure how you were able to import your data with a SQL import, shouldn't be compatible with Core Data since Core Data creates a proprietary SQL database schema that contains a ton of metadata.
Maybe it's better to think of Core Data as an "object store" and a database as a "data store". Core Data is good when you have a variety of types of object, with relationships to each other. The familiar example is a company with employees, who have bosses and reports, belong to departments, are assigned to clients, projects, etc., have schedules, go to meetings. Employees can get reassigned, etc. Even the types of relationships defined vary from time to time. That's a more heavyweight process even with Core Data, but Core Data makes it more easy than with a raw database.
If you just have "data", and not "objects", it's easier to use a database. For example if you just have a table of the elements with atomic weights, etc., you might want to just use a database.
For your application it sounds like you just have one table. It will be easy to just use SQLite, which is available, so use it if it's more convenient.
On the other hand, iOS SDK has some pre-built features that interact with Core Data. If you use SQLite you don't get those. So you might avoid custom code to import your data but have to write custom code to display your data. Tough luck. When creating software sometimes you have to write code. Weird, I know.
I want to migrate part of a Drupal 6 site to a Django application, specifically a Drupal based questions and answers section that I think would work better with OSQA. I've already created another question related to the authentication part of this integration and for the purposes of this question we can assume that all Drupal users will be recreated, at least their usernames, in the Django database. This question is about the data migration from Drupal to Django.
In Drupal I have all questions as nodes of a 'question' content type with some CCK fields and the answers to these questions are standard comments. I need help to find the best way of moving this data to OSQA in Django.
At first I thought I could use South but I'm not sure if it would be the best fit for my needs.
For now I think my best approach would be to write a Django app that connects to the Drupal database, query for all the questions with their corresponding comments and users and then insert directly to Django's database using the correct models and Django methods.
Am I on the right path? Any other suggestions?
Thanks!
At first I thought I could use South but I'm not sure if it would be the best fit for my needs.
No, South is not for this kind of migration. It is for intra-project migrations, and you will want to have it, but it doesn't really do you any good here.
"Migration" is really not a good term for what you need. What you really want to do is export data from Drupal and import it into Django.
I haven't made an in-depth analysis of the possible solutions for this, but were I asked to do the same thing, I would simply define a JSON- or XML-based interchange format for the transfer, then write one set of code to export the data from Drupal to this format, then another to import data from this format into Django. I strongly recommend against using a binary format for this interchange; the ability to load the data into a text editor to verify your data and fix things is really important.
For now I think my best approach would be to write a Django app that connects to the Drupal database, query for all the questions with their corresponding comments and users and then insert directly to Django's database using the correct models and Django methods.
If you want to skip the interchange file and do it in one step, then you don't want to write a new Django app just for the import; that's (IMHO) overkill. What you want to write is a Django management command within the app that you will be importing data into, and you probably want to use Django's support for multiple databases as well as model properties (such as db_table and db_column) for using existing database schemas. This is why I recommend the interchange file method: you wouldn't need to reimplement Drupal tables in Django models.
Mike's answer is the good path to follow. However in real world scenario you can find useful to mix different techniques, for example connect to the original Drupal database for the files referencing a local directory for file content (query for files are simple join from few tables) but processing the most structured data via a custom JSON view (e.g. nodes).
In these case a JSON View created via Views Datasource module can help you to design and select your data via a simple Drupal view. Then you can write a management command to read and parse the data as suggested before. You have to page the view in a way that doesn't request too much to process and you can even do asynchronous requests to speed up the retrieval using gevent.
In this way I parsed more than 15k of contents in less than 10 minutes, not so fast but acceptable for one-time import. If you want to store content for process it later you can save raw data on a custom model on the database or on a on-memory redis data store via python redis integration. If you want some detail I've written a detailed howto for Drupal-Django migration deepening these techniques.
I'm developing an application for GAE that has a fairly complex data model.
Based on my understanding, a good way to handle a complex data model using a noSQL database, especially GAE, is to use de-normalized "views" of data. If the browser-client wants to update some data, the server performs a write on some core data, returns "200 OK" so the client can continue, then uses the task que to update any "views" that the written data may have affected.
Then whenever the client wants to query for some objects that would normally require a SQL-join, it can instead query a "view" where all the data it needs is right there in the same "row" (or entity, in app engine's case).
The problem I am having is that all this creating and updating of views seems to be something that a library should be doing, not something I should be doing manually. Is there a tool that works with GAE where you can specify some views of your data and then expect that they will be created and handled appropriately? I believe CouchDB does this...
Views are more usually an RDBMS feature than a nonrel one. What you're describing sounds like materialized views, though, and while they're a valid solution, they're not a common one, and thus there aren't any libraries for App Engine (that I'm aware of) that do this.
More common is to simply store your data denormalized in a fashion that facilitates efficient reading, and update the data directly at write time.