I've been struggling with this issue for a while. Our company servers lack any sort of database, i.e. no MySQL, MongoDB, etc in sight.
Since we can't install any for reasons beyond the scope of this question, I was wondering if there was any alternative to that that I could use to save data from a form. (We collect prospect data through a form on our site which then sends this data in the form of an email and is plugged in our internal database through email2DB...)
You could use a library like SQLite
You could also use indexed files like Gdbm
However, you should think about backup strategies. Perhaps serialization should be a concern (and using textual or portable data formats like XDR, ASN1, JSON, YAML, ...).
But you might also try to discuss with managers to install e.g. a MySQL server on a machine. You don't need a dedicated hardware for that, it can run (at least for development and test) on a machine used for some other things.
textfile?:)
or perhaps TinySQL?
You can save it as a flat file. Flat files work great when you are just saving things like logs, or output from a webform. They quickly start to fail if you have any *-to-many relationships.
Do you have access to PHP?
Related
I was thinking - let's take a look at a computer game of any kind, or any program in general.
(Chrome, Skype, Warcraft,...)
They need to save some things that a user wanted them to save.
How do they do it?
Do they save it in a simple text file, or do they pack a database system (like MySQL,...) with themselves?
That really depends on your needs. If you only need to store some key value pairs, an application can use a simple text file (e.g. an *.ini file) That however is a plain text file readable by everybody.
An application can of course also use a database like MySql, MS SQL. However, these are not very handy if you want to distribute your application as they run as a seperate service on a server and need to be installed seperately. Then, there are databases like Sqlite which is also a SQL database, but which stores everything inside a single file. Your application just needs a way to interact with this file.
Yet another way would be to serialize/deserialize an object which holds your data you want to store.
There are other ways to store data, like NoSQL databases. I personally haven't used one of those yet, but here is a listing of some of them: http://nosql-database.org/
XML could also be used.
There are endless way an application can store its data
There is literally no end to the ways programs will store data. OTOH:
home-made archive formats: every game company seems to have a few of their own (Blizzard MoPaQ,
XML files: usually used for simple configuration (Apple's plist files, Windows application configurations, Skype's user preferences, ...)
SQLite databases: usually used for larger amounts of personal data (Firefox: bookmarks, history, etc.; iOS personal information databases, etc.)
"In the cloud" in someone else's database (basically all web apps)
Plain text or simple text formats (Windows .ini/.inf, Java MANIFEST.MF, YAML, etc.)
...
A single program might use multiple methods depending on what they're storing. There is no unified solution, and there is no one solution that is right for every task since every system has tradeoffs (human-readability vs. packing efficiency, random access vs. sequential archive, etc.)
A lot of programs use Sqlite to store data (http://www.sqlite.org). Sqlite is a very compact cross platform SQL database. Many programs do use text files.
I'm interested in opinions on what database system to select for this project where I basically need to persist a constant stream of messages at potentially high speed. There's basically four types of messages with some commonalities. No relations needed. I guess you could call it an event store.
I will need to read (query by a non-unique key), but I don't need to update any data. I will have to delete old data though.
Considerations:
Database must be able to scale out
Performance is crucial
as well as up-time (system allowing live updates would be nice)
Preferably something running on Windows Server, but this is not a requirement
I'm familiar with document databases (MongoDB), but don't know what other kinds of NoSQL solutions would fit my problem, or how they compare.
MongoDb would be ideal. But if all you want to do is read from the stream and serve up content, more than database consideration (use any db - mysql, access, sql server express, xml files), I would suggest you look at putting all your data in memory (maybe at app startup); and then serve up data from memory.
You should also look at some caching solutions like Memcached (http://memcached.org/)
I need to upgrade an old system based on Zope, I need to be able to export the data to something like SQL Server...does anyone know of a way I can open the Zope DB in .NET or directly export it to SQL Server?
Thanks,
Kieron
I am a Plone web developer, and Jason Coombs is correct. The ZODB is an Object Data Base, and contains python objects. These objects can be python code, data, meta-data, etc and are stored in a hierarchy. This is very different from the world of SQL tables and stored procedures. (The growing NoSQL movement, shows that Zope is not the only one doing this.) Additionally, since these are complex python objects you really want to be working on the ZODB with the version of python it was created with, or make sure that you can do a proper migration. I don't think that you will be able to do this with IronPython.
Without knowing what you are trying to get out of the ZODB, it is hard to give specific advice. As Jason suggested, trying WebDAV/FTP access to the ZODB might be all that you need. This allows to pull out the basic content of pages, or image files, but you may loose much of the more complex data (for example an event page may not have all of its datetime data included) and you will loose much of of the meta-data.
Here is how someone migrated from Plone to Word press:
http://www.len.ro/2008/10/plone-to-wordpress-migration/
There are a number of articles on migrating from one Plone version to another. Some of this information maybe useful to you. stackoverflow is not allowing me to post more links but search for:
"when the plone migration fails doing content migration only"
"Plone Product ContentMigration"
The first important thing to note is that the Zope Object Database (ZODB) stores Python objects in its hierarchy. Therefore, getting "data" out of the ZODB generally does not make sense outside of the Python language. So to some extent, it really depends on the type of data you want to get out.
If the data you're seeking is files (such as HTML, documents, etc), you might be able to stand up a Zope server and turn on something like WebDAV or FTP and extract the files that way.
The way you've described it, however, I suspect the data you seek is more fine-grained data elements (like numbers or accounts or some such thing). In that case, you'll almost certainly need Python of some kind to extract the data and transform it into some format suitable to import into SQL Server. You might be able to stay inside the .NET world by using IronPython, but to be honest, I would avoid that unless you can find evidence that IronPython works with the ZODB library.
Instead, I suggest making a copy of your Zope installation and zope instance (so you don't break the running system), and then use the version of Python used by Zope (often installed together) to mount the database, and manipulate it into a suitable format. You might even use something like PyODBC to connect to the SQL Server database to inject the data -- or you may punt, export to some file format, and use tools you're more familiar with to import the data.
It's been a while since I've interacted with a ZODB, but I remember this article was instrumental in helping me interact with the ZODB and understand its structure.
Good luck!
What are the problems associated with storing your Data in files rather than databases? I'm thinking in terms of something like a blog engiene. I read that MoveableType used to do this. What are the pros/cons of working this way?
Databases provide means to perform interesting queries more easily.
Examples: You would want to list the 10 most recent posts on the front page. Make an archive page that lists all articles published in a given year (taken from the url).
I think the main one is data consistency. If you keep everything together in one db table, you don't have to worry (as much) about the file being externally modified or deleted without the meta data being modified in sync. There's also the possibility of an incomplete write if the server fails while you're updating. In this case you have to take your own steps to implement transactions.
I think that with an appropriate level of care and file permissions though, these problems can be overcome.
It is much easier and more comfortable to specify access rights (to data or file) in database than to use OS specific access rights.
You can easily share data across machines and/or websites using database-stored files.
Unfortunately, it is (often) much slower to serve files stored in database.
I have been hired to help write an application that manages certain information for the end user. It is intended to manage a few megabytes of information, but also manage scanned images in full resolution. Should this project use a database, and why or why not?
Any question "Should I use a certain tool?" comes down to asking exactly what you want to do. You should ask yourself - "Do I want to write my own storage for this data?"
Most web based applications are written against a database because most databases support many "free" features - you can have multiple webservers. You can use standard tools to edit, verify and backup your data. You can have a robust storage solution with transactions.
The database won't help you much in dealing with the image data itself, but anything that manages a bunch of images is going to have meta-data about the images that you'll be dealing with. Depending on the meta-data and what you want to do with it, a database can be quite helpful indeed with that.
And just because the database doesn't help you much with the image data, that doesn't mean you can't store the images in the database. You would store them in a BLOB column of a SQL database.
If the amount of data is small, or installed on many client machines, you might not want the overhead of a database.
Is it intended to be installed on many users machines? Adding the overhead of ensuring you can run whatever database engine you choose on a client installed app is not optimal. Since the amount of data is small, I think XML would be adequate here. You could Base64 encode the images and store them as CDATA.
Will the application be run on a server? If you have concurrent users, then databases have concepts for handling these scenarios (transactions), and that can be helpful. And the scanned image data would be appropriate for a BLOB.
You shouldn't store images in the database, as is the general consensus here.
The file system is just much better at storing images than your database is.
You should use a database to store meta information about those images, such as a title, description, etc, and just store a URL or path to the images.
When it comes to storing images in a database I try to avoid it. In your case from what I can gather of your question there is a possibilty for a subsantial number of fairly large images, so I would probably strong oppose it.
If this is a web application I would use a database for quick searching and indexing of images using keywords and other parameters. Then have a column pointing to the location of the image in a filesystem if possible with some kind of folder structure to help further decrease the image load time.
If you need greater security due to the directory being available (network share) and the application is local then you should probably bite the bullet and store the images in the database.
My gut reaction is "why not?" A database is going to provide a framework for storing information, with all of the input/output/optimization functions provided in a documented format. You can go with a server-side solution, or a local database such as SQLite or the local version of SQL Server. Either way you have a robust, documented data management framework.
This post should give you most of the opinions you need about storing images in the database. Do you also mean 'should I use a database for the other information?' or are you just asking about the images?
A database is meant to manage large volumes of data, and are supposed to give you fast access to read and write that data in spite of the size. Put simply, they manage scale for data - scale that you don't want to deal with. If you have only a few users (hundreds?), you could just as easily manage the data on disk (say XML?) and keep the data in memory. The images should clearly not go in to the database so the question is how much data, or for how many users are you maintaining this database instance?
If you want to have a structured way to store and retrieve information, a database is most definitely the way to go. It makes your application flexible and more powerful, and lets you focus on the actual application rather than incidentals like trying to write your own storage system.
For individual applications, SQLite is great. It fits right in an app as a file; no need for a whole DRBMS juggernaut.
There are a lot of factors to this. But, being a database weenie, I would err on the side of having a database. It just makes life easier when things changes. and things will change.
Depending on the images, you might store them on the file system or actually blob them and put them in the database (Not supported in all DBMS's). If the files are very small, then I would blob them. If they are big, then I would keep them on he file system and manage them yourself.
There are so many free or cheap DBMS's out there that there really is no excuse not to use one. I'm a SQL Server guy, but f your application is that simple, then the free version of mysql should do the job. In fact, it has some pretty cool stuff in there.
Our CMS stores all of the check images we process. It uses a database for metadata and lets the file system handle the scanned images.
A simple database like SQLite sounds appropriate - it will let you store file metadata in a consistent, transactional way. Then store the path to each image in the database and let the file system do what it does best - manage files.
SQL Server 2008 has a new data type built for in-database files, but before that BLOB was the way to store files inside the database. On a small scale that would work too.