I have a situation here:
In my company, there is a caotic production database which has some tables, views, procedures and functions. BUT the former project managers weren't aware of designing some naming strategies to import and export easily all the items of the database, so when they use programs like Sequel Pro or Navicat, that export items in alphabetic order, it starts giving a lot of errors because there are views created after the tables they use, or SPs that use some functions that doesn't exist in the moment of their creation.
Is there a way to import and export without checking the existence of the items? or a way to create a dependencies tree and export in that order?
Our technology is MySQL, btw.
Thank you in advance
mysqldump can handle most of the issues by creating tables for views at the beginning of the script, which will be dropped later and you can also optionally turn off foreign key checks in the dump.
There is still a known issue described in this bug report, whereas a view may contain more columns than a table engine allows.
Related
Problem:
Installing wordpress plugins to test their effectiveness, sometimes creates additional tables on database. After testing some plugins, it becomes difficult to quickly identify and delete not needed tables created by them.
What I want:
I want to add my needed tables to phpmyadmin's favorite list so that I can quickly identify newly created tables (tables without yellow star) so that I can drop those tables easily.
Question:
Is there a way to make phpmyadmin remember that favorite list without enabling phpMyAdmin configuration storage?
Or is there any other method that makes it easy to identify newly generated tables from old tables?
Using the phpMyAdmin Configuration Storage is really the only way to enable and manage the favorite tables. Is there some reason you don't want to configure it? You can set it up in another database or the same database or a handful of other ways meant to make it easy for all sorts of situations (shared hosting, limited rights, etc) so hopefully there's a solution that works for you. If not, there may be some other tool that helps you track changes.
I need to transfer sales tables form an old magento database to a new one
How can i do that without deleting the records from the new database and which are the sales tables
Given how complex Magento's schema is, I would advise against directly writing into the database. If you still want to do it there is this article that might help you understand the process as well as all the tables involved. Generally speaking the tables are prefixed with 'sales_'.
Since Magento's core import/export functionality is limited to Products and Customers, your best option is probably to look for an extension that will do this, or write your own. Here is another question on SO that is related to this, providing a link to some paid extensions that do this.
The sales ID's shouldn't conflict, assuming you do intend to transfer your customers as well since there may also be new that correspond to the sales. To keep this simple and short, which this process really is once you know which tables to export, you do the following:
Export all customer_ and sales_ tables (these may have a prefix, and will look like something like yourprefix_customer.
Then, you make sure that the last orders ID is updated in the eav_entity_store table, to make sure that Magento created new orders with the correct ID's. You would do the same for the other 3 rows within eav_entity_store which are for invoices, shipping and creditmemos.
Here more detailed tutorial on this topic if needed.
I strongly recommend use a plugin for that situation.
This free plugin worked for me, just install through Magento Connect and then (after refresh cache) you'll see "Exporter" tab on menu with option for import and export.
I'm working with a client who has a piece of custom website software that has something I haven't seen before. It has a MySQL Database backend, but most of the tables are auto-generated by the php code. This allows end-users to create tables and fields as they see fit. So it's a database within a database, but obviously without all the features available in the 'outermost' database. There are a couple tables that are basically mappings of auto-generated table names and fields to user-friendly table names and fields.* This makes queries feel very unintuitive :P
They are looking for some additional features, ones that are immediately available when you use the database directly, such as data type enforcement, foreign keys, unique indexes, etc. But since this a database within a database, all those features have to be added into the php code that runs the database. The first thing that came to my mind is Inner Platform Effect* -- but I don't see a way to get out of database emulation and still provide them with the features they need!
I'm wondering, could I create a system that gives users nerfed ability to create 'real' tables, thus gaining all the relational features for free? In the past, it's always been the developer/admin who made the tables, and then the users did CRUD operations through the application. I just have an uncomfortable feeling about giving users access to schema operations, even when it is through the application. I'm in uncharted territory.
Is there a name for this kind of system? Internally, in the code, this is called a 'collection' system. The name of 'virtual' tables and fields within the database is called a 'taxonomy'. Is this similiar to CCK or the taxonomy modules in Drupal? I'm looking for models of software that do this kind of this, so I can see what the pitfalls and benefits are. Basically I'm looking for more outside information about this kind of system.
Note this is not a simple key-value mapping, as the wikipedia article on inner-platform effect references. These work like actual tuples of multiple cells -- like simple database tables.
I've done this, you can make it pretty simple or go completely nuts with it. You do run into problems though when you put it into customers' hands, are we going to ask them to figure out primary keys, unique constraints and foreign keys?
So assuming you want to go ahead with that in mind, you need some type of data dictionary, aka meta-data repository. You have a start, but you need to add the ideas that columns are collected into tables, then specify primary and foreign keys.
After that, generating DDL is fairly trivial. Loop through tables, loop through columns, build a CREATE TABLE command. The only hitch is you need to sequence the tables so that parents are created before children. That is not hard, implement a http://en.wikipedia.org/wiki/Topological_ordering
At the second level, you first have to examine the existing database and then sometimes only issue ALTER TABLE ADD COLUMN... commands. So it starts to get complicated.
Then things continue to get more complicated as you consider allowing DEFAULTS, specifying indexes, and so on. The task is finite, but can be much larger than it seems.
You may wish to consider how much of this you really want to support, and then make a value judgment about coding it up.
My triangulum project does this: http://code.google.com/p/triangulum-db/ but it is only at Alpha 2 and I would not recommend using it in a Production situation just yet.
You may also look at Doctrine, http://www.doctrine-project.org/, they have some sort of text-based dictionary to build databases out of, but I'm not sure how far they've gone with it.
How are applications providing import / export (or backups) of data in SaaS based multi-tenancy applications, particularly single database designs?
Imports:
Keeping things simple I think basic imports are useful, ie CSV to a spec (or a way of providing a mapping between CSV columns and fields in the database.
Exports:
In single database designs I have seen XML exports and HTML (basic sitse generated) exports of data? I would assume that XML is a better option? How does one cater for relational data? Would you reference various things within XML and provide documentation of the relationships or let users figurethis out?
Are vendors providing an export/backup that can be imported back in/restored?
Your comments appreciated.
I don't know how it's done, but these are the possible scenarios:
DB-per-customer
schema-per-customer
single-schema
Case 1 is trivial in terms of backup/restore (or import/export), case 2 is similar. I would venture a guess that these 2 are the most used approaches.
The third option makes export/import difficult, but not impossible. The basic idea is that a table holds data from all companies, but distinguishes the company by a foreign key. Export and import would require the same kind of ETL tool to be used because these actions require filtering by company ID. The export procedure takes a company as a parameter and runs the task for that company only. The dump would take the form of insert statements (like the one you can get with MySQL or PostgreSQL) or XML (like the one created by DDLUtils).
There are situations where the single-schema setup comes in handy, but I don't think multi-tenancy is one of them.
This is linked to my other question when to move from a spreadsheet to RDBMS
Having decided to move to an RDBMS from an excel book, here is what I propose to do.
The existing data is loosely structured across two sheets in a work-book. The first sheet contains main record. The second sheet allows additional data.
My target DBMS is mysql, but I'm open to suggestions.
Define RDBMS schema
Define, say, web-services to interface with the database so the same can be used for both, UI and migration.
Define a migration script to
Read each group of affiliated rows from the spreadsheet
Apply validation/constraints
Write to RDBMS using the web-service
Define macros/functions/modules in spreadsheet to enforce validation where possible. This will allow use of the existing system while the new comes up. At the same time, ( i hope ) it will reduce migration failures when the move is eventually made.
What strategy would you follow?
There are two aspects to this question.
Data migration
Your first step will be to "Define RDBMS schema" but how far are you going to go with it? Spreadsheets are notoriously un-normalized and so have lots of duplication. You say in your other question that "Data is loosely structured, and there are no explicit constraints." If you want to transform that into a rigourously-defined schema (at least 3NF) then you are going to have to do some cleansing. SQL is the best tool for data manipulation.
I suggest you build two staging tables, one for each worksheet. Define the columns as loosely as possible (big strings basically) so that it is easy to load the spreadsheets' data. Once you have the data loaded into the staging tables you can run queries to assess the data quality:
how many duplicate primary keys?
how many different data formats?
what are the look-up codes?
do all the rows in the second worksheet have parent records in the first?
how consistent are code formats, data types, etc?
and so on.
These investigations will give you a good basis for writing the SQL with which you can populate your actual schema.
Or it might be that the data is so hopeless that you decide to stick with just the two tables. I think that is an unlikely outcome (most applications have some underlying structure, we just have to dig deep enough).
Data Loading
Your best bet is to export the spreadsheets to CSV format. Excel has a wizard to do this. Use it (rather than doing Save As...). If the spreadsheets contain any free text at all the chances are you will have sentences which contain commas, so make sure you choose a really safe separator, such as ^^~
Most RDBMS tools have a facility to import data from CSV files. Postgresql and Mysql are the obvious options for an NGO (I presume cost is a consideration) but both SQL Server and Oracle come in free (if restricted) Express editions. SQL Server obviously has the best integration with Excel. Oracle has a nifty feature called external tables which allow us to define a table where the data is held in a CSV file, removing the need for staging tables.
One other thing to consider is Google App Engine. This uses Big Table rather than an RDBMS but that might be more suited to your loosely-structured data. I suggest it because you mentioned Google Docs as an alternative solution. GAE is an attractive option because it is free (more or less, they start charging if usage exceeds some very generous thresholds) and it would solve the app sharing issue with those other NGOs. Obviously your organisation may have some qualms about Google hosting their data. It depends on what field they are operating in, and the sensitivity of the information.
Obviously, you need to create a target DB and the necessary table structure.
I would skip the web services and write a groovy script which reads the .xls (using the POI library), validates and saves the data in the database.
In my view, anything more involved (web services, GUI...) is not justified: these kinds of tasks are very well suited for scripts because they're concise and extremely flexible while things like performance, code base scalability and such are less of an issue here. Once you have something that works, you will be able to adapt the script to any future document with different data anomalies you run into in a matter of minutes or a few hours.
This is all assuming your data isn't in perfect order and needs to be filtered and/or cleaned.
Alternatively, if the data and validation rules aren't too complex, you can probably get good results with using a visual data transfer tool like Kettle: you just define the .xls as your source, the database table as the table, some validation/filter rules if needed and trigger the loading process. Quite painless.
If you'd rather use a tool that roll your own, check out SeekWell, which lets you write to your database from Google Sheets. Once you define your schema, Select the tables into a Sheet, then edit or insert the records and mark them for the appropriate action (e.g., update, insert, etc.). Set the schedule for the update and you're done. Read more about it here. Disclaimer--I'm a co-founder.
Hope that helps!
You might be doing more work than you need to. Excel spreadsheets can be saved as CVS or XML files and many RDBMS clients support importing these files directly into tables.
This could allow you skip writing web service wrappers and migration scripts. Your database constraints would still be properly enforced during any import. If your RDBMS data model or schema is very different from your Excel spreadsheets, however, then some translation would of course have to take place via scripts or XSLT.