I have two AWS RDS Postgres Instances. Sometimes I create new instances for applications that are (very) vaguely related to other applications. Which always leads me to the question; should I just create a new database in an existing instance or keep things separate and create a new instance instead?
I would recommend that you use the same database server (Amazon RDS instance).
You can logically separate the data via either:
CREATE DATABASE: Full logical separation. You login to one database and never see the other one. OR
CREATE SCHEMA: Data is kept separate, but can be referenced from the other. Quite common for staging areas, such as doing ETL in a Staging Schema, then publishing to a Production Schema.
From your description, I'd say that CREATE DATABASE would be appropriate.
The benefit is that you only need to manage one database and there is little impact on cost unless you need to increase the size of the database instance to handle the higher load (but it would still be cheaper than running two separate databases).
Just keep an eye on the CloudWatch metrics to be sure that the database is handling the increased load correctly.
Normally, the biggest reason for using a different server is because they are owned/managed by different teams. However, in your situation the same team seems to 'own' both data stores, so that wouldn't be an issue.
Related
I got a database in SQL where everything is in the dbo schema. Now we want to copy that schema to have two schemas in the same database with the exact same content. Is there any easy way to do this in an Azure Database?
(We want to separate our Development and UAT but still only use one database)
While the other answer posted here, using the SSIS to transfer the SQL objects, will work, I feel compelled to point out that your approach raises a lot of other concerns.
Using a single database for two environments is not a good practice. The first big issue with is how do you handle deployments? Let's say UAT is in the uat schema and development is in development is the dev schema. You make a change to the Customers table, how do you deploy the table change to both schemas? If you use SSIS, you will need an on-premise SSIS server that handles copying the changes to the various shcemas in the target database. This will create a large maintenance headache and likely lead to important changes being wiped out.
Another issue this results in how does your application target a specific schema? You can have a login defaulted to a specific schema when it runs, but many ORM tools will want to the schema ahead of time. This will force to write the code in way that could potentially force to deploy different code to different environments. This opens up the possibility that parts of the code won't get test until production.
The last concern I have is with this approach, versioning your database becomes difficult and many of the tools that are out there, won't support what you are doing. This means you will likely be creating custom processes and tools to deploy a database instead of leveraging tools built by vendors like Microsoft or Red Gate. This puts you in a position where you need to support not only the application you make for your customers, but also an application to do your job (basically doubling your work).
My suggestion is think about the need to run two environments in a single database. I'm assuming this is likely due to cost, in which case you might find this to be false. Azure has many pricing tiers to support customers with various budgets. Depending on your application workload for both environments, you will likely find you need a large DTU database to support both. You might find that by having two databases, you can leverage smaller DTUs tiers which may end up being cheaper.
Please use the CopySchema option of the Transfer SQL Server Objects Task in SSIS as explained here.
We have a requirement where we will have to move data between different database instance on regular basis. (For e.g. some customers willing to pay more for the better performance). So this is not going to be one off.
The database tables has referential integrity. Is there a way in which this can be done without rewriting sql script (or some other method) every time we migrate customers data?
I came across this How to move data between multiple database's table while maintaining foreign-key relationships/referential integrity?. However it appears that we have write script every time we migrate data (please correct me if I misunderstood the answer on this thread).
Thanks
Edit:
Both servers are using SQL Server 2012 (same version). Its an Azure SQL Server database.
They are not necessarily linked (no firewall between them)
We are only transferring some data, not the whole database. This is only for certain customers who opted pay more.
The schema are exactly same in both databases.
Preyash - please see the documentation on the Split-Merge tool. The Split-Merge tool enables you do move data between databases, as you have described, based on a sharding key (e.g., customer ID). One modification that you will need for your application is to add a shard map (i.e., a database that understand the global state of which customers resides in which databases).
Have a look into Azure Data Sync. It is much more aligned with your requirements. But you may end up in having another SQL Azure DB to maintain a Hub. Azure data Sync follows hub-spoke pattern and will let you do all flexible directional syncs with a few minutes of syncing gap. It is more simple and can set it up very fast without any scripts and all as you wanted.
I have a client/server application currently that has a Oracle 10G database. The company that I purchased the application form is not providing support. The company when I purchased the application provided me a SQL tool with a READ Only access access to approx 30-40 views.
Based on my analysis the views provide some but not all the data and I want access to data which may be in other tables
I am not a developer but the business owner so excuse my naivety in some of the questions below.
Can I export/duplicate/replicate the Oracle DB to another Oracle DB and will a Oracle DBA be able to view/access all the tables and understand the relationships
What is the best way to create a duplicate DB that keeps in sync with the application DB which we currently have. We would like to use the Duplicate DB as a backend for a website.
Thanks a lot!
ML
Assuming that the Oracle database resides on a server in your organization, it seems premature to be talking about talking about replicating the data to a different database. It is certainly possible to do so. But you can also run many, many different applications against the same database. Unless you know that the current database server would not be able to cope with the additional workload of the new application or you are planning on investing the time and effort to transform the data into better data model as part of replicating the data (which is extremely unlikely if you don't already know what the underlying data model is and if you don't already know that this data model isn't going to work well for the new application), you probably want to start with the assumption that you can probably build the new application against the existing database.
A database developer or a DBA should be able (again, assuming that you own the server) to determine what underlying tables exist. That person should be able to at least get some idea of how the tables relate to each other based on the existing view definitions. If the original company did a good job building the database, a new developer/ DBA should have a relatively easy time understanding the relationships. If the original company did shoddy work or was intentionally secretive, it will be a more challenging undertaking.
If you have another application that uses data of an existing database and needs some more, and you don't want to change the schema of the existing database, how do you do that?
Background of my question: We use an IBM product (Connections) to store user profiles. But we have lots of custom requirements (lots of custom fields and logics), so currently we create a few more tables, views and functions in the backend database of Connections to store the custom data. However, as it is IBM's internal database and we are not supposed to touch it, when we upgrade Connections, all our custom tables, views and functions are gone.
So we decide to move out our custom things. But the problem is we still need to join with the data from Connections. (Or not database join, just some other way to integrate with the data before presenting to the users. )
If we create a federated table in our own database, we can create tables and views like we used to. But would it have performance issues? And we are still going to be heavily depend on IBM's schema and have to assume they don't change it. Is it a good approach?
What are the other options we could consider?
If we create a federated table in our own database, we can create tables and views like we used to. But would it have performance issues?
Probably. Your application code would have to do joins between the IBM database tables and your database tables.
I'm assuming that Connections uses DB2. If you bring up your own DB2 database, I think you can do SQL joins between two separate DB2 databases.
Either way, this code should reside in a separate data access package made up of data access objects. The rest of your applications would use the data access package.
And we are still going to be heavily depend on IBM's schema and have to assume they don't change it.
IBM will change their schema, and you have to plan on making corresponding changes to your database and / or application.
What are the other options we could consider?
You could copy the IBM data from their database to your database. You still have to make changes to the copy process when the IBM schema table definitions change.
I can see in the AdventureWorks database that different schemas are used to group tables. Why is this done (security, ...?) and are there best-practices I can find?
thx, Lieven Cardoen
As a manager of Business Intelligence, we rely on schema for logical grouping and managing security. Here are some cases as to how we use schema:
LOGICAL ORGANIZATION
We have a general database that is loaded by SSIS packages solely for staging data before we load our operational data store (ODS). In this database, with the exception of the schema all objects are indentical in structure (table names, column names, data types, nullability, etc.) to their original source. We use the schema to indicate the original source system of the table. In some rare instances, two different databases have tables with the same name and schema allows us to continue to use the original name in the staging database.
In every database on our BI servers each team member has a test_username schema. When we create test objects in a database, this makes it easy to keep track of who made the object. It also makes it a lot easier to purge the test objects later since everyone knows who made what. Frankly, just knowing that we made it is usually enough to know it can be deleted safely, especially when we can't remember when or why we made it!
In our data controller database, we rely on schema to separate different types of processes between reports, etl, and generic resources.
In our star schema data warehouse, all objects are devided into dimension and fact schemas.
When we push data to other departmental servers, we make all BI objects on their servers use the schema bi. This makes it REALLY easy to know bi loads and maintains the table even though it isn't on our server. If the target server isn't a 2008/2005 SQL Server box, then we prefix the table with bi_.
When it gets down to it, we use schema for logical organization anytime we WOULD have appended a prefix or suffix to an object to help organize it in the absence of schema. Having said this, there are a few instances where we don't use schema on our BI servers. In our WorkingDB, everything is dbo. Our WorkingDB is used like TempDB to create temporary tables, but these tables are temporary tables that we know we will create everytime an ETL process runs. The special property of WorkingDB is that we don't ever backup the database and all ETL processes that use the database must be able to recreate their objects from scratch in the absence of the table. In this instance, we felt using schema didn't add ANY organizational value since we don't actually use the objects outside of their temporary ETL process.
SECURITY
Since we are a BI group, we don't generally build and support our own applications. We almost exclusively use other people's applications and bring data from their back-end databases to our server. However, we do have one database called bi_applications that is the back-end for a variety of small CRUD applications. These applications are usually data entry forms that we provide to the business so that they can capture data we would otherwise have to maintain in BI. It is a way of getting data that should be in production applications into BI while we wait for our low priority application enhancements to gather dust in the future development lists. Each application has a separate schema and the application account used to update the underlying tables ONLY has access to objects of the associated schema. This makes it really easy to understand, secure, and maintain the separate applications.
In a few instances, I have let power users have direct database access to our tables or stored procedures. We rely on using schema combined with roles to secure the objects. We grant permissions to the schema and users are added to roles. This allows us to easily understand which objects are used by whom without having to dig through roles to figure it out.
In short, we use schema for security purposes when we probably would have considered separating the objects out into their own databases and when we expect an application or user outside of BI to access our databases.
Although these aren't best business practices for application developers, I hope my bi use-cases may help you think of some of the ways to use schema in your end of the business.