We have a ASP.NET MVC application with Linq2Sql and a SQL Server-Backend. The application is run on the main site of our customer, but every site has their own database in SQL Server. (Due to different reasons, they shouldn't share most of the information between each other.) However some general information is shared, this is stored in a Shared-Database, and every site-specific database has views which represent those tables in their respective databases.
For example when I have Sites S1, S2, S3 with their databases D1, D2, D3 and a shard database DS with a shared table TS
now in the databases S1-S3 I'll have a View which underlying query is simply:
SELECT * FROM DS.TS;
Having it written like this, SQL Server somehow automagically propagates all inserts, updates and deletes to DS.TS without the need of explicit instead of triggers. Which makes our lives much easier, since we also need to handle only one connection to one database and don't need to bother with two different databases.
While we write our Delete and Update commands ourselves in the application and don't use Linq2Sql, they work fine. However the insert command on the shared table is using InsertOnSubmit and fails with the following exception:
Can't perform Create, Update, or Delete operations on 'Table(TS)' because it has no primary key.
at System.Data.Linq.Table`1.InsertOnSubmit(TEntity entity)
Is there any way to make this work, or will I have to create the Insert-Commands on those shared tables by myself and execute them with DbCommand.ExecuteNonQuery()
LINQ to SQL with views doesn't know which column contains the key. You may be able to tweak the generated model and only set the column(s) that should be primary keys as appropriate. Be aware in your view however that you shouldn't do select * as it may kill performance over time.
You can set IsPrimaryKey property of the primary key in your model to True
Related
It's my first time creating an audit log for a PoS WPF application and was wondering on how exactly do I implement an auditing system because it seems like each option available has its ups and downs. So far from reading numerous articles/threads, I've narrowed down a few common practices on audit logs:
1. Triggers - Unfortunately, due to the nature of my app, I can't make use of triggers as it has no way of knowing which user has done the action. So what I did instead was to create a Stored Procedure which will handle the customer insert along with the customer log insert and its details. The Customer_Id will be provided by the application when using the Stored Procedure.
2. Have an old and new value - My initial plan was to only include the latter since I can reference its old value with the new value from the row before it but storing the the old and new value seemed more sensible, complexity-wise.
3. Use a separate database for the log / 4. Foreign Keys - This is probably my main concern, if I decide to use a separate database for the audit table, then I couldn't setup foreign keys for the customer and employee involved.
I created a mock-up erd with a master-detail table result to be shown on the wpf app to display the log to an admin and would really like your thoughts on possible problems that may arise (There's also an employee table but I forgot to put it):
https://ibb.co/dheaNK
Here's a few info that might be of help:
The database will reside together with the wpf app, which is a single computer.
The amount of customers will be less than 1000.
The amount of regular employees will be 3.
The amount of admins will be 2.
You can enable CDC Change Data Capture on SQL Server database for a specific table
This will enable you to collect all data changes on the database table logged in special tables.
You can also refer to official documents too
Here is a list of DML commands and how data changes are logged in the CDC table created for the source database table
What is good about CDC is it comes default with SQL Server and you don't have to do anything for logging. The only requirement is SQL Server Agent should be running so that the changes can be reflected on log table.
I'm working on a web-based business application where each customer will need to have their own data (think basecamphq.com type model) For scalability and ease-of-upgrades, I'd prefer to have a single database where each customer gets a filtered version of the data. The problem is how to guarantee that they stay sandboxed to their own data. Trying to enforce it in code seems like a disaster waiting to happen. I know Oracle has a way to append a where clause to every query based on a login id, but does Postgresql have anything similar?
If not, is there a different design pattern I could use (like creating a view of each table for each customer that filters)?
Worse case scenario, what is the performance/memory overhead of having 1000 100M databases vs having a single 1Tb database? I will need to provide backup/restore functionality on a per-customer basis which is dead-simple on a single database but quite a bit trickier if they are sharing the database with other customers.
You might want to look into adding Veil to your PostgreSQL installation.
Schemas plus inherited tables might work for this, create your master table then inherit tables into per-customer schemas which provide a company ID or name field default.
Set the permissions per schema for each customer and set the schema search path per user. Use the same table names in each schema so that the queries remain the same.
This involves data replication, kind of:
We have many sites with SQL Express installed, there is an 'audit' database on each site that has one table in 1st normal form (to make life simple :)
Now I need to get this table from each site, and copy the contents (say, with a Date Time Value > 1/1/200 00:00, but this will change obviously) and copy it to a big 'super table' in sql server proper, that also has the primary key as the Site Name (That needs injecting in) and the current primary key from the SQL Express table)
e.g. Many SQL Express DBs with the following table columns
ID, Definition Name, Definition Type, DateTime, Success, NvarChar1, NvarChar2 etc etc etc
And the big super table needs to have:
SiteName, ID, Definition Name, Definition Type, DateTime, Success, NvarChar1, NvarChar2 etc etc etc
Where items in bold are the primary key(s)
Is there a Microsoft (or non MS I suppose) app/tool/thing to manager copying all this data accross already, or do we need to write our own?
Many thanks.
You can use SSIS (which comes with SQL Server) to populate, it can be set up with variables to change the connection string to the various databases. I have one that loops through the whole list and does the same process using three differnt files from three differnt vendors. You could so something simliar to loop through the different site databases. Put the whole list of database you want to copy the audit data from in a table and loop through it changing the connection string each time.
However, why on earth would you want one mega audit table per site? If every table in the database populates the audit table as changes happen, then the audit table eventually becomes a huge problem for performance. Every insert, update and delete has to hit this table and then you are proposing to add an export on top of that. This seems to me to be a guaranteed structure for locking and deadlocks and all sorts of nastiness. Do yourself a favor and limit each audit table to the table it is auditing.
Things to consider:
Linked servers and sp_msforeachdb as part of a do-it-yourself solution.
SQL Server Replication (by Microsoft) (which I believe can pull data from SQL Server Express)
SQL Server Integration Services which can pull data from SQL Server Express instances.
Personally, I would investigate Integration Services first.
Good luck.
You could do this with SymmetricDS. SymmetricDS is open source, web-enabled, database independent, data synchronization/replication software. It uses web and database technologies to replicate tables between relational databases in near real time. The software was designed to scale for a large number of databases, work across low-bandwidth connections, and withstand periods of network outage.
As of right now, however, you would need to implement a custom IDataLoaderFilter extension point (in Java) to add the extra column. The metadata would be available though because your SiteName would be the external_id.
We are in the process of a multi-year project where we're building a new system and a new database to eventually replace the old system and database. The users are using the new and old systems as we're changing them.
The problem we keep running into is when an object in one system is dependent on an object in the other system. We've been using views, but have run into a limitation with one of the technologies (Entity Framework) and are considering other options.
The other option we're looking at right now is replication. My boss isn't excited about the extra maintenance that would cause. So, what other options are there for getting dependent data into the database that needs it?
Update:
The technologies we're using are SQL Server 2008 and Entity Framework. Both databases are within the same sql server instance so linked servers shouldn't be necessary.
The limitation we're facing with Entity Framework is we can't seem to create the relationships between the table-based-entities and the view-based-entities. No relationship can exist in the database between a view and a table, as far as I know, so the edmx diagram can't infer it. And I cannot seem to create the relationship manually without getting errors. It thinks all columns in the view are keys.
If I leave it that way I get an error like this for each column in the view:
Association End key property [...] is
not mapped.
If I try to change the "Entity Key" property to false on the columns that are not the key I get this error:
All the key properties of the
EntitySet [...] must be mapped to all
the key properties [...] of table
viewName.
According to this forum post it sounds like a limitation of the Entity Framework.
Update #2
I should also mention the main limitation of the Entity Framework is that it only supports one database at a time. So we need the old data to appear to be in the new database for the Entity Framework to see it. We only need read access of the old system data in the new system.
You can use linked server queries to leave the data where it is, but connect to it from the other db.
Depending on how up-to-date the data in each db needs to be & if one data source can remain read-only you can:
Use the Database Copy Wizard to create an SSIS package
that you can run periodically as a SQL Agent Task
Use snapshot replication
Create a custom BCP in/out process
to get the data to the other db
Use transactional replication, which
can be near-realtime.
If data needs to be read-write in both database then you can use:
transactional replication with
update subscriptions
merge replication
As you go down the list the amount of work involved in maintaining the solution increases. Using linked server queries will work best if its the right fit for what you're trying to achieve.
EDIT: If they're the same server then as suggested by another user you should be able to access the table with servername.databasename.schema.tablename Looks like it's an entity-framework issues & not a db issue.
I don't know about EntityToSql but I know in LinqToSql you can connect to multiple databases/servers in one .dbml if you prefix the tables with:
ServerName.DatabaseName.SchemaName.TableName
MyServer.MyOldDatabase.dbo.Customers
I have been able to click on a table in the .dbml and copy and paste it into the .dbml of the alternate project prefix the name and set up the relationships and it works... like I said this was in LinqToSql, though have not tried it with EntityToSql. I would give it shot before you go though all the work of replication and such.
If Linq-to-Entities cannot cross DB's then Replication or something that emulates it is the only thing that will work.
For performance purposes you probably want either Merge replication or Transactional with queued (not immediate) updating.
Thanks for the responses. We're going to try adding triggers to the old database tables to insert/update/delete records in the new tables of the new database. This way we can continue to use Entity Framework and also do any data transformations we need.
Once the UI functions move over to the new system for a particular feature, we'll remove the table from the old database and add a view to the old database with the same name that points to the new database table for backwards compatibility.
One thing that I realized needs to happen before we can do this is we have to search all our code and sql for ##Identity and replace it with scope_identity() so the triggers don't mess up the Ids in the old system.
I am working on creating the necessary views, triggers and stored procedures so I can make it easier for people to use Integration Service to copy data to and from our database, which is an entity-attribute-value schema, so the foreign key relationships are not always explicitly stated in the schema, but in my view I can hopefully make it more explicit.
So if I have a vehicle entity and I want to copy it, and have all the related parts of the vehicle also be copied, what should I be looking at with the service?
I am not very comfortable with Integration Service so I may ask for some clarification after responses.
Thank you.
SSIS typically loads a single branch of a dataflow into a table. A branch can split to load multiple tables.
I'd say it would be better to load to a staging table which always matches the required expectations for an entity, have the users make their dataflows to populate the staging table and then use a single INSERT/UPDATE in a SQL Command task to update your view (via an INSTEAD OF trigger, right?).
Another good possibility is to create a custom data destination component which enforces all your expectations.