I'm building a 3-tier application with AngularJS, NodeJS+Express and SQL Server. I'm now sending each individual query from the backend to the database to be executed. Thought that's working well, I have now the legal requirement to audit many changes in the database.
For that, I thought about using triggers. However, the problem is that the user identified in the web application is different from the generic user logged into the database, so I don't know how to deal with this.
I thought about converting every query in the backend to a call to a stored procedure, passing each time the user id as a extra field from the frontend to the backend. However, the user id I get here is not the database user. I can't use a temporary table, as I have read in other posts, because this database is about to be accessed at the same time by thousands of users, so it's not secure in my case. I also saw others solutions, but applied to Postre (I need a generic solution, because this application has different versions, working with SQL Server, Oracle and MySQL). Finally, I can't also write code nor configurations in the server for each user, because potential clients will try the trial version without human intervention.
I need a solution that doesn't overload the performance, so it's important for the success of this application.
Thank you very much.
Related
Ok, let me explain the environment we are facing here:
We have an ASP.NET MVC 4 app that uses a SQL Server database.
This app isolates data in "projects", so when any user connects to it only can work on the data of one of this projects.
Sometimes... a group of users have to travel to remote regions for some days to retrieve data for a single project, and quite often they won't be able to have an internet connection (even mobile or satellite solutions are often out of reach).
While the displaced team works on a project, people at the office still can work on the rest of the projects (but no on the one that is abroad).
So... we are pondering the possibility of using a laptop to act as a "mobile server", where users can download the data from a specific project before travelling. While abroad, they can work against this "mobile server", update any data on their project and, when they come back, they could upload their updated data to the main server.
Our idea is to create stored procedures on both servers (main and mobile) that executes different queries to update data from a project between them, passing the project identifier as a parameter. Probably using Linked servers to allow main and mobile to see themselves during update operations.
Our questions here are:
Is this a good aproach?
Is there any other better approach that we're not seeing?
Are there any risks we should pay attention to in this or other approachs?
I've never used Bidirectional transaction replication so if that works for you, problem solved. I do have quite a bit of experience with data migration, including merging large data sets into software driven systems. And from that experience, replication has hurt us more than it has helped us (from a migration/merge view).
The biggest challenge in my opinion is going to be conflict resolution. I know you say that all of the data is in project specific databases, but there is no shared data at all? What about multiple remote users updating the same data? In that case you're going to need a little more than just replication.
Instead of maintaining two databases at all times (one for mobile, one as the regular in-house db), why not a system where a job is called to your main system indicating that a project needs to be prepared for "offline mode" (the job could be stored procedures or SSIS packages or straight T-SQL). Whatever the technology used, this job would copy all of the requested project data to a new database on the remote server/laptop and mark it somehow in the main database as read-only to prevent users in the office from updating that data.
Once the data is in offline mode on the remote server, the users can update and use the data as much as they want from that remote server. Then when the users get an internet connection or they are back in the office they can kick off another job that syncs the data to the main server, removes offline mode, and deletes/archives the remote database. Almost like a temporary project database.
Seriously, it sounds like a fun project.
Technologies to look at:
SSIS (Sql Server Integration Services) - In my experience, this is extremely fast at moving data and allows you the ability to add logic to handle conflict resolution, error logic, etc. It's free (with certain Sql Server editions) and the community is huge so supporting it should be easy. SSIS is not as dynamic as some of the specialized solutions out there.
A data migration suite like Pervasive's Data Integrator - I loved this but it's expensive. You could right an entire solution in this product that could handle the processing of your data bidirectionally and like SSIS it allows for complex programming logic.
T-SQL - With a linked server you could just write straight queries (using stored procedures if you wanted). The problem here is security on the linked server. We don't use them because of this issue. Linked Servers: Good or Bad?
Start using some of Microsoft's built in change detection technologies right off the bat. It's harder to implement when you're already using the system. Change Data Capture (CDC) will give you a full history of the records updated while Change Tracking will give you a light-weight summary of your changes. Using either technology will make syncing the data many times easier.
Change Tracking: http://msdn.microsoft.com/en-us/library/bb933874.aspx
Change Data Capture: http://msdn.microsoft.com/en-us/library/cc645937.aspx
SSIS: http://msdn.microsoft.com/en-us/library/ms169917.aspx
SQL Server Agent Jobs: http://msdn.microsoft.com/en-us/library/ms189237.aspx
I'm just starting to spec out a project that will be a fairly advanced DB with a fairly simple MVC front end, accessible over the internet, I'm unsure of how to handle users, I can see two options:
Option 1 - Use SQL Servers in-built Logins/Users to handle user authentication and use the in-built user access to control who can access what (almost everything will be selects or stored procs).
Option 2 - Use my own list of users (with hashed salted passwords) and my own access control list (possibly using proc name and OBJECT_NAME(##PROCID) passed to another proc) and do all writes/reads of the DB through an application profile with access to all procedures/views.
I've searched but can't find any reasons I should pick one over the other, can anyone share a link or provide reasons why one is better that the other (or if there are glaring issues with both)?
If you need any more details please let me know.
If using SQL authentication (Option 1 for each user) via a connection pool, you're going to get comparatively poor performance. You'll get pooling for each user, but that could still mean 100's or 1000's of connections. There are ways around that, but the solution begins to start making Option 1 work like Option 2, so you might as well use Option 2.
I've seen a few solutions that create SQL Accounts to do the inital authentication for each user (via a single application based SQL account), then the rest works like Option 2 (again via a single application based SQL account). Those cases were using lots of SQL accounts, so each user could own a set of Views, and imitate Schemas. As Schemas & Aliases have been available since SQL Server 2005, it wouldn't be worthwhile to pick option 2 anymore for that reason.
Finally, don't use SQL Server Application roles either - they're bad security practice. Brian Kelly covers a few of the issues (pos & cons) in an article about them (http://www.sqlservercentral.com/articles/Security/sqlserversecurityprosandconsofapplicationroles/1116/).
We currently have a SQL Server 2008 instance set up for our ASP.Net MVC application, and the SQL Server instance uses built-in auditing (I believe CDC?)
We also have our ASP.Net application set up to use one connection string specified in web.config for the entire application, no matter who is logged in (of about ~50 users)
The problem: We want to be able to include among the audit information the username of the user who made the particular change in question.
It looks like we can only do that in one of two ways:
Change our application setup so that every single user gets their own database login. This would require us to use dynamic connection strings (perhaps not too terrible), but moreover it would be a pain in the ass to add a new user to the system and the admins could no longer do it automagically via the application's interface (I think).
Use another solution from within the ASP.Net app itself. This would allow us to trivially add any bit of information in the application we wish, but would involve scrapping the entire builtin solution and essentially starting from scratch with a significant effort.
Someone must have run into this problem before with auditing - is #1 feasible? Or is #2 the only way we can go here?
Thanks
Option #1 is pain for the admins, a little complicated code for "dynamic" connection strings, and calling sp_addlogin to create users. But the worst is that separate DB login for every user also allows them to query DB directly (if they know instance & database names, which could leak to them in any way and "see" DB server through the network). Direct access to the DB is something that may crash every application. Don't do it, unless you're very sure that DB server is not directly reachable to the users.
About #2. How about adding column "LastModifiedLogin" to every table, where you put login of logged in user during every insert/update? This shouldn't crash CDC and you got what you want. However auditing delete is little problematic, because you issue statement and have no longer row where you could place the login. You may organize seperate "DeleteAudit" table where you put name of table, row identifier and user login on every delete, but that's only rough idea.
If you use NHibernate for data access, I would advise you to consider switching CDC to NHibernate Envers, which is very neat solution.
I am currently been assigned to develop a sync application for my company. We have SQL server on our database server which will be synced with the client database. Client databases are not known, they can be SQLite or MYSQL or whatever.
What this sync app does is, detect changes that occur on server & client databases. Save these changes and sync. If changes occur on server database it will be synced with the client database and vice versa.
I did some research on it and came to know many solutions. One of them is to use a Microsoft Sync Framework. But I hardly found a good implementation example on it for syncing with remote databases.
Then I came across Change Data Capture(CDC) on SQL Server 2008. CDC works by detecting the change on the source table through triggers and put these changes on a separate table called sync_table, this table is then used for syncing.
Since, I cannot use the CDC feature because I don't have sufficient database rights on my machine, I have started to develop my own solution which works like how CDC does. I create separate sync_table for each source table, create triggers to detect data change and put this data in the sync_table.
However, I am advised to do some more research on it for choosing the best implementation methodology.
I need to keep the following things in mind,
Databases may/may not be on the same network.
On server side, the user must be able to select which tables will take part in the sync process.
Devices that will sync with the server database need to be registered first. Meaning that all client devices will be registered by the user before they can start syncing.
As usual any help will be appreciated :)
There is an open source project called SymmetricDS with many of the same goals. Take a look at the documentation and data model to see how the problem was solved, and maybe you will get some ideas. Instead of a separate shadow table for each source table, there is a single sym_data table where all the data is captured in comma separated value format. The advantage is one place to look for captured data and retrieve changes that were part of the same transaction. The table is kept small by purging it often after data is transferred successfully. It uses web protocols (HTTP) for data transfer. The advantage is leveraging existing web servers for performance, administration, and known filtering through firewalls. There is also a registration protocol used before clients are allowed to sync. The server admin "opens registration" for a client ID, which allows the client to connect for the first time. It supports many different databases, so you'll find examples of how to write triggers and retrieve unique transaction IDs on those systems.
I've got a Python application that connected to Microsoft SQL Server 2000. The application checks for updates on startup and automatically applies them. Soon, it will need to handle database schema changes as well. Based on my research, it seems that creating a baseline script of my current database and then creating a new script for each schema change is the way to go. That way, any version of database can be updated to the newest version.
My question is how do I manage the permissions for these updates? Right now there's about 50 people using my application, most of which have read-only access. Ideally, I'd like any user to be able to perform the necessary changes such as creating or altering tables so that the first person to receive the new update will apply the new schema changes. If that doesn't happen, then he/she might not be able to use the application at all until someone with appropriate permissions updates the database.
I can see a problem occurring if every user can update the schema. What would prevent them from logging into the SQL Server Management Studio and causing issues like dropping tables, etc.?
Right now, this application is only deployed in once place, so it's easy for me to manage schema changes manually. But we do have plans to deploy to more areas and I'd for all this to be handled automatically.
You want to restrict users admin privileges, but at the same time you want them to perform admin activities... Bit of a "chicken and egg" problem :-)
Option 1: Use webapp
Convert your program to run as a web application.
Bit drastic, however, much simpler maintanence. Your users no longer need to install Python, centralized upgrades and a shared connection pool to the database.
This solves your database authentication problem. Users authenticate themselves to the web application, not the database.
I use liquibase to manage my database schemas. It has a servlet listener which can automatically upgrade the database when a new version of my app is deployed. (Liquibase also has a command-line interface for use by alternative technologies like Python).
Option 2: encrypt admin password
You encrypt the admin password and make it available as a text file retrievable from a corporate web server, enabling any users application to download it at startup.
There is a security issue with this solution.... In order to decrypt the admin password, a shared secret needs to be built into your python application.... This is security by obscurity.