I have a web application that has a requirement to take data from an Excel workbook and load it to the database. I am using ADO.NET 2.0 and need to use DataSet/DataTable/DataAdapter/etc. to perform this task.
We also have a requirement of verifying the data before it is uploaded and informing the user if there are any PK/FK/Unique/Other constraint violations.
Currently, we are uploading the data from Excel into a DataTable and getting the data from the database into another DataTable, merging the two tables and (if everything merged successfuly) then we update the database.
Can I use methods on DataTable or DataSet (or another ADO.Net data structure) to check for PK/FK/Unique/Other constraints? What would the recommended workflow be to upload the data to the database given my requirements?
Can I use methods on DataTable or
DataSet (or another ADO.Net data
structure) to check for
PK/FK/Unique/Other constraints?
I don't think so.
Instead you can start transaction and catch all the SqlException-s.
Then you could check the Number of the exception to see what kind of error it is (FK violation, Unique constraint etc)
What would the recommended workflow be
to upload the data to the database
given my requirements?
You may also consider using Sql Server Integration Services. It allows you to do pretty much everything.
What you describe is a "standard" ETL job. Although you can do it with custom scripts, there are available tools which allow you to focus on he job itself, as opposed to custom development. There is the oracle data integrator and Pentaho suite (open source) available for download and trial.
Related
I'm exploring options on how to one-way sync from a table available via API to an SQL database. Does anyone have any suggestions on how to achieve this?
The data from the "Source" is often updated and should be copied to the "Destination" as the changes happen (live).
Source
Read Only table from an ERP available via an API. Webhooks on the source are not possible. Entries to this table may be created, updated or deleted. There would be approximately 150,000 entries in the table with about 1000 changes per day.
Destination
Azure MS SQL database which I have full control over.
I'm looking for best practice or any ideas on how to achieve this. There seems to be very few articles that I can find with anything helpful.
I'm open to using any tool on Azure including Logic Apps and Azure Functions but want to stay away from using 3rd party tools.
If you are trying to achieving this through logic apps, Below is the flow that you can follow.
Note: Make sure you preprocess the data before sending the data to SQL database using appropriate actions based on the type of data that you are receiving.
I want to be able to create a database and tables the same way Amazon's DynamoDb client does it but with Sql Server. Is that possible?
I'm using .net Core and this is for integration tests. Figured I can throw in the code in the fixture.
Anyone have any ideas?
EF Core Migrations:
"The migrations feature in EF Core provides a way to incrementally
update the database schema to keep it in sync with the application's
data model while preserving existing data in the database."
Create and Drop APIs:
"The EnsureCreated and EnsureDeleted methods provide a lightweight
alternative to Migrations for managing the database schema. These
methods are useful in scenarios when the data is transient and can be
dropped when the schema changes. For example during prototyping, in
tests, or for local caches."
to create your tables at runtime.
And then use one of the Data Seeding techniques:
Data seeding is the process of populating a database with an initial
set of data. There are several ways this can be accomplished in EF
Core:
Model seed data
Manual migration customization
Custom initialization logic
to populate them with known data.
You could start the SQL Server (at least the logfiles) on a RAM disk. and/or use delayed durability ALTER DATABASE x SET DELAYED_DURABILITY = forced. You could also use memory optimized tables but I think you won’t get full compatibility.
BTW: it is dangerous to use such shortcuts if your development process relies entirely on it since developers very late get feedback on bad habits and performance problems.
For that kind of volatile databases (also applies to containers) you need to add some code to your test pipeline or product tomactually create and populate the DB. (If you use containers you can think about packaging a pre-populated DB snapshot)
I'm modeling a new microservice architecture migrating some part of a monolithic software to microservices.
I'm adding a new PostgreSQL database and the idea is in the future use that database but for now I still need to keep updated the old SQL Server database and also synchronize the PostgreSQL database if something new appears in the old database.
I've searched for ETL tools but are meant to move data to a datawarehouse (that's not what I need). I just can't replicate the information because the DB model is not the same.
Basically I need a way to detect new rows inserted in the SQL Server database, transform that information and insert it in my PostgreSQL.
Any suggestions?
PostgreSQL's foreign data wrappers might be useful. My approach would be, to change the frontend to use PostgreSQL and let postgreSQL handle the split via it's various features (triggers, rules, ...)
Take a look at StreamSets Data Collector. It can detect changes in SQL Server and insert/update/delete to any DB that has a JDBC driver including Postgres. It is open source but you can buy support. You can also make field changes/additions/removals/renaming to the data stream so that the fields match the target table.
I have a Dynamics CRM 2011 instance where the database has become corrupted. The corrupted data appears to be isolated to a few tables (e.g. PrincipalObjectAccess) and the instance still functions normally to all appearances. The data is irretrievable (all forms of DBCC CHECKDB, etc. have been run) and a backup is not available (preaching on backups will not help resolve the issue).
I've tried using schema and data synchronization tools like those offered by dbForge and Red-Gate, the schema sync works but the data sync always seems to come up inconsistent.
At this juncture I think my best route is probably to export all data from Dynamics CRM 2011 and then import it into a new instance of Dynamics CRM 2011. Any thoughts on the best way to accomplish this? Or alternative methods of rectifying the situation?
Exporting all data and importing it into new organization will likely create more errors and I wouldn't really go with that option unless everything else fails.
You said data synchronization failed: have you tried deleting all data from new instance first and then running data synchronization. It should be simpler than synchronization when data already exists there.
Have you tried synchronizing data using ApexSQL Data Diff ?
Another option you can try that doesn't require you to create new organization is reading your SQL Server transaction logs and checking if corrupted data can be found there. If you can retrieve the data then you can just re-create tables with valid data and you'll be all good. Unfortunately this is only possible using 3rd pary tools such as ApexSQL Log
I would recommend looking into the CRM 2011 Instance Adapter
Unlike Scribe, it's free.
Microsoft blog post: http://blogs.msdn.com/b/crm/archive/2012/10/24/the-microsoft-dynamics-crm-2011-instance-adapter-has-released.aspx
PowerObjects wrote an article about it as well:
http://www.powerobjects.com/blog/2012/10/26/introduction-microsoft-dynamics-crm-2011-instance-adapter/
Peter
If you can, export to Excel and import from there.
Advantage: easy, fast, graspable
If you can't, design a console application that connects to the server, queries it, fetches data and shoves it into the other instance.
Advantage: full control, repeatibility, configurability, coolness factor and you get to type some code
This really depends on the scope of your data. Are you talking about millions of records with a huge list of entities or are you talking a couple entities with a thousand or so records?
If it's something small, you could always try exporting via excel and then importing into the new org.
SSIS, CozyRoc or Scribe will do the trick. I'd opt for Scribe and go entity by entity if it is a mission critical situation.
We have an application that requires our customers to have a SQL server instance on site. At their request, the application needs to synchronize the data in their database with a copy in our datacenter.
We're using .Net 3.5 SP1. We need to synchronize the data exactly, including IDENTITY columns.
We'd prefer to use something like LINQ to SQL that would let us make some simple select and insert/update calls against mapped entities. However, the IDENTITY columns seem to be a problem with LINQ and similar approaches.
We can do this all with built-up SQL statements and turn IDENTITY INSERT on / off as needed, but I'd prefer a more elegant solution.
Thanks!
** Edit - We DO need to write our own solution, and we do need to use .Net 3.5 SP1 to do it. I won't spend your time explaining all the reasons why, but please limit suggestions to options within the .Net playground.
Microsoft Sync Framework can be your solution. This is framework description from Microsoft:
Microsoft Sync Framework is a data synchronization platform from Microsoft that can be used to synchronize data across multiple data stores. Sync Framework includes a transport-agnostic architecture, into which data store-specific synchronization providers, modelled on the ADO.NET data provider API, can be plugged in.
Sync Framework is a comprehensive data synchronization solution that enables developers to build solutions that support synchronization of any database, on any data protocol over any network topology. msdn.microsoft.com
For your convinience providing link to good tutorial on the subject
If it is just a couple of tables that need to be synchronized and there is not a lot of data in the tables (now and future) you could develop some sort of bulk copy from your servers and bulk insert routine on the customer's server.
Since you said you can't use SQL Server replication services or SSIS, then perhaps a backup/restore procedure could be written. You could take a scheduled backup of your database and make it available to calling applications which could then copy the backup, restore it to another instance on the customers server, then pull all data you need via any number of methods and it would exist locally on the customers servers.
Beyond that, I think you may be asking for a maintenance and synchronization nightmare if you can't base your solution on tools that are made to do this sort of thing.