I have master SQL server with DB Central and a lot of satellite SQL servers with DB Client. I need to collect data from log tables(LogTable) on Client(each client has own ID in log table) to one big table on Central(LogTableCentral).
Data must go only from Client to Central
On each Client I want to have only data for this Client
I need solution with minimal amount of work on client side because of count of clients
Central is MS SQL server Enterprise, Clients are MS SQL server 2005, 2008
Thanks a lot
EDIT: data can be collected periodically(for example: every day at 01:00)
As it is done periodically, have you considered using SSIS for this task?
You can add multiple data connections, and then have a series of data sources (each connected to one of the connections) feeding into a data destination (also connected to one of the data sources)
You could then schedule this as part of a SQL Agent Job. Something like this:
Related
For my .NET Application I have an Excel file that is used as the data source. This Excel file is updated every hour. My current back-end database technology is MS Access. The Access file has a linked table to the mentioned Excel file. I would like to switch over my back-end to SQL Server.
My question is how can I create a linked table in SQL Server such that I always have the current data in my SQL Server database. Right now I have managed to import the Excel sheet through the import wizard in SQL Server Mgmt. Studio. But obviously this is static (not updated hourly like the excel sheet).
Can I create some sort of procedure that runs every hour, reads the Excel sheet and import to my SQL Server database? How would I go about doing that? Or any other method as long as my table in SQL Server gets updated on an hourly basis with data from the Excel sheet.
One, do not code read "in process" from the excel file. That is a disaster waiting to happen. Some user opens the file, locks it, and now your sql-server starts failing. :) That is a bad scenario.
You should create an ETL package to run (hourly)?
https://learn.microsoft.com/en-us/sql/integration-services/ssis-how-to-create-an-etl-package?view=sql-server-ver15
What is SQL Server Integration Services (SSIS)? MicrosoftSQL Server
Integration Services (SSIS) is a platform for building
high-performance data integration solutions, including extraction,
transformation, and load (ETL) packages for data warehousing. SSIS
includes graphical tools and wizards for building and debugging
packages; tasks for performing workflow functions such as FTP
operations, executing SQL statements, and sending e-mail messages;
data sources and destinations for extracting and loading data;
transformations for cleaning, aggregating, merging, and copying data;
a management database, SSISDB, for administering package execution and
storage; and application programming interfaces (APIs) for programming
the Integration Services object model.
data sources and destinations for extracting and loading data;
Deploy and Schedule:
https://www.mssqltips.com/sqlservertutorial/9069/deploy-and-schedule-an-sql-server-integration-services-ssis-package-step-by-step/
Scheduling the SSIS Package with SQL Server Agent
Manually executing packages is one thing, but normally you will schedule packages so your ETL can run in a specific time windows
(probably at night). The easiest option is SQL Server Agent. You can
right-click on the Jobs node to create a new job:
I'm trying to query my Hortonworks cluster Hive tables from SQL Server. My scenario below:
HDP 2.6, Ambari, HiveServer2
SQL Server 2016 Enterprise
Kerberos configuration for secure logins in HDP
I was reading about the PolyBase service in SQL Server 2016 and I suppose later versions. However, I realize that according to the documentation what this service is going to perform in SQL Server is a bridge to reach out my HDFS and recreate external tables based in this data source.
Otherwise what I'm expecting is query Hive objects like these would be SQL Server objects as well, such as a linked server.
Someone has an example or knows if this is possible within SQL Server and Hive?
Thanks so much
Hive acts more as a job compiler than a database. This means every SQL statement you are writing will be translated into a job for Hadoop, sent over to the cluster and become executed there. From the user perspective it looks like querying a table.
The already mentioned approach by reading the HDFS data source and re-create it in SQL Server is the correct one. Since both, Hive and database server are different technologies, something like a linked server seems to technically not possible for me.
Hive provides nowadays a JDBC interface which could be used to connect to it. But even with Hive JDBC, every query will end up as cluster job for distributed computing, running over the files in HDFS, create a result set and present that to you.
If you want to querying Hive from SQL server, you can download ODBC driver (Microsoft or Hortonsworks) and create a Data Source Name (DSN) for Hive. In Advanced option check Use Native Query. Then just create new linked server in the SQL server with the same name of datasource as Data Source Name in ODBC driver.
Write openquery something like:
select top 100 * from
openquery(HadoopLinkedServer,
'column1, column2 from databaseInHadoop.tableInHadoop')
We have two databases, one at our client's site and another (Main) in our data centre.
Both databases have the same schema, we want to sync the databases periodically (twice a day) so that both should have the same data.
We are you using SQL Server 2008 R2.
Please suggest a good methodology.
The number of rows modified is approx to 400 rows a day
You can create a Job that selects from the client's database tables and inserts in your database tables.
If both the databases are in different servers you can create a linked server at your server and connect it to the client server.
Then you can schedule the job to run whenever you want.
I have a host on a server and that contains an SQL Server Database.
I have another server in another country and i want have a backup from the database every 5 minutes or after each transaction only insert new row to another database.
After some research i found out i can use linkedservers for this goal.
Is this procedure works for me for doing this operation?
I don't know what the linkedserver will do for you.
You are connected from both server via a vpn?
You are in different network (domain) probably?
If you are using a linked server, it means you will probably create trigger or stored proc. You will have to configure msdtc (for trigger).
You can use :
Replication
Log shipping
Custom replication process
I had to configure 2 times a replications to move the data from a server to another, than manage these data with trigger. It was easier to work on the data localy
my client has SQL Server with some customer info and I am developing e-shop using MySQL database. What we need is to keep the database of customer loyalty points synchronized. When customer buys a product in an ordinary shop (not e-shop) these are recorded on SQL Server (via some accounting app). The problem is that I need this information to get to the MySQL server which stores information for the e-shop application, so the amount of loyalty points gets sync'ed on both servers.
Is there any way how can I send http and/or xml/rpc request from SQL Server via either trigger or stored procedure (I suppose trigger can trigger a stored procedure, so either of these is fine)?
Is it essential to interface with the MySQL DB via xml/rpc?
I might try to accomplish this by linking the MySQL DB to the MSSQL DB... the process will be version dependent so your MSSQL version is helpful...
But this site:
http://www.infi.nl/blog/view/id/4/How_To_MySQL_as_a_linked_server_in_MS_SQL_Server
may be a good start for you
EDIT: And here, for MSSQL2008
http://dbperf.wordpress.com/2010/07/22/link-mysql-to-ms-sql-server2008/