Is it possible to run a SQL query from AWS SystemsManager with aws:runPowerShellScript" as a document.
My Scenario:
We have 100's of SQL Servers. I have to read 1 particular table in all 100+ MS SQL servers then I have to generate one output file which holds data from all the server reads.
Please guide me with solutions. Thanks in advance!!
Related
I want to copy tables in various schemas from SQL Server to snowflake. I understand that snowflake COPY works well to load huge amount of data into snowflake, provided I have CSV data as input.
However, I am unable to figure out an efficient way to export SQL Server data in CSV format. I went through some of the threads in this forum on this topic and found that PowerShell tool export-csv is a good option. But does it work well with thousands of tables in SQL Server?
If not, what other option should I try to move the data from SQL Server to snowflake? Please note that this is not a one time data load. I am looking for a process that can run daily to load data from SQL Server to snowflake.
Thanks in advance!
P.S: I tried the SQL Server bcp tool. But it doesn't generate a standardized CSV file.
I'm trying to query my Hortonworks cluster Hive tables from SQL Server. My scenario below:
HDP 2.6, Ambari, HiveServer2
SQL Server 2016 Enterprise
Kerberos configuration for secure logins in HDP
I was reading about the PolyBase service in SQL Server 2016 and I suppose later versions. However, I realize that according to the documentation what this service is going to perform in SQL Server is a bridge to reach out my HDFS and recreate external tables based in this data source.
Otherwise what I'm expecting is query Hive objects like these would be SQL Server objects as well, such as a linked server.
Someone has an example or knows if this is possible within SQL Server and Hive?
Thanks so much
Hive acts more as a job compiler than a database. This means every SQL statement you are writing will be translated into a job for Hadoop, sent over to the cluster and become executed there. From the user perspective it looks like querying a table.
The already mentioned approach by reading the HDFS data source and re-create it in SQL Server is the correct one. Since both, Hive and database server are different technologies, something like a linked server seems to technically not possible for me.
Hive provides nowadays a JDBC interface which could be used to connect to it. But even with Hive JDBC, every query will end up as cluster job for distributed computing, running over the files in HDFS, create a result set and present that to you.
If you want to querying Hive from SQL server, you can download ODBC driver (Microsoft or Hortonsworks) and create a Data Source Name (DSN) for Hive. In Advanced option check Use Native Query. Then just create new linked server in the SQL server with the same name of datasource as Data Source Name in ODBC driver.
Write openquery something like:
select top 100 * from
openquery(HadoopLinkedServer,
'column1, column2 from databaseInHadoop.tableInHadoop')
I am trying to find a way to, as close to real time as possible, have Informix 11.50.FC9GE database data available for SQL Server 2014 SSRS reports.
Right now, we have SSIS (Integration Packages) that are on a 4 hour schedule to go out to our 8 Informix databases via ODBC, gather all of their table data, and update tables on the SQL Server side.
So, table 'abc' exists on all 8 databases. All of that data is input into a single table on the SQL Server. As that data is gathered, an artificial column is created to say which database the data came from.
Select *, "250" as db from abc
This process takes about 1-2 hours to complete. If someone attempts to run a report during this time, they get skewed data.
My hope is to have all of the table data in the SQL Server, and only pass over changed data.
I was looking at SQL Server Replication, but it doesn't look like it can replicate from a Non-SQL Server database?
I also started looking at IBM InfoSphere Change Data Capture 6.5. I installed the Access Server and Management Console on a Windows server with one of my Informix databases.
I installed InfoSphere CDC Configuration Tool (Instance) on the SQL Server with the Database entries pointing to the Informix server, but when I try to start that Instance I get the error:
IBM InfoSphere Change Data Capture could not identify a supported default database encoding. The detected encoding is null. Please override the encoding with a supported IANA encoding name that matches or is very close to your default database encoding and restart IBM InfoSphere Change Data Capture. Use dmset command line utility to override the encoding.
I found this command to enter:
dmset -I instanceName database_default_character_encoding=UTF-8
But that gives an error:
C:\Program Files (x86)\IBM\InfoSphere Change Data Capture\Replication Engine for
IBM Informix Dynamic Server\bin>dmset -I vsqldev2014 database_default_character
_encoding=UTF-8
There is a problem with the IBM InfoSphere Change Data Capture service.
Frankly, I probably didn't set it up right, because there's hardly any instructions out there. :(
I did find a 3rd party software that appears to work, but they are quoting tens of thousands of dollars. No way, my company would go for that.
Any help/suggestions?
Is there any way to transfer the large volume of data from Azure SQL to on-premises SQL Server 2016 Enterprise/Standard? The requirements prescribed as follows:
Weekly full database transfer
Daily delta transfer before midnight
I read about SSIS for Azure Blob Storage but am not sure whether it is applicable to this context.
Updated: I found an article on Azure Data Sync; according to that article, it seems doable. Please share your experiences. That would be extremely helpful.
https://www.mssqltips.com/sqlservertip/3062/understanding-sql-data-sync-for-sql-server/
Weekly full database transfer
SSIS Doesn't provide a way to do Full transfer of data(i mean backup),unless you want to truncate and insert from source..
For Weekly full database transfer,i would go with SQLAzure Export/Import functionality
Refer below links for more details..
1.https://github.com/richorama/SQLDatabaseBackup
2.I need to automate SQL Azure database backup in SQL Script files. How can i do so?
Daily delta transfer before midnight
You will need a way to identify delta..so create a table with all table names and last run time
create a console application which uses bulk insert functionality,which uses above table as base and insert in onpremises
I want to do simple transformation with source as Flat file and target as SQL Data Warehouse.In target Analyser using ODBC Connection I am able to connect the SQL Data Warehouse and connection is successful,but no tables are listed and I am unable to select a table to import.(Sample Tables are created in SQL Data warehouse)
Whether it is possible to create a Source/target as SQL Data warehouse?If so kindly help me to solve the issue.
Thanks & Regards
Prakash
Informatica does offer support for Azure SQL Data Warehouse and are listed as a Partner solution https://azure.microsoft.com/en-gb/documentation/articles/sql-data-warehouse-integrate-solution-partners/
Here is a link to all the configuration required for Informatica Cloud for example: https://kb.informatica.com/proddocs/Product%20Documentation/5/IC_Winter2016_MicrosoftAzureSQLDataWarehouseConnectorGuide_en.pdf
What version are you using Prakash?
If you are trying to select data from SQLDW I would also check that your connection is targeting the specific database you have created. You could be connecting to the master db on the logical server rather than the sqldw database itself.