snowflake drivers for sql server - snowflake-cloud-data-platform

as a newbie to snowflake, with a potential project for snowflake/sql server integration (for SSAS), I'd like to know whether there are some mature ODBC or OLE drivers for ingesting data from Snowflake into SSAS, or whether dumping data to text files is the only high-speed/volume data mechanism to import to SSAS, particularly for older version, such as SSAS 2014 (while considering upgrade paths).
thank you kindly,
Cos

Yes, there are odbc drivers but odbc is not a good solution (for any source/target) for high volume data movement. Whether it would be a workable solution for you is dependent on your data volumes, required performance, etc.

Related

Convert SQL Server queries to Postgres on the fly

I have a scenario where I get queries on a webservice that need to be executed on a database.
The source for these queries is from a physical device so I cant really change the input to my queries.
I get the queries from the device in MSSQL. Earlier the backend was in SQL Server, so things were pretty straight forward. Queries would come in and get executed as is on the DB.
Now we have migrated to Postgres and we don't have to the option to modify the input data (SQL queries).
What I want to know is. Is there any library that will do this SQL Server/T-SQL translation for me so I can run the SQL Server queries through this and execute the resulting Postgres query on the database. I searched a lot but couldn't find much that would do this. (There are libraries that convert schema from one to another but what I need is to be able to translate SQL Server queries to Postgres on the fly)
I understand there are quite a bit of nuances that will be different between SQL and postgres so a translator will be needed in between. I am open to libraries in any language(that preferably runs on linux : ) ) or if you have any other suggestions on how to go about this would also be welcome.
Thanks!
If I were in your position I would have a look on upgrading your SQL Sever to 2019 ASAP (as of today, you can find on Twitter that the officially supported production ready version is available on request). Then have a look on the Polybase feature they (re)introduced in this version. In short words it allows you to connect your MSSQL instance to other data source (like Postgres) and query the data in as they would be "normal" SQL Server DB (via T-SQL) then in the background your queries will be transformed into the native pgsql and consumed from your real source.
There is not much resources on this product (as 2019 version) yet, but it seems to be one of the most powerful features coming with this release.
This is what BOL is saying about it (unfortunately, it mostly covers the old 2016 version).
There is an excellent, yet very short presentation by Bob Ward (
Principal Architect # Microsoft) he did during SQL Bits 2019 on this topic.
The only thing I can think of that might be worth trying is SQL::Translator. It's a set of Perl modules that have been around for ages but seem to be still maintained. Whether it does what you want will depend on how detailed those queries are.
The no-brainer solution is to keep a SQL Server Express in place and introduce Triggers that call out to the Postgres database.
If this is too heavy, you can look at creating a Tabular Data Stream (TDS is SQL Server network transport) gateway with limited functionality and map each possible incoming query with any parameters to a static Postgres query. This limits any testing to a finite, small, number of cases.
This way, there is no SQL Server, and you have more control than with the trigger option.
If your terminals have a limited dialect demand then this may be practical. Attempting a general translation is very likely to be worth more than the devices cost to replace (unless you have zillions already deployed).
There is an open implementation FreeTDS that you could use if you are happy with C or Java.

Lightweight ETL or database Sync - Sybase to SQL Server

I have been doing some investigations into some light weight database Syncing tools to trial. The initial task we want to perform is a simple data sync from a few tables on a Sybase ASE database (15) to a SQL Server database (2008 R2). Timing wise, I'd like to keep my options open, but ultimately, I would like to have the ability to sync every minute or less.
I have been looking at SymmetricDS, which at face value seems to do exactly what I want it to. The drama is I have hit a couple of roadblocks on the Sybase side of things, which is proving to be very frustrating (Jumpmind support are assisting). It appears that Java has a problem with the default collation we have on our server, being HP-roman8. Unfortunately, to change this charset is way bigger then this project itself.
I have also started investigating Talend, but have hit a few roadblocks in relation to requiring older versions of drivers for Sybase and downgrading the installed version of Java.
Without having to go to Replication Server, does anyone have any suggestions on a relatively lightweight ETL or database Syncing tool that will do what I want? The biggest gotcha thus far is Sybase support - I really need something that will seamlessly work without having to hack too much.
Cheers
You should try uniVocity. It is a Java-based ETL framework that certainly can help you do what you need. You can use any JDBC driver, define your mappings with a few lines of code and have this working faster than a traditional ETL tool.
Have a read through its tutorial and also check out a few sample projects here
Disclosure: I am the author of this library. It's open-source and free (Apache V2.0 license).

How to convert .gdb database to SQL Server

I have a .GDB database (old one) and the data in it is very important
I need to convert that .gdb database to a SQL Server database - can anyone help me...
Create connections to both source GDB and Destination SQL Server in ArcCatalog. Copy everything from source and paste it into the destination. You won't be able to do it with SQL tools alone.
Lacking ESRI software, for simple cases, my workflow is to use the GDAL C++ API to read the GDB. This requires the GDAL File GDB driver. Then I will use Microsoft.SqlServer.Types to transfer to SQL Server. This involves low-level APIs and you need to understand the spatial types in the respective libraries. It gets complex if you have polygons with rings, for example.
I'm not aware of a tool that will automatically convert between these database types. You'll need to use an application that can read the old database type (Firebase), learn the table design, create a similar table design in SQL Server, and use the application to load the data from Firebase to SQL Server.
Typically, this kind of work is called ETL (Extract/Transform/Load) and is done with migration tools like SQL Server Integration Service (SSIS). SSIS is free with SQL Server, and there are a lot of books available on how to use it - but like learning to develop software, this isn't a small task.
The easiest way to export Esri File Geodatabase FGDB (.gdb) data to MS SQL Server is with ArcGIS for Desktop at the Standard or Advanced level.
You may also want to try exporting to shapefile (SHP) format (an open transitional format) then import to your MS SQL Server. I've seen a tool online that has worked for me called Shape2SQL.
Esri also has an open File Geodatabase API that you can use to write your own too.
I highly recommend FME Workbench for GIS data conversion. It's like SQL Server Integration Services (ETL) but for GIS. Graphical interface, connect data readers with data writes, insert transforms, run them, etc.

Converting SQL Server 2012 schema to PostgreSQL 9.1

For a new project we have to export data from a SQL Server 2012 database to a PostgreSQL database. We have the SQL Server schema but have to create one for PostgreSQL. As far as possible we would like the schemas to match. Can anyone give any advice on the best way of converting a SQL Server schema to a PostgreSQL one? Are there any tools or scripts which will help? I have seen a PostgreSQL function but to be honest I have no PostgreSQL experience and our remit stops at the data being imported into PostgreSQL so I would like to do everything from the SQL Server side (planning to use SSIS with the 64-bit ODBC driver for PostgreSQL to export the data once we have the schema created)
Although not free, I've used Toad Data Modeler for this in the past. We never used it on any particularly complex schemas, but it did do a good job of keeping schemas in sync between various DB platforms.
Your mileage may vary, but it's worth a look.
I don't know a direct schema converter but most data modeling tools offer such conversion functionality. We use Dezign for Databases. This tool has got a function "switch target dbms". This a data modeling tool just like Toad Data Modeler mentioned here before. With the database independent modeling functionality you can keep schemas on different db platforms in sync. For data synchronization (data pump) between different database platforms you can use DataDiff CrossDB.

Replication across heterogenous databases

Is it possible using SQL Server Replication to replicate data to AND from (bi-directional) Oracle and SQL Server? The schemas are completely different. In real-time would be a bonus.
Have already investigated Oracle Golden Gate, which seemed to do the job, although the licence cost is not insignificant!
I wondered if anyone has had any experience in replicating data across different schemas, and what other tools they employed? I realise this is a bit of an open-ended question but any advice and previous experiences would be most useful.
Thanks
Duncan
I recently had to create a solution to import periodically lots of data from different databases (most of the time from Oracle databases) to a SQL Server database (a data warehouse). To do so, I used SQL Server Integration Service to create a package able to import, transform and insert the data as I wanted (since it was from heterogeneous sources too). This software comes with SQL Server and the version 2005 and superior is really easy to use (graphical programming). In your case, you could trigger your created services when needed. I am not sure it is the best solution since you would need to create a SSIS service for each direction (from Oracle to SQL Server and from SQL Server to Oracle).

Resources