I have a requirement to import data to Snowflake from SAP BO (WEBI) system.
I know the other way round can be done that is getting Snowflake data to SAP BO. But how do I connect the SAP BO system to Snowflake to get data into Snowflake.
Thank You!
There are 2 basic options:
Use an ETL tool/process - but recommendations for tools are out of scope for this forum
Copy the data to a Snowflake-supported cloud file systems (e.g. AWS S3) and use COPY INTO to load the data from there to Snowflake
However, my understanding is that SAP BO is a data visualisation tool and as such it connects to data sources to visualise the data those sources contain; therefore SAP BO doesn't hold any data - so I'm not sure what data you're trying to load into Snowflake from SAP BO. Are you trying to load data that is held in a data source that SAP BO is connecting to - in which case you need to get that data from that source and not via SAP BO?
Related
I need to design a scalable database architecture in order to store all the data coming from flat files - CSV, html etc. These files come from elastic search. most of the scripts are created in python. This data architecture should be able to automate most of the daily manual processing performed using excel, csv, html and all the data will be retrieved from this database instead of relying on populating within csv, html.
Database requirements:
Database must have a better performance to retrieve data on day to day basis and it will be queried by multiple teams.
ER model, schema will be developed for the data with logical relationship.
The database can be within cloud storage.
The database must be highly available and should be able to retrieve data faster.
This database will be utilized to create multiple dashboards.
The ETL jobs will be responsible for storing data in the database.
There will be many reads from the database and multiple writes each day with lots of data coming from Elastic Search and some of the cloud tools.
I am considering RDS, Azure SQL, DynamoDB, Postgres or Google Cloud. I would want to know which database engine would be a better solution considering these requirements. I also want to know how ETL process should be designed- lambda or kappa architecture.
To store the relational data like CSV and excel files, you can use relational database. For flat files like HTML, which doesn't required to be queried, you can simply use Storage account in any cloud service provider, for example Azure.
Azure SQL Database is a fully managed platform as a service (PaaS) database engine that handles most of the database management functions such as upgrading, patching, backups, and monitoring without user involvement. Azure SQL Database is always running on the latest stable version of the SQL Server database engine and patched OS with 99.99% availability. You can restore the database at any point of time. This should be the best choice to store relational data and perform SQL query.
Azure Blob Storage is Microsoft's object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Your HTML files can be stored here.
The ETL jobs can be performed using Azure Data Factory (ADF). It allows you to connect almost any data source (including outside Azure) to transform the stored dataset and store it into desired destination. Data flow transformation in ADF is capable to perform all the ETL related tasks.
We choosed Snowflake as our DWH and we would like to connect different data sources like (Salesforce, Hubspot and Zendesk).
Is there a way to extract data from these sources and store them in Snowflake in a staging schema without having to store the data in cloud storage like S3 then reading the data into Snowflake?
Many thanks in advance.
You can use any of the connectors Snowflake provide (odbc, jdbc, python, etc) and any tool that can use one of these connectors. However they wont perform well compared to the COPY INTO approach that is optimised for bulk loading.
There are ETL tools, such as Matillion, that use the stage/copy into approach but do it in the background so that it appears that you are loading directly into Snowflake.
Our organization uses Elastic Logstash & Kibana (ELK) and we use a SQL Server data warehouse for analysis and reporting. There are some data items from ELK that we want to copy into the data warehouse. I have found many websites describing how to load SQL Server data into ELK. However, we need to go in the other direction. How can I transfer data from ELK to SQL Server, preferably using SSIS?
I have implemented a similar solution in python, where we are ingesting data from elastic cluster into our sql dwh. You can import Elasticsearch package for python which allows you to do that.
you can find more information here
https://elasticsearch-py.readthedocs.io/en/master/
I use several Microsoft Access databases on a regular basis to create reports. To get the source data, I currently have to log in to SAP BW (via SAP NetWeaver), run the source data report, export the results as a .csv file (but actually saving it as a .txt file), and then import that file into Microsoft Access. Is there a way that I can have Access pull the data from SAP BW directly?
Any help is appreciated!
All of the databases used by SAP are industry standard databases and the data is thus going to be stored in a system that supports ODBC.
As far as I know, SAP in general uses Sybase which is also what SQL server was originally based on.
So SAP is running on an industry standard SQL server (Sybase or SQL server). If running on IBM, then the data is in DB2 (often the as400 system).
You thus simply need to contact your IT department and obtain the required ODBC connection strings to the database. You “might” also need to install the latest Sybase drivers if you not running SAP on SQL server but again such information would be available from your SAP support folks.
So you simply setup linked tables in access to the SAP database, and thus no export or download or importing of data is required – you be reporting on live data at all times. The “challenge” is thus of course to grasp the table structures in SAP - a LARGE challenge since in most cases a report you been using for exports is the result of MANY related tables joined together into a "easy" view for exporting. So be prepared for some complex quires to get the data the way you want.
I am trying to create SSIS package for loading the data from SAP BW to SQL Server. But after read through many books people are saying we need Data Provider to load the data from SAP BW to SQl Server. Can some suggest me where can i get these Providers for ssis.
A good article on SSIS with SAP Connectors directly from Microsoft:
http://social.technet.microsoft.com/wiki/contents/articles/ssis-with-sap-connectors.aspx
Generally useful page, you can find a SAP connection manager for download here: http://ssisctc.codeplex.com/
You should probably bookmark the second link as it has a wide range of SSIS tools that will definitely be useful for you.