I wish to do a PowerBI dashboard using data from IP21 historian.
There is currently no automated export from IP21 and there are lots of tables to update manually.
Thus I am wondering, is it possible to directly use IP21 as the data source in PowerBI ?
Many thanks
Related
Is it possible to easily find BigQuery data sources in Google Data Studio by direct connection from BigQuery tables vs custom query? Currently, you will have to open one by one to see if the connection is a direct one or use a custom query.
I have to deal with about 50+ connections and was wondering if there is a better way to see which one has a direct connection to the BigQuery table and which one has a custom query in use. The goal is to build custom tables for the one where we are using custom queries.
Google Data Studio Data Source Tab:
This is not currently supported.
I need to update a table in snowflake by taking data from oracle database.
Is there a way to connect to oracle database from snowflake?
If answer is NO how can i update the table in snowflake using data from oracle.
Not sure exactly what you are looking for here. The best way to get data into Snowflake is via the COPY INTO command, which would then allow you to update the Snowflake table with that data. If you are looking at ways to keep the 2 systems in-sync, then you might want to look into the various data replication tools that are in the marketplace. If this is a transactional update, then you can use a connector (ODBC, JDBC, Python, etc.) to update the data from one system to another. I wouldn't recommend that for bulk updates, though.
There are several ways you can integrate your data from oracle to snowflake. If you are familiar with ETL tool you can use any one of them or you can use any program language to extract and load.
I have a question regarding incremental refresh from Snowflake to Tableau. I know the feature for Incremental refresh/Incremental extract is available in Tableau but can it be used for incremental loads from Snowflake? And how does it work?
The reason for me asking is because I know that query folding which other BI-tools on the market uses for incremental refreshes, isn't possible in Snowflake.
Thanks!
/P
Tableau incremental refreshes work the same for Snowflake as it does for other databases.
"Query Folding" looks like a Microsoft (and specifically PowerBI) term. According to this article https://exceleratorbi.com.au/how-query-folding-works/ "query folding" is the process of pushing the work load down to the database, which is what Tableau does when querying Snowflake tables directly.
With Snowflake I would recommend querying the tables directly as they are already setup in columnar format, and you can avoid moving the data to a Tableau Server and waiting on refreshes. Snowflake has unlimited storage whereas you might be limited by your Tableau Server.
If you need the tables in Snowflake to only show data as of a point in time, there are different ways you could accomplish this including:
Preset date filters (or parameters as filter within Tableau) that are pushed down to Snowflake
Using Tasks in Snowflake to run at a specific time to:
Clone your tables, and use the clones for reporting
Update existing reporting tables
I agree with Chris' answer accept for avoiding the extracts on Tableau Server. There can be a lot of performance gains had by using Tableau to extract the data. We run extracts out of Snowflake for most of our data sources. We also test both live connections and extracts for each to see which performs best. If timing is an issue, extracts can be set to refresh every 15 minutes at the most.
To get extracts loaded and refreshing use the following steps.
Switch your data source to an extract in Tableau Desktop
This will create a local copy of the data to be used to publish next.
Select Server/Publish Workbook
In the Publish settings, choose your refresh schedule and publish to Tableau Server. The workbook and data source will be loaded to Server.
You can also update the refresh schedules directly in Server by navigating to the new data source and going to the Extract Refreshes tab.
If you don't have the correct schedule available, you can create one in the Schedules menu for the site.
Our team is trying to create an ETL into Redshift to be our data warehouse for some reporting. We are using Microsoft SQL Server and have partitioned out our database into 40+ datasources. We are looking for a way to be able to pipe the data from all of these identical data sources into 1 Redshift DB.
Looking at AWS Glue it doesn't seem possible to achieve this. Since they open up the job script to be edited by developers, I was wondering if anyone else has had experience with looping through multiple databases and transfering the same table into a single data warehouse. We are trying to prevent ourselves from having to create a job for each database... Unless we can programmatically loop through and create multiple jobs for each database.
We've taken a look at DMS as well, which is helpful for getting the schema and current data over to redshift, but it doesn't seem like it would work for the multiple partitioned datasource issue as well.
This sounds like an excellent use-case for Matillion ETL for Redshift.
(Full disclosure: I am the product manager for Matillion ETL for Redshift)
Matillion is an ELT tool - it will Extract data from your (numerous) SQL server databases and Load them, via an efficient Redshift COPY, into some staging tables (which can be stored inside Redshift in the usual way, or can be held on S3 and accessed from Redshift via Spectrum). From there you can add Transformation jobs to clean/filter/join (and much more!) into nice queryable star-schemas for your reporting users.
If the table schemas on your 40+ databases are very similar (your question doesn't clarify how you are breaking your data down into those servers - horizontal or vertical) you can parameterise the connection details in your jobs and use iteration to run them over each source database, either serially or with a level of parallelism.
Pushing down transformations to Redshift works nicely because all of those transformation queries can utilize the power of a massively parallel, scalable compute architecture. Workload Management configuration can be used to ensure ETL and User queries can happen concurrently.
Also, you may have other sources of data you want to mash-up inside your Redshift cluster, and Matillion supports many more - see https://www.matillion.com/etl-for-redshift/integrations/.
You can use AWS DMS for this.
Steps:
set up and configure DMS instance
set up target endpoint for redshift
set up source endpoints for each sql server instance see
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html
set up a task for each sql server source, you can specify the tables
to copy/synchronise and you can use a transformation to specify
which schema name(s) on redshift you want to write to.
You will then have all of the data in identical schemas on redshift.
If you want to query all those together, you can do that by wither running some transformation code inside redsshift to combine and make new tables. Or you may be able to use views.
I'm working on setting up PowerBI desktop for a client where they'll import 8 specific tables from a database to work on in Power BI.
However, when I go to import the tables, every table in the database is shown and the specific tables must be selected. There are hundreds of tables imported, and a lot of scrolling to select the correct tables, which leaves room for confusion and error for the clients.
The client accesses the database using windows authentication, and I can't change their settings on SSMS as that would affect how they access data on other applications.
I was wondering if there was an simpler way to save a query in Power BI the users can quickly select when setting up a new report.
You could always use raw sql instead:
Or select "empty query" and specify it like so:
Edit: not sure if this is of any help but there's also a filter field: