I am actually working on integration of real time data to my forge model. I use Postgres database which is linked to timescale DB. I want to import the data of each element based on its ID in real time as showed in this screen by red.
To sum up, two tasks:
- connecting forge model to my database and import the data of each element
- update the data in real time of each element
In red, the properties that should be connected to my database:
Related
We have an on-premises oracle database installed on a server. We have to create some Charts/Dashboards with Tableau CRM on those data on-premises. Note that, tableau CRM is not Tableau Online, it is a Tableau version for the Salesforce ecosystem.
Tableau CRM has APIs, so we can push data to it or can upload CSV programmatically to it.
So, what can be done are,
Run a nodeJS app on the on-premise server, pull data from Oracle DB, and then push to Tableau CRM via the TCRM API.
Run a nodeJS app on the on-premise server, pull data from Oracle DB, create CSV, push the CSV via TCRM API
I have tested with the 2nd option and it is working fine.
But, you all know, it is not efficient. Because I have to run a cronJob and schedule the process multiple times in a day. I have to query the full table all the time.
I am looking for a better approach. Any other tools/technology you know to have a smooth sync process?
Thanks
The second method you described in the questions is a good solution. However, you can optimize it a bit.
I have to query the full table all the time.
This is can be avoided. If you take a look at the documentation of SObject InsightsExternalData you can see that it has a field by name Operation which takes one of these values Append, Delete, Overwrite, Upsert
what you will have to do is when you push data to Tableau CRM you can use the Append operator and push the records that don't exist in TCRM. That way you only query the delta records from your database. This reduces the size of the CSV you will have to push and since the size is less it takes less time to get uploaded into TCRM.
However, to implement this solution you need two things on the database side.
A unique identifier that uniquely identifies every record in the database
A DateTime field
Once you have these two, you have to write a query that sorts all the records in ascending order of the DateTime field and take only the files that fall below the last UniqueId you pushed into TCRM. That way your result set only contains delta records that you don't have on TCRM. After that you can use the same pipeline you built to push data.
I'm implementing a SSIS package where I want to incrementally load data from SQL Server to D365. From source to staging, we are loading data to tables with 15 minutes frequency. This incremental data is based on the DATELASTMAINT (Last Date Maintenance). We have created a few views on top of these tables. We are loading data from these views into D365 entities.
But in this workflow, we just want to incrementally load data into D365 as it's taking a long time for INSERT and UPDATE. We are using KingswaySoft for D365 data connection.
I tried couple of scenarios to incrementally get data, but couldn't succeed. What is the best way to incrementally fetch data from views (which are based on multiple tables) and push that data into D365?
Is it possible to have a choice to store Database at different locations"?
eg. on Dropbox/Google-Drive or other place and update it in real time
Can a database be selected from a list of Databases to use?
Can MongoDB be stored locally or on a server elsewhere?
I need multi database concept for my project. I create user on real time automatically create new database for that new user .that new user login switched to master database to user database.. how to do this .. I am trying larval 5.4 version .
1.How to create database on real-time for new user ?
2.How to switch users login master database to user database ?
You can create schemas and migrate them real time, I've done exactly this before in a project. Use the following
DB::connection('mydb')->statement(DB::raw('CREATE DATABASE....BLAH BLAH BLAH'))
to execute raw queries for creating new databases. You'll need your master connection to execute these queries.
Then you can use Artisan::call('migrate', ['--database' => 'newdb']) to migrate and optionally seed it.
Finally, you will need to either have a customers entry in your database config file that determines which schema the user connects to, or you can set this real time using Config::set(['my.user.db.config').
My strategy for creating unique databases for the customer was to use a guid as the database name and keep a master log of that in my master database. When a user would log in, their database name would be read from master and then the application would switch his connection on the fly. My company has hundreds of small customer databases with names like 123151-2135151-5132123-545-231231 that keep them unique.
It's quite involved and there are a lot of moving pieces but that will get you started.
I have an Access table with venue information. I'm toying with nearest neighbor stuff to show the website viewer the upcoming event that's nearest to them.
The main, in-house, database is Access but the website pulls the data from MSSQL. What I currently do is maintain the Access database, export the table as Excel 2003, transfer .xls to the web server, delete the table and importing the .xls (within SQL Management Studio).
It is a laborious process and I just realized that, when I import the .xls, I will have to go through and reset all the spatial information (set a primary key, set the data type for lat/lng to geography, give it a spatial index).
Is there a way to automate this process? Is there a way to set the data types and keys during the import process? Obviously, the right thing to do is use MSSQL as the back-end and forgo all the work. Unfortunately, my superiors haven't been receptive to making the change.
I found some information on a page working with basic spatial data that shows you can have lat/lng as separate float fields and/or a POINT(lng,lat) geography field.
With this being said, I could just have a separate lat/lng fields in Access and they should import into MSSQL. By that page, I don't even require a geography field (float will work with some extra code. Assumingly, I should be able to populate the geography field from the float fields with a query.