DOes anyone know how to set up 1 single metadata store for multiple Airflow Servers?
I am talking about something here - https://www.astronomer.io/guides/airflow-database/
But with the intent to run multiple airflow instances which can use single server
Related
I started a MongoDB server with a default configuration, without replSet value, and it is acting as a standalone server. I have created other MongoDB servers with replSet:rs0 which I want to add to an existing server for replication, without bringing the existing server down.
Definitely, there's a way to take down the existing MongoDB server and start with the flag replSet:rs0 but there's a huge set of external tasks which are writing to the standalone server and I don't want to take the standalone server down.
Is there a way I can update the standalone server from mongo shell so that it's converted to a replica set?
I've currently got an ETL process that dynamically builds and executes sql jobs based on job steps that are saved in my database. Included in these jobs are steps to call SSIS packages that move data from one server to another and/or call stored procs on target servers to do further processing. I'm looking at what it would take to migrate our process from SQL Server to a Azure Managed Instance. One of the specific things I'm looking at is the feasibility of replacing the steps that call the SSIS packages with steps that execute Azure Data Factory pipelines or other ADF actions that accomplish the same results. So far I have not run across any examples of this. Anyone have any experience with accessing Data Factory functionality with SQL Agent jobs?
You can run powershell scripts via SQL agent as mentioned in below MSFT docs:
https://learn.microsoft.com/en-us/sql/powershell/run-windows-powershell-steps-in-sql-server-agent?view=sql-server-ver16
And via powershell and ADF REST APIS, you can trigger the ADF pipelines
I have a service on a remote server (server1) that uses a MongoDB database. Now i want to copy that service and the database to a second server (server2) to have a failsafe in case server1 has any problems.
How would i go about synchronizing the database so it's always the same on both servers and so that when server1 goes up after a problem it migrates any changes made to server2's database to it's own?
You can create a replica set of your Server 1. MongoDB provides this feature.
There is an extensive explanation on MongoDB Replication in Mongo's Official Documentation.
I have installed windows connector for a database(V 4.1.1) from http://googlegsa.github.io/adaptor/index.html.
I want to use 11 SQL statements to crawl by using this connector. When I deployed the connector, I filled one form which looks like this:
There is only one option to add an SQL query.
Do I have to installed 11 instances of this connector on my server in different directories to use every SQL statement one by one or Can I use multiple SQL statements to crawl in a single installation?
There is only support for one SQL statement per connector instance. You would need to modify the original connector if you wish to do this in one. Alternatively, you can deploy 11 instances of the connector.
Based on our requirement we need to have one database per account as we cannot have a single database. So we were thinking about multiple database on the SQL Server and based on my research we can have appx 32,000 database on single database instance but the resource limit will be reached far sooner.
How many is too many databases on SQL Server?
Another point was to use no-sql database like MongoDB. As far as I know we can create database per account with different database name.
I wanted to check reference to resources used by MongoDB, do I need more ram, processor and lots of servers for the architecture.
We are looking to have about 10,000 databases max on a server. Can that work on the a single mongodb server?
If you do have similar architecture currently running, please do share.