Is there an existing way to get a message from RabbitMQ and to use it to create an SQL query such as an "INSERT INTO"? For example, take the data that a webservice retrieves to a client (that data must go through Rabbit as well) and insert into a MS-SQL Server the data that the client is supposed to get.
Pretty much any language can talk to RabbitMQ. There is also a powershell module built for it.
For your needs, I've found the following link which demonstrates how you can set up rabbitmq with sql server.
I would suggest creating a sandbox example on your local machine using the link as a guide so that you can get familiar with RabbitMQ and its setup.
Hopefully that will provide you with the knowledge you need to solve your current problem.
Related
I am trying to make ARX connect to a SQL Server database with hostname\instance and Windows auth. ARX keeps reporting "Unknown server host name". I need input for what ARX expects in "Server" when there is also an instance name.
The reason for looking at ARX, is because I am searching for a tool, which is capable of anonymizing a SQL Server database.
Connect to a SQL Server database, import the data and pattern, anonymize the data, overwrite the data in the database with the anonymized data.
Author of ARX here. Some feedback:
ARX is relased in two forms: a Java programming library as well
as a GUI. There is more flexibility when using the programming
library than with the GUI. You will be able to (in theory)
import the data from a SQL Server DBMS, but you will not be able to
write data back to the DBMS. This functionality is easy to implement using the library,
however.
We have been struggling with DBMS connection problems for quite some time. The reason is that we don't have that many different types of DBMSs around to be able to test wide range of possible connection settings. For example, I don't have SQL Server installation around, so I cannot investigate your problem.
That said, please report this issue on our GitHub issue tracker (https://github.com/arx-deidentifier/arx/issues). If you are able to run ARX from source, we would be happy to help to debug your problem. An issue requesting an export feature to DBMSs has already been created (https://github.com/arx-deidentifier/arx/issues/332). If you want, you can upvote it by commenting.
I am trying to connect a Nifi QueryDatabaseTable processor to Salesforce in order to retrieve data to load into a Marklogic data hub. I found a couple of shareware JDBC drivers for Salesforce but have not been able to successfully connect to the SF Connected App endpoint I've been given. Can anyone advise whether this is a good approach for automated data retrieval from SF? Should I be able connect to a SF Connected App URL via JDBC driver like the one at ascendix?
TIA!
I ended up finding a free Salesforce JDBC driver (not the ascendix driver, which did not work for me) that seems to work fine for simple select queries to pull data from a Salesforce Connected App. So, the JDBC approach for this solution works using a Nifi QueryDatabaseTable processor which produces an Avro dataset that is easily split into records and converted to JSON for ingest to MarkLogic.
Yes, the RelierSoft driver works for me, though the 0.4 version (Oct 2019) of the JAR is packaged with an old version of the apache commons.csv library that causes errors for some latest NiFi services, like CSVReader. So, I repackaged it without the old commons.csv library classes to make that service work for me.
Is it possible to implement a service broker to receive messages from SQL Server in a language like Python, or does it always have to be in some .Net language?
Yes it is possible (but not much fun)
Service Broker is configured and used via T-SQL commands, so if you have a way to talk to SQL Server from python (e.g. using the excellent pyodbc) then you just have to connect and issue the appropriate queries. Figuring out the appropriate T-SQL and getting Service Broker to work for you is the hard part, not the python bit.
import pyodbc
conn = pyodbc.connect(your_connection_string)
conn.autocommit = True
conn.execute("ALTER DATABASE CURRENT SET NEW_BROKER WITH ROLLBACK IMMEDIATE")
# ... etc ...
The rest of the (many hundreds of lines of) SQL is an exercise in understanding Service Broker, for which there is much MSDN documentation, such as the Developer's Guide for SQL Server 2008 R2.
The community tutorials are very useful for figuring out stuff that Microsoft doesn't tell you e.g. these which I found helpful enough to get a proof-of-concept running before I gave it up :)
http://rusanu.com/2006/04/06/fire-and-forget-good-for-the-military-but-not-for-service-broker-conversations/
http://www.davewentzel.com/content/service-broker-demystified-fire-and-forget-anti-pattern
http://www.davewentzel.com/content/service-broker-demystified-default-not-default
I have a JSON file of data which I have pulled from an API and I would very much like to just dump this data into an SQL Server.
The reason it's SQL Server specifically is that the database is already in place for the current project. I have spent time googling this and searching on here but was unable to find anything useful thus far. I'm familiar with Python but I'm open to any solution.
TLDR: I'm interested in which languages and packages provide easy solutions to automate JSON to an SQL Server table, do you have any suggestions or know of any packages that already achieve this?
You can use something like SSIS to accomplish this (you may already have it) by writing a script task. This could do custom parsing then load it into the correct table. This can be easily automated. I mention SSIS because it's very easy to add future tasks to this, if you're ever required.
Alternatively you could create a script outside of the database (ie. Python) that parses the JSON, connects to the database through ODBC/OLEDB and writes the records. This can be automated using Task Scheduler or something similar. An example implementation of this could use PYODBC.
you can use WCF Web Service, sending json data to SQL Server .
refer the links below hope it will be helpful for you
http://www.codeproject.com/Articles/167159/How-to-create-a-JSON-WCF-RESTful-Service-in-sec
http://mikesknowledgebase.azurewebsites.net/pages/Services/WebServices-Page2.htm
you cant directly fetch json data's in sql server instead you can use wcf service
I need to be able to extract and transform data from a data source on a client machine and ship it off via a web service call to be loaded into our data store. I would love to be able leverage SSIS but the Sql Server licensing agreement is preventing me from installing Integration Services on a client machine. Can I just provide the client copies of the Integration Services' assemblies to be referenced by my app? Does anyone have any ideas on how to best implement a solution to this problem apart from building a custom solution from the ground up? Ideally the solution would include leveraging an existing ETL tool?
Thanks for your suggestions.
If you are providing your client with a service around their data, you should develop a standard that they need to deliver their data in, and negotiate a delivery method for that file well before you ever consider what to do with SSIS. Since from comments it appears that your data is on a machine in a client's remote location, the most common method I have seen is either having the client SecureFTP a file into your network for processing, or to have a job on your end that gets the file using SecureFTP. Once you have the file on your network, writing the SSIS to process it is trivial.
If the server can reach out to the client machine, then you can just run the SSIS package on the server. What kind of data are you moving? If it's a flat file, you could FTP it to the server.
Another way to go about this is to use BCP. I'm not a big fan of this approach (SSIS is much faster, more robust, etc), but it can work in a pinch.
http://msdn.microsoft.com/en-us/library/ms162802.aspx