Reading from CosmosDB and write to Azure SQL - sql-server

I have an Azure function to read from Cosmos DB and write to SQL. Since I am new to coding I have a little of struggle to understand how to read the incoming document. I can see that documents are shown at input:
public static async Task Run([CosmosDBTrigger(
databaseName: "ToDoList",
collectionName: "Items",
ConnectionStringSetting = "CosmosDB",
LeaseCollectionName = "leases")]IReadOnlyList<Document> input, ILogger log)
{
if (input != null && input.Count > 0)
{ }
I know that I have to read the document and deserialise it to a C# object which I have this code for (assuming it is correct):
Record resultRecord = JsonConvert.DeserializeObject<Record>(jsonString);
I am lost how to get the data from the json document and write it to the C# object. The connecting part is confusing for me.
I also have a SQL code, and again I dont understand how I should connect my C# object so the data can be read and written to SQL database.
var cnnString = "sqlConnection"; // Connecting to Azure SQL Database
using (var sqlConnection = new SqlConnection(cnnString)) // Start up sql connectin with connectionstring
{
sqlConnection.Open();
var cmd = new SqlCommand
{
//Insert into command (used to insert data into a table)
CommandText = #"insert into [dbo].[Player] ([User] values(#User)",
CommandType = CommandType.Text,
Connection = sqlConnection,
};
var record = new Record();
//set parameters
cmd.Parameters.Add(new System.Data.SqlClient.SqlParameter("#User", record.Email));
await cmd.ExecuteNonQueryAsync();
I am not sure if this is the right way of asking a question about a code, but I appreciate any help.

You want to get data from a json document,we can use the Newtonsoft.Json.dll file to parse the json document.
I think you can change you code like this:
List<Info> jobInfoList = JsonConvert.DeserializeObject<List<Info>>(json);
And here is a sample about how to get data from a json documnet:
class Program
{
static void Main(string[] args)
{
string json = #"[{'id':9527,'username':'admin'}]";
List<Info> jobInfoList = JsonConvert.DeserializeObject<List<Info>>(json);
foreach (Info jobInfo in jobInfoList)
{
Console.WriteLine("UserName:" + jobInfo.username);
}
}
}
public class Info
{
public string id { get; set; }
public string username { get; set; }
}
You can declare variables to receive the data which you want to get from the json String in foreach . Then you can insert these data into your sql database as the parameters.
How to connect to the sql database and write data to sql database,you can see:
Quickstart:
Use .NET (C#) with Visual Studio to connect and query an Azure SQL database
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-connect-query-dotnet-visual-studio#insert-code-to-query-sql-database

Related

MSSQL replication triggers, how to handle conditional HasTrigger in EntityFrameworkCore

I am using EntityFrameworkCore version 7 to implement data access across a number of client databases.
I have recently run into the error 'Could not save changes because the target table has database triggers.' on one of the clients. The error is obviously self explanatory and I understand how to fix it using HasTrigger.
The problem is that this error has occurred because this specific client is replicated and has what I assume are auto generated triggers MSmerge_upd, MSmerge_ins, MSmerge_del. Concurrently the majority of my clients are not replicated and would therefore not have any of these triggers in their database.
So, what is the correct way to handle replication triggers in EntityFrameworkCore particularly when your clients have a mishmash where some are replicated and some are not? Is there a way to check inside IEntityTypeConfiguration if you are running on a replicated database and conditionally add the replication triggers? Is there some sort of best practice in terms of how to handle this scenario with the new HasTriggers requirement?
Given that nobody has posted any answer I will post what my workaround is for now.
I have created a class called AutoTriggerBuilderEntityTypeConfiguration which basically attempts to configure all the triggers for a given EF model.
There are some performance implications with this approach and it could potentially be improved by caching the triggers for all tables across the database but its sufficient for my use case.
It looks like this:
public abstract class AutoTriggerBuilderEntityTypeConfiguration<TEntity> : IEntityTypeConfiguration<TEntity>
where TEntity : class
{
private readonly string _connectionString;
public AutoTriggerBuilderEntityTypeConfiguration(string connectionString)
{
this._connectionString = connectionString;
}
public void Configure(EntityTypeBuilder<TEntity> builder)
{
this.ConfigureEntity(builder);
var tableName = builder.Metadata.GetTableName();
var tableTriggers = this.GetTriggersForTable(tableName);
var declaredTriggers = builder.Metadata.GetDeclaredTriggers();
builder.ToTable(t =>
{
foreach (var trigger in tableTriggers)
{
if (!declaredTriggers.Any(o => o.ModelName.Equals(trigger, StringComparison.InvariantCultureIgnoreCase)))
t.HasTrigger(trigger);
}
});
}
private IEnumerable<string> GetTriggersForTable(string tableName)
{
var result = new List<string>();
using (var connection = new SqlConnection(this._connectionString))
using (var command = new SqlCommand(#"SELECT sysobjects.name AS Name FROM sysobjects WHERE sysobjects.type = 'TR' AND OBJECT_NAME(parent_obj) = #TableName", connection)
{
CommandType = CommandType.Text
})
{
connection.Open();
command.Parameters.AddWithValue("#TableName", tableName);
using (var reader = command.ExecuteReader())
{
while (reader.Read())
result.Add(reader.GetString("Name"));
}
}
return result;
}
public abstract void ConfigureEntity(EntityTypeBuilder<TEntity> builder);
}

accesing azure database from azure function

I want to access a database from Xamarin and it looks like a good approach is to create a azure database (free for a while)
So I created an account in azure, I created a DBSQLServer, and a SQLDataBase, I set the admin user and password and opened the firewall as part of the process.
I then created a project with azure function in VS 2019, Created a function that just returns an string in the OkObjectResult, and it works (both visiting the local url and the public (after publishing)).
Then I installed the System.Data.SqlClient Nuget package and then tried to connect to the data base usign admin user and password like this :
//did not use "new line"'s in the actual code, included them here for readability.
using (SqlConnection conn =
new SqlConnection("
Server=tcp:javrsserver.database.windows.net,
1433;
Initial Catalog=JaviRS;
Persist Security Info=False;
User ID={javirs};Password={-The one set in the azure portal-};
MultipleActiveResultSets=False;
Encrypt=True;
TrustServerCertificate=False;
Connection Timeout=30;"))
{
try{
conn.Open();
}catch (Exception exc)
{
//here I collect "Login failed for user {javirs}"
//evntID : 18456
}
}
In theory is just wrong password, but im pretty sure the passowrd is Ok. So I guess is more of some windows weird stuff about user account control ...
Any clue ??
PS: If there is a simplier way to get simple sql into my Xamarin app .. I'm also interested. this Azure thing is an over engineering for what I need and is not free forever..
EDIT:
I tried connecting using SQLServver Object Explorer in visual studio, entered the same credentials I entered in the connection string and It does allow me in..
I have done it with following steps:
Azure Sql Script:
CREATE TABLE AzureSqlTable(
[Id] [int] PRIMARY KEY IDENTITY(1,1) NOT NULL,
[FirstName] [nvarchar](max) NULL,
[LastName] [nvarchar](max) NULL,
[Email] [nvarchar](max) NULL,
)
GO
Function Class:
public class AzureFunctionV2SqlTableClass
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
public string Email { get; set; }
public string DbOperationType { get; set; }
}
Azure Function Body:
[FunctionName("FunctionV2SqlConnectionExample")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
//Read Request Body
var content = await new StreamReader(req.Body).ReadToEndAsync();
//Extract Request Body and Parse To Class
AzureFunctionV2SqlTableClass objFuncV2Sql = JsonConvert.DeserializeObject<AzureFunctionV2SqlTableClass>(content);
// variable for global message.
dynamic validationMessage;
// Validate param because, I am checking here.
if (string.IsNullOrEmpty(objFuncV2Sql.FirstName))
{
validationMessage = new OkObjectResult("First Name is required!");
return (IActionResult)validationMessage;
}
if (string.IsNullOrEmpty(objFuncV2Sql.LastName))
{
validationMessage = new OkObjectResult("Last Name is required!");
return (IActionResult)validationMessage;
}
if (string.IsNullOrEmpty(objFuncV2Sql.Email))
{
validationMessage = new OkObjectResult("Email is required!");
return (IActionResult)validationMessage;
}
//Read database Connection
var sqlConnection = "Data Source =tcp:sqlserverInstancenNameFromAzurePortal.database.windows.net,1433;Initial Catalog=YouDbName;Persist Security Info=False;User ID=ServerUserName;Password=ServerPass;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;";
//Sql Execution Message varible
dynamic sqlExecutionMessage;
//Define Db operation Type
if (objFuncV2Sql.DbOperationType.ToUpper() == "INSERT")
{
using (SqlConnection conn = new SqlConnection(sqlConnection))
{
conn.Open();
var text = "INSERT INTO AzureSqlTable VALUES ('" + objFuncV2Sql.FirstName + "', '" + objFuncV2Sql.LastName + "', '" + objFuncV2Sql.Email + "') ";
using (SqlCommand cmd = new SqlCommand(text, conn))
{
sqlExecutionMessage = await cmd.ExecuteNonQueryAsync();
}
conn.Close();
}
validationMessage = new OkObjectResult(sqlExecutionMessage + " ROW INSERTED");
return (IActionResult)validationMessage;
}
//As we have to return IAction Type So converting to IAction Class Using OkObjectResult We Even Can Use OkResult
var result = new OkObjectResult("Operation Falid! No Relevant Command Found!");
return result;
}
Reference Required:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using System.Net.Http;
using System.Data.SqlClient;
Nuget Paackage I used:
System.Data.SqlClient(4.6.1)
Download from Nuget package manager. See the screen shot below:
Post Man Sample:
{
"FirstName": "Kiron New Sql FunctionV2",
"LastName":"Kiron New Local Sql",
"Email":"KironTest#microsoft.com",
"DbOperationType":"INSERT"
}
Point To Remember:
Follow what I exactly tried to demonstrate here, no engineering
before make it run
Just update the connection string with your Azure Sql Server
Credentials
Get rid of {} from password as I did not specified it in my example
also, While you copy connection string from portal in contain
{password} just omit {}
If your function encountered error from Azure Portal SQL Db
regarding your Client IP address. In that case just add your Client
IP like below:
Step:1
Step:2
Note: I have just tried to show Insert operation. Hope it will work accordingly.
The error says: "Login failed for user {javirs}"
This hints that you included the {} in your user name (and provably in your password. Removing the { and the } will do the trick.
It works now.

Switching or Selecting Database Server's Connection String Best Practices

I'm developing a software that will be used in different locations with different Servers. It differs in Server Name, Database name, etc.
Example:
Location 1 : Server Name: ChinaServer; Database Name: ChinaDB
Location 2 : Server Name: USServer; Database Name: USDB
Currently, I am using .ini file, I store the server name, database name and other configurations to it. I read it and use it runtime for my connection string. The problem here is that every time we switch locations, I need to change/edit the .ini file.
I'm asking everyone that has more experience that mine to give me other options or best approach on this matter.
Client's Environment : Windows 7
Developers : Windows 7, Visual Studio 2015, MS SQL, VB.NET
Thanks IA.
There's a couple of ways, each with their own advantages and disadvantages, the configuration file would work, could also store it in the registry of the server and read it that way. Or you could even use a My.Setting variable that can be updated in a settings page (probably not the most suitable for your situation)
You can get the basic idea from the The Twelve-Factor App "manifesto":
The twelve-factor app stores config in environment variables...
So what you need is to establish a machine-wide (setx /M NAME VALUE) environment variable which you'll later use like this:
var connectionString = Environment.GetEnvironmentVariable("MY_APP_CONNECTION_STRING");
var dbContext = new DbContext(connectionString);
You can use the SQL Server enumerator in System.Data.Sql, and run a query to get the database names. From there bind those lists to combo boxes, and use a SqlConnectionStringBuilder to keep track of your connection settings. You can save them to disk, or just ask the user to choose the server and database. Note that the enumerator is not guaranteed to always find all servers, so make sure you have a way to enter it manually if necessary.
private SqlConnectionStringBuilder _connString = new SqlConnectionStringBuilder();
private void RefreshConnectionString()
{
_connString.ApplicationName = AppDomain.CurrentDomain.ApplicationIdentity.FullName;
_connString.ApplicationIntent = ApplicationIntent.ReadWrite;
_connString.DataSource = GetSqlDatasources().FirstOrDefault();
_connString.InitialCatalog = GetSqlDatabases().FirstOrDefault();
_connString.AsynchronousProcessing = true;
_connString.ConnectTimeout = 5;
_connString.IntegratedSecurity = true;
_connString.Pooling = true;
}
private IEnumerable<string> GetSqlDatabases()
{
using (var conn = new SqlConnection(_connString.ConnectionString))
{
using (var cmd = new SqlCommand(#"SELECT [name] FROM sys.databases WHERE [name] NOT IN ('master', 'model', 'msdb', 'tempdb')", conn))
{
var dbnames = new List<string>();
try
{
conn.Open();
var reader = cmd.ExecuteReader();
while (reader.Read()) dbnames.Add(reader.GetString(0));
}
catch {}
return dbnames;
}
}
}
private IEnumerable<string> GetSqlDatasources()
{
var sqlEnum = SqlDataSourceEnumerator.Instance;
return sqlEnum.GetDataSources().Rows.OfType<DataRow>().Select(row => row[0].ToString());
}

Parse JSON data within SQL Server Integration Services Package?

I'm trying to set up an SSIS job that will pull a JSON-encoded mailing list from MailChimp, compare it to a list of customers in our CRM database (SQL Server), and upload via JSON any new customers not already there. I can't seem to find anything on serializing/deserializing JSON within SSIS, other than writing a script task, and it seems that I can't import the .Net serialization libraries into a script. Any suggestions? Thanks in advance!
Couple things to address here:
First, your problem with adding new libraries in the scripting component. I assume you're using VS 2008 to do your SSIS development and want to use the .net 3.5 library to do this. You go to project, add reference and you don't see any of the dll's you need. This may be in part that you're using windows 7 and the compact 3.5 framework. .net 3.5.1 comes with Windows 7, you just have to enable it. Go to control panel, programs and features. In that screen you will see Turn Windows features on or off, click on that. In that window check Microsoft .NET Framework 3.5.1, this way take a few minutes to run. Once it finishes look for a directory similar to these C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework.NETFramework\v3.5\Profile\Client and C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5. Between these 2 directories, you will find any dll you will need for serialization/deserializtion of JSON. These can be added to your project by going to Project-->Add Reference-->Browse Tab, then navigate to the v3.5 directory and select the dlls you need(System.Web.Extensions.dll(v3.5.30729.5446)is used in this example).
To get JSON from a web service, deserialize it, and send the data to your CRM database, you will have to use a script component as a source in your data flow and add columns to your output buffer that will be used to hold the data coming from the JSON feed(on the Input and Output screen). In the code, you will need to override the CreateNewOutputRows method. Here is an example of how to do this:
Say Your JSON looked like this...[{"CN":"ALL","IN":"Test1","CO":0,"CA":0,"AB":0},{"CN":"ALL","IN":"Test2","CO":1,"CA":1,"AB":0}]
I would fist define a class to mirror this JSON feed attributes (and the columns you defined on the inputs and outputs screen) that will eventually hold these values once you deserialize...as such:
class WorkGroupMetric
{
public string CN { get; set; }
public string IN { get; set; }
public int CO { get; set; }
public int CA { get; set; }
public int AB { get; set; }
}
Now you need to call your web service and get the JSON feed using an HttpWebRequest and a Stream:
string wUrl = "YOUR WEB SERVICE URI";
string jsonString;
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl);
HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();
Stream responseStream = httpWResp.GetResponseStream();
using (StreamReader reader = new StreamReader(responseStream))
{
jsonString = reader.ReadToEnd();
reader.Close();
}
Now we deserialize our json into an array of WorkGroupMetric
JavaScriptSerializer sr = new JavaScriptSerializer();
WorkGroupMetric[] jsonResponse = sr.Deserialize<WorkGroupMetric[]>(jsonString);
After deserializing, we can now output the rows to the output buffer:
foreach (var metric in jsonResponse)
{
Output0Buffer.AddRow();
Output0Buffer.CN = metric.CN;
Output0Buffer.IN = metric.IN;
Output0Buffer.CO = metric.CO;
Output0Buffer.CA = metric.CA;
Output0Buffer.AB = metric.AB;
}
Here is what all the code put together would look like(I have a step by step example here):
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Net;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.IO;
using System.Web.Script.Serialization;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void CreateNewOutputRows()
{
string wUrl = "YOUR WEB SERVICE URI";
try
{
WorkGroupMetric[] outPutMetrics = getWebServiceResult(wUrl);
foreach (var metric in outPutMetrics)
{
Output0Buffer.AddRow();
Output0Buffer.CN = metric.CN;
Output0Buffer.IN = metric.IN;
Output0Buffer.CO = metric.CO;
Output0Buffer.CA = metric.CA;
Output0Buffer.AB = metric.AB;
}
}
catch (Exception e)
{
failComponent(e.ToString());
}
}
private WorkGroupMetric[] getWebServiceResult(string wUrl)
{
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(wUrl);
HttpWebResponse httpWResp = (HttpWebResponse)httpWReq.GetResponse();
WorkGroupMetric[] jsonResponse = null;
try
{
if (httpWResp.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = httpWResp.GetResponseStream();
string jsonString;
using (StreamReader reader = new StreamReader(responseStream))
{
jsonString = reader.ReadToEnd();
reader.Close();
}
JavaScriptSerializer sr = new JavaScriptSerializer();
jsonResponse = sr.Deserialize<WorkGroupMetric[]>(jsonString);
}
else
{
failComponent(httpWResp.StatusCode.ToString());
}
}
catch (Exception e)
{
failComponent(e.ToString());
}
return jsonResponse;
}
private void failComponent(string errorMsg)
{
bool fail = false;
IDTSComponentMetaData100 compMetadata = this.ComponentMetaData;
compMetadata.FireError(1, "Error Getting Data From Webservice!", errorMsg, "", 0, out fail);
}
}
class WorkGroupMetric
{
public string CN { get; set; }
public string IN { get; set; }
public int CO { get; set; }
public int CA { get; set; }
public int AB { get; set; }
}
This can now be used as an input for a data destination (your CRM database). Once there you can use SQL to compare the data and find mismatches, send the data to another script component to serialize, and send any updates you need back to the web service.
OR
You can do everything in the script component and not output data to the output buffer. In this situation you would still need to deserialze the JSON, but put the data into some sort of collection. Then use the entity framework and LINQ to query your database and the collection. Determine what doesn't match, serialize it, and send that to the web service in the same script component.

Can we connect Sharepoint to SQL Server 6.5?

Not able to import application definition file!! Error: The metadata object with Name 'XYZ' and of Type 'LobSystemInstance' has a Property with Name 'DatabaseAccessProvider' that has either an invalid value or Type. Error was encountered at or just before Line: '10' and Position: '10'.
line 10 in ADF:
<"Property Name="DatabaseAccessProvider" Type="System.String">SqlOledb<"/Property>
Please give me ideas on how to display data from SQL Server 6.5 in Sharepoint?
The value of the node is invalid. You need to use SqlServer or OleDb. Check out this page for more information:
http://msdn.microsoft.com/en-us/library/ms550725(office.12).aspx
Im just starting on a similar task (so I found your unanswered question). I am trying to copy our documentation library in Sharepoint to an SQL db. Its not opening your file directly from SQL its using some c# code to setup a job which opens the sharepoint which may be what you are wanting.
There are two methods I have found so far:
One is to copy your data from sharepoint to a linked list in Access and then use the OLEDB methods in to open it.
Found here: C# Sync MS Access database to sql server
private static void BulkCopyAccessToSQLServer
(CommandType commandType, string sql, string destinationTable)
{
string connectionString = #"C:\Migration\Sharepoint Access SQL Batch Job\Database11.accdb";
using (DataTable dt = new DataTable())
{
string ConnStr = #"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Migration\Sharepoint Access SQL Batch Job\Database11.accdb;Jet OLEDB:Database Password=password";
//using (OleDbConnection conn = new OleDbConnection(Settings.Default.CurriculumConnectionString))
using (OleDbConnection conn = new OleDbConnection(ConnStr))
using (OleDbCommand cmd = new OleDbCommand(sql, conn))
using (OleDbDataAdapter adapter = new OleDbDataAdapter(cmd))
{
cmd.CommandType = commandType;
cmd.Connection.Open();
adapter.SelectCommand.CommandTimeout = 240;
adapter.Fill(dt);
adapter.Dispose();
}
using (SqlConnection conn2 = new SqlConnection(Settings.Default.qlsdat_extensionsConnectionString))
using (SqlConnection conn2 = new SqlConnection(connectionString))
{
conn2.Open();
using (SqlBulkCopy copy = new SqlBulkCopy(conn2))
{
copy.DestinationTableName = destinationTable;
copy.BatchSize = 1000;
copy.BulkCopyTimeout = 240;
copy.WriteToServer(dt);
copy.NotifyAfter = 1000;
}
}
}
}
The other is to use the Microsoft.Sharepoint libraries and open your sharepoint directly from the c# then copy it into your SQL.
Found here: http://www.dotnetspark.com/kb/3573-fetching-lists-from-sharepoint-2010-site.aspx
using (SharePointclientObj.ClientContext ctx = new SharePointclientObj.ClientContext(clientContext))
{
//Get the site
SharePointclientObj.Web site = ctx.Web;
ctx.Load(site);
//Get Lists
ctx.Load(site.Lists);
//Query
ctx.ExecuteQuery();
//Fill List
foreach (SharePointclientObj.List list in site.Lists)
{
Console.WriteLine(list.Title);
}
}

Resources