I have a hybrid application where part of data (mostly legacy) is stored in SQL Server and another part in Mongodb. I just converted all primary key types in SQL Server to use ObjectId which I generate in the application when inserting new records into SQL Server.
Now, I found that I need to clone some template records (about 10-20 records at a time), and in order to do that I need to be able to generate ObjectId values via a SQL Server function or stored proc.
Is it possible and is there code available?
This question is old, but I was trying to do the same thing. This is what I came up with on SQL Server 2012.
Create Function NewObjectId(#counter binary(3))
returns binary(12)
begin
declare #epoch datetime2, #seconds binary(4), #process binary(2), #hostname binary(3)
set #epoch = '1/1/1970'
select #seconds = cast(Datediff(ss, #epoch, getutcdate()) as binary(4))
select #hostname = cast(HashBytes('MD5', HOST_NAME()) as binary(3))
select #process = cast(##SPID as binary(2))
declare #objectId binary(12)
select #objectId = (#seconds + #hostname + #process + #counter)
return #objectId
end
This can be called like this:
select NewObjectId(CRYPT_GEN_RANDOM(3))
The reason CRYPT_GEN_RANDOM(3) is passed in is because calling that function is apparently side-effecting and it can't be used inside another function. I would have preferred to use an incrementing sequence for the counter portion, but a random number works as well.
Also, I noticed that you said you are using a char(24) to store the value. This returns a binary(12) since that's what the MongoDB ObjectIds are. Using binary(12) also requires half of the space to store the value.
I'm sure this isn't helpful now, but it was a fun problem to solve.
I'm going to try taking my ObjectID C# code and see if I can load it as a CLR function into SQL Server. That might give better results and performance.
I think, you can use NEWID function which generates 16-byte uniqueidentifier.
But in MongoDB The BSON ObjectId Datatype is a 12-byte binary value.
Try this
SELECT LEFT(REPLACE(CAST(NEWID() as varchar(36)),'-',''),24)
Hope, this helps.
EDITED
In article Object IDs described BSON ObjectID specification. The format includes:
TimeStamp. This is a unix style timestamp. It is a signed int representing the number of seconds before or after January 1st 1970
(UTC).
Machine. This is the first three bytes of the (md5) hash of the machine host name, or of the mac/network address, or the virtual
machine id.
Pid. This is 2 bytes of the process id (or thread id) of the process generating the object id.
Increment. This is an ever incrementing value, or a random number if a counter can't be used in the language/runtime.
The server itself and almost all drivers use the format above.
So, it is impossible to generate MongoDB ObjectID in SQL Server.
The only way to solve this problem is to change logic of the application.
Related
First question here so hoping that someone can help!
Im doing a lot of conversions of Access backends on to SQL server, keeping the front end in Access.
I have come across something that i need a little help with.
In Access, I have a query that is using a user-defined function in order to amalgamate some data from rows in a table into one variable. (By opening a recordset and enumerating through, adding to a variable each time.)
For example:
The query has a field that calls the function like this:
ProductNames: Product(ContractID)
And the VBA function "Product()" searches a table based on the ContractID. Cycles through each row it finds and concatenates the results of one field into one variable, ultimately returned to the query.
Obviously, moving this query to SQL server as a view means that that function will not be found as its in Access.
Can I use a function or stored procedure in order to do the same thing? (I have never used them before)
I must stress that I cannot create, alter or drop tables at run-time due to very strict production environment security.
If someone could give me an example id be really grateful.
So i need to be able to call it from the view as shown above.
Let say the table im looking at for the data is called tbl_Products and it has 2 columns:
| ContractID | Product |
How would that be done?! any help massively appreciated!
Andy
Yes you can most certainly do the same thing and adopt the same approach in SQL like you did in the past with VBA + SQL.
The easy solution would be to link to the view, and then build a local query that adds the additional column. However, often for reasons of performance and simply converting sql from Access to T-SQL, then I often “duplicate” those VBA functions as T-SQL functions.
The beauty of this approach is once you make this function, then this goes a “long” way towards easy converting some of your Access SQL to t-sql and views.
I had a GST calculation function in VBA that you would pass the amount, and a date (because the gst rate changes at a known date (in the past, or future).
So I used this function all over the place in my Access SQL.
When I had to convert to sql server, then I was able to use “views” and pass-though quires from Access and simply use “very” similar sql and include that sql function in the sql just like I did in Access.
You need to create what is called a SQL function. This function is often called a scaler function. This function works just like a function in VBA.
So in t-sql store procedure, or even as a expression in your SQL just like in Access!!!!
In your example, lets assume that you have some contract table, and you want to grab the “status” column (we assume text).
And there could be one, 1 or “several” or none!.
So we will concatenate each of the child records “status” code based on contract id.
You can thus fire up SSMS and in the database simply expand your database in the tree view. Now expand “programmability”. Now expand functions. You see “scaler-valued functions”. These functions are just like VBA functions. Once created, you can use the function in code (t-sql) or in views etc.
At this point, you can now write t-sql code in place of VBA code.
And really, you don’t have to “expand” the tree above – but it will allow you to “find” and “see” and “change” your functions you create. Once created then ANY sql, or code for that database can use the function as a expression just like you did in Access.
This code should do the trick:
CREATE FUNCTION [dbo].[ContractStatus]
(#ContractID int)
RETURNS varchar(255)
AS
BEGIN
-- Declare a cursor (recordset)
DECLARE #tmpStatus varchar(25)
DECLARE #MyResult varchar(255)
set #MyResult = ''
DECLARE rst CURSOR
FOR select Status from tblContracts where ID = #ContractID
OPEN rst
FETCH NEXT FROM rst INTO #tmpStatus
WHILE ##FETCH_STATUS = 0
BEGIN
IF #MyResult <> ''
SET #MyResult = #MyResult + ','
SET #MyResult = #MyResult + #tmpStatus
FETCH NEXT FROM rst INTO #tmpStatus
END
-- Return the result of the function
RETURN #MyResult
END
Now, in sql, you can go:
Select ProjectName, ID, dbo.ProjectStatus([ID]) as MyStatus from tblProjects.
I'm trying to test a SQL query in SQL Server Management Studio that normally requires a multivalue parameter from the SSRS report its a part of.
I'm not sure to how hard code a multi value parameter in management studio. The report was created by a vendor, I'm just trying to make it runnable for testing outside of SSRS.
For example the parameter in SSRS is a collection of numbers that the user selects - ie "3100, 3102, 3105" would be the selections for the multivalue parameter called #object_code
I've got something like this - but it's not working.
Declare #Object_Code varchar(100)
Set #object_Code = ('3100','3102','3105')
....really long vendor written query I don't thoroughly understand...
IN(#object_code)
You have to use String-Split function to separate comma separated values.
For example-
Declare #Object_Code varchar(100)
Set #Object_Code = '3100,3102,3105'
....really long vendor written query I dont thoroughly understand...
--T.object_code IN (#object_code)
Inner Join dbo.Split(#Object_Code, ',') as S On S.data = T.object_code
Search your database first for any string-split function.
If you want to create string-split function then follow this -
T-SQL split string
If you use SQL Server 2016 you might want to check out the function STRING_SPLIT.
If you use a lower version of SQL Server and you can't or don't want to create a separate function, the following could be an alternative:
declare #object_code varchar(100);
set #object_code = '3100,3102,3105';
select
ltrim(rtrim(x.par.value('.[1]','varchar(max)'))) as object_code
from (
select convert(xml,'<params><param>' + replace(#object_code,',', '</param><param>') + '</param></params>') as c
) tbl
cross apply
c.nodes('/params/param') x(par);
Everybody seems to be getting hung up on splitting a string that doesn't have to be a string. We're just trouble shooting a query here and need a way to feed it values. It's not important how SSRS does it, just that we can reproduce the result.
Declare #Object_Code table (params varchar(20));
INSERT #object_Code
VALUES ('3100'),('3102'),('3105')
....really long vendor written query I don't thoroughly understand...
IN (SELECT params FROM #object_code)
Then spend some quality time getting to know the query.
I'm having a really tough time figuring how to use an xml data column in SQL Server, specifically for use with Entity Framework.
Basically, one of our tables stores "custom metadata" provided by users in the form of XML, so it seemed sensible to store this in an Xml column in the table.
One of the requirements of our application is to support searching of the metadata, however. The users are able to provided an XPath query string, as well as a value to compare the value of the XPath with, to search for elements that contain metadata that matches their query.
I identified the SQL Server xml functions as ideal for this (eg, [xmlcol].exist('/path1/path2[0][text()=''valuetest'''] ), but they're not supported by Entity Framework, irritatingly (or specifically, xml columns aren't supported). As an alternative, I tried creating a UDF that passes the user-provided XPath to the xml functions, but then discovered that the xml functions only allow string literals, so I can't provide variables...
At this point, I was running out of options.
I created a small bit of code that performs a regular expression replace on the result of a IQueryable.ToString(), to inject my XPath filter in, and then send this string to the database manually, but there are problems with this too, such as the result doesn't seem to lazily load the navigational properties, for example.
I kept looking, and stumbled upon the idea of SQLCLR types, and started creating a SQLCLR function that performs the XPath comparison. I thought I was onto a winner at this point, until a colleague pointed out that SQL Server in Azure doesn't support SQLCLR - doh!
What other options do I have? I seem to be running very close to empty...
You could do this in a stored procedure where you build your query dynamically.
SQL Fiddle
MS SQL Server 2008 Schema Setup:
create table YourTable
(
ID int identity primary key,
Name varchar(10) not null,
XMLCol xml
);
go
insert into YourTable values
('Row 1', '<x>1</x>'),
('Row 2', '<x>2</x>'),
('Row 3', '<x>3</x>');
go
create procedure GetIt
#XPath nvarchar(100)
as
begin
declare #SQL nvarchar(max);
set #SQL = N'
select ID, Name
from YourTable
where XMLCol.exist('+quotename(#XPath, '''')+N') = 1';
exec (#SQL);
end
Query 1:
exec GetIt N'*[text() = "2"]'
Results:
| ID | NAME |
--------------
| 2 | Row 2 |
To remain "customisable", the SqlQuery method on DbSet can be used:
var query = #"SET ARITHABORT ON;
select * from [YourTable] where
[xmlcol].exist('/path1/path2[0][text()=''{0}''']";
var numOfResults = 5;
var offsetPage = 1;
var results = Context.YourTable.SqlQuery(String.Format(query,"valuetest"))
.OrderBy(x => x.col)
.Skip(offsetPage * numOfResults)
.Take(numOfResults).ToList();
Note, due to its dynamic nature, this method would also most likely expose some degree of sql injection security holes.
Note: I'm running under SQL Server 2008 R2...
I've taken the time to read dozens of posts on this site and other sites on how to execute dynamic SQL where the query is more than 4000 characters. I've tried more than a dozen solutions proposed. The consensus seems to be to split the query into 4000-character variables and then do:
EXEC (#SQLQuery1 + #SQLQuery2)
This doesn't work for me - the query is truncated at the end of #SQLQuery1.
Now, I've seen samples how people "force" a long query by using REPLICATE a bunch of spaces, etc., but this is a real query - but it gets a little more sophisticated than that.
I have SQL View with a name of "Company_A_ItemView".
I have 10 companies that I want to create the same exact view, with different names, e.g.
"Company_B_ItemView"
"Company_C_ItemView"
..etc.
If you offer help, please don't ask why there are multiple views - just accept that I need to do it this way, OK?
Each company has its own set of tables, and the CREATE VIEW statement references several tables by name. Here's BRIEF sample, but remember, the total length of the query is around 6000 characters:
CREATE view [dbo].[Company_A_ItemView] as
select
WE.[Item No_],
WE.[Location Code],
LOC.[Bin Number],
[..more fields, etc.]
from
[Company_A_Warehouse_Entry] WE
left join
[Company_A_Location] LOC
...you get the idea
So, what I am currently doing is:
a. Pulling the contents of the CREATE VIEW statement into 2 Declared Variables, e.g.
Set #SQLQuery1 = (select text
from syscomments
where ID = 1382894081 and colid = 1)
Set #SQLQuery2 = (select
from syscomments
where ID = 1382894081 and colid = 2)
Note that this is how SQL stores long definitions - when you create the view, it stores the text into multiple syscomments records. In my case, the view is split into a text chunk of 3591 characters into the first syscomment record and the rest of the text is in the second record. I have no idea why SQL doesn't use all 4000 characters in the syscomment field. And the statement is broken in the middle of a word.
Please note in all my examples, all #SQLQueryxxx variables are declared as varchar(max). I've also tried declaring them as nvarchar(max) and varchar(8000) and nvarchar(8000) with the same results.
b. I then do a "Search and Replace" for "Company_A" and replace it with "Company_B". In the code below, the variable "#CompanyID" is first set to "Company_B":
SET #SQLQueryNew1 = #SQLQuery1
SET #SQLQueryNew1 = REPLACE(#SQLQueryNew1, 'Company_A', #CompanyID)
SET #SQLQueryNew2 = #SQLQuery2
SET #SQLQueryNew2 = REPLACE(#SQLQueryNew2, 'Company_A',#CompanyID)
c. I then try:
EXEC (#SQLQueryNew1 + #SQLQueryNew2)
The message returned indicates that it's trying to execute the statement truncated at the end of #SQLQueryNew1, e.g. 80% (approx) of the query's text.
I've tried CAST'ing the final result into a new varchar(max) and nvarchar(max) - no luck
I've tried CAST'ing the original query a new varchar(max) and nvarchar(max)- no luck
I've looked at the result of retrieving the original CREATE VIEW statement, and it's fine.
I've tried various other ways of retrieving the original CREATE VIEW statement, such as:
Set #SQLQuery1 = (select VIEW_DEFINITION)
FROM [MY_DATABASE].[INFORMATION_SCHEMA].[VIEWS]
where TABLE_NAME = 'Company_A_ItemView')`
This one returns only the first 4000 characters of the CREATE VIEW
Set #SQLQuery1 = (SELECT (OBJECT_DEFINITION(#ObjectID))
If I do a
SELECT LEN(OBJECT_DEFINITION(#ObjectID))
it returns the correct length of the query (e.g. 5191), but if I look at #SQLQuery1, or try to
EXEC(#SQLQuery1), the statement is still truncated.
c. There are some references that state that since I'm manipulating the text of the query after retrieving it, the resulting variables are then truncated to 4000 characters. I've tried CAST'ing the result as I do the REPLACE, e.g.
SET #SQLQueryNew1 = SELECT (CAST(REPLACE(#SQLQueryNew1,
'Company_A',
#CompanyID) AS varchar(max))
Same result.
I know there are other methods, such as creating stored procedures for creating the views. But the views are being developed and are somewhat "in flux", so placing the text of the CREATE VIEW inside a stored proc is cumbersome. My goal is to be able to take Company_A's view and replicate it exactly - multiple times, except reference Company_B's view name and table names, Company_C's view name and table names, etc.
I'm wondering if there is anyone out there who has done this type of manipulation of a long SQL "CREATE VIEW" statement and try to execute it.
Just use VARCHAR(MAX) or NVARCHAR(MAX). They work fine for EXEC(string).
FYI,
Note that this is how SQL stores long definitions - when you create
the view, it stores the text into multiple syscomments records.
This is not correct. This is how it used to be done on SQL Server 2000. Since SQL Server 2005 and higher they are saved as NVARCHAR(MAX) in a single entry in sys.sql_modules.
syscomments is still around, but it is retained read-only solely for compatibility.
So all you should need to do is to change your #SQLQuery1,2,etc. variables to a single NVARCHAR(MAX) variable, and pull your View code from the [definition] column of the sys.sql_modules table instead.
Note that you should be careful with your string manipulations as there are certain functions that will revert to (N)VARCHAR(4000) output if all of their input arguments are not (N)VARCHAR(MAX). (Sorry, I do not know which ones, but REPLACE() may be one). In fact, this may be what has been causing so much confusion in your tests.
declare your sql variables (#SQLQuery1...) as nvarchar(4000)
be sure each sql part did't exceed 4000 byte (copy each part to a text file and test the file size in bytes)
I'm migrating from SQL Server to PostgreSQL. I've seen from How to declare a variable in a PostgreSQL query that there is no such thing as temporary variables in native sql queries.
Well, I pretty badly need a few... How would I go about mixing in plpgsql? Must I create a function and then delete the function in order to get access to a language? that just seems error prone to me and I'm afraid I'm missing something.
EDIT:
cmd.CommandText="insert......" +
"declare #app int; declare #gid int;"+
"set #app=SCOPE_IDENTITY();"+ //select scope_identity will give us our RID that we just inserted
"select #gid=MAX(GROUPID) from HOUSEHOLD; set #gid=#gid+1; "+
"insert into HOUSEHOLD (APPLICANT_RID,GROUPID,ISHOH) values "+
"(#app,#gid,1);"+
"select #app";
rid=cmd.ExecuteScalar();
A direct rip from the application in which it's used. Note we are in the process of converting from SQL server to Postgre. (also, I've figured out the scope_identity() bit I think)
What is your schema for the table being inserted? I'll try and answer based on this assumption of the schema:
CREATE TABLE HOUSEHOLD (
APPLICANT_RID SERIAL, -- PostgreSQL auto-increment
GROUPID INTEGER,
ISHOH INTEGER
);
If I'm understanding your intent correctly, in PostgreSQL >= 8.2, the query would then be:
INSERT INTO HOUSEHOLD (GROUPID, ISHOH)
VALUES ((SELECT COALESCE(MAX(GROUPID)+1,1) FROM HOUSEHOLD), 1)
RETURNING APPLICANT_RID;
-- Added call to the COALESCE function to cover the case where HOUSEHOLD
-- is empty and MAX(GROUPID) returns NULL
In PostgreSQL >= 8.2, any INSERT/DELETE/UPDATE query may have a RETURNING clause that acts like a simple SELECT performed on the result set of the change query.
If you're using a language binding, you can hold the variables there.
For example with SQLAlchemy (python):
my_var = 'Reynardine'
session.query(User.name).filter(User.fullname==my_var)
If you're in psql, you have variables:
\set a 5
SELECT :a;
And if your logic is in PL/pgSQL:
tax := subtotal * 0.06;
Must I create a function and then
delete the function in order to get
access to a language?
Yes, but this shortcoming is going to be removed in PostgreSQL 8.5, with the addition of DO command. 8.5 is going to be released in 2010.
You can also declare session variables using plperl - http://www.postgresql.org/docs/8.4/static/plperl-global.html
you install a language that you want to use with the CREATE LANGUAGE command for known languages. Although you can use other languages.
Language installation docs
CREATE LANGUAGE usage doc
You will have to create a function to use it. If you do not want to make a permanent function in the db then the other choice would be to use a scrip in python or something that uses a postgresql driver to connect to the db and do queries. You can then manipulate or look through the data in the script. For instance in python you would install the pygresql library and in your script import pgdb which you can use to connect to the db.
PyGreSQL Info
I think that PostgreSQL's row-type variable would be the closest thing:
A variable of a composite type is
called a row variable (or row-type
variable). Such a variable can hold a
whole row of a SELECT or FOR query
result, so long as that query's column
set matches the declared type of the
variable.
You mentioned the post (How to declare a variable in a PostgreSQL query).
I believe there is a suitable answer farther down the chain of solutions if using psql and the \set command:
my_db=> \set myvar 5
my_db=> SELECT :myvar + 1 AS my_var_plus_1;