Background: First tech job. Got hired as a JR Dev but my first task is fixing an API call that is calling for way more data than normal, hitting the monthly API limit for the company I'm working for.
Person who wrote the entire app left the company. I don't have anyone in my company to ask questions to in regards to this syntax question.
Before I can look into the API call- I need to understand the data that's being generated from the DB.
Its nested in a slew of stored procedures in MS SQL Server Management. I dont have experience with stored procedures and am having a difficult time reading the queries and the flow of data.
Stored procedure file path leads to a .dtsx file.
The first step in the .dtsx file is this:
SELECT ? = CONVERT(VARCHAR(10),COUNT(DISTINCT Email)) FROM [DB1].[dbo].[AdvisorData] d INNER JOIN [DB1].[dbo].[AdvisorLists] l ON d.ListId = l.Id WHERE l.[DataSent] =0 AND Email IS NOT NULL
from there it goes into :
SELECT ? = COUNT(*) FROM [DB1].[dbo].[AppendedData_Staging] WHERE DATEPART(MONTH,[UpdateDate]) = DATEPART(MONTH,GETDATE()) AND DATEPART(YEAR,[UpdateDate]) = DATEPART(YEAR,GETDATE());
From there it states it generates a file and the file is sent to the API call.
Really my first question to understand the data flow is what does "?=" mean in this context?
I tried running the query without the =? and it returns nothing in the DB tables.
I've also tried breaking up the queries to further try to break down the data being manipulated but I cant seem to wrap my head around this.
Any help is appreciated.
Tried breaking up the queries and SELECT the individual tables but the ?= has me confused as to what the purpose of that is.
Related
So I have two queries that I'm working on, one comes from an Oracle DB and the other SQL Server DB. I'm trying to use PowerBI via Power Query as the cross over between the two. Because of the size of the Oracle DB I'm having a problem with running it, so my thought is to use one query as a clause/sub-query of the other to limit the number of results.
Based on the logic of MSFT's M language I'd assume there's a way to do sub-queries of another but I've yet to figure it out. Does anyone have any ideas on how to do this?
What I have learned to do is create a connection to the two under lying data sets, but do not load them. Then you can merge them, but do not use the default Table.NestedJoin().
After the PQ generates this, change it to:
= Table.Join(dbo_SCADocument,{"VisitID"},VISIT_KEY,{"VisitID"}) .
Also remove the trailing name. The reason is , it keeps query folding alive. For some reason, Table.NestedJoin() kills query folding. Note, if there are similar fields in the two sources other than the join it will fail.
Also it brings everything from both sources, but that is easy to alter . Also you will need to turn off the function firewall as this does not allow you to join potential sensitive data with non sensitive data. This is setting your privacy level to ignore all.
I would attempt this using the Merge command. I'm more of a UI guy, so I would click the Merge button. This generates a PQL statement for Table.Join
https://msdn.microsoft.com/en-us/library/mt260788.aspx
Setting Join Kind to Inner will restrict the output to matching rows.
I say attempt as your proposed query design will likely slow your query, not improve it. I would expect PQ to run both queries against the 2 servers and download all their data, then attempt the Join in Memory.
Suppose someone has to work on a lot of different SQL Server Databases which have got a lot of Tables and Queries / Views inside them.
After a period of time, it becomes very difficult to remember exactly what kind of columns are present within a given Table and View.
Please suggest some method by which one can keep a systematic list of all the Tables and Views that are present within a SQL Server Database, along with the columns that are present within them.
Are there any Add-on products or services etc. available that helps in making this type of work systematic?
Currently I add comments to each queries inside SQL Server to remind me of what this query is doing, but this method is not great. I am looking for some better and more efficient methods.
Please share any ideas that you might have in this direction.
Thanks a lot
You may find the following useful for each database.
select s.name, s.type, c.name , s.refdate
from syscolumns c
inner join sysobjects s on s.id = c.id
where s.xtype in('U','V')
order by s.refdate --use refdate for manual quick looks
-- use s.name for file output and long term analysis
I output this to text files with the exact same format and check them into source control for each database. I even make comments about fields as things change. This is not part of the formal process, it is just sanity big picture version tracking independent of the formal deployments.
I have made a search but couldn't find a solution which works for me.
I just wonder how Facebook or Linkedin manages to handle same type activity with one sentence?
I mean, if you store every activity with different IDs in an Activity Table, how can you list them as "Member_a and 15 more people changed their photos"
I'm trying to make a social activity wall for my web-site, it's not that big but I just wanted to know the logic on this situation.
For example, when first page loads, I make an Ajax call and listing 0-10 records and if user scrolls down, page makes another ajax call which lists 11-20 records.
Now; if I try to combine same type of activity after sql select query with using if else, if this 10 records are the same, the user will only see 1 item. I hope I could explain what I want to say :)
So, I need a solution which makes this query in SQL Statement.
I'm not asking from you to write a query for me, I just want to know the logic.
Here is a screenshot what I want to achieve:
You see, they are actually different stored data but they combined it and made it as a 1 item network update.
By the way, I'm using C# and SQL Server 2008.
for example:
SELECT Min(b.MemberName), COUNT(*) as Total FROM Network_Feed a
JOIN Member b on a.MemberID = b.MemberID
WHERE a.FeedType = 1
did I understand your question right?
It's not easy to manage petabytes of data as a one table. So, big projects running on SQL Server are used some advanced scaling(distributing data and load) tricks like Service Brokers and Replication.
You can check
http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=4000004532 as an SQL Server example.
am used lot of time , i know the diff between sql query and sp ,
SQL query will be compiled everytime it is executed.
Stored procedures are compiled only once when they are
executed for the first time.
This is general database question
But one big doubt is ,
For example ,
one dynamic work , that is i pass the userid to SP and sp will return the username,password,full details,
So for this scenario the query should execute once again know, so what is the necessary of SP instead of SQL QUERY ,
Please clear this doubt ,
Hi thanks for all your updates,
but i dont want the advantage, comparison ,
just say ,
How sp executing , while we go with dynamic works,
For example ,
if pass userid 10 then sp also read records 10 ,
if i pass 14 then, SP again look the 14 records , see this same work NORMAL SQL QUERY
doing , but on that time execute and fetching ,so why should i go for sp ,
Regards
Stored procedures, like the name says, are stored on the database server. They are transmitted to the server and compiled when you create them, and executed when you call them.
Simple SQL queries, on the other hand, are transmitted to the server and compiled each time you use them.
So transmitting of a huge query (instead of a simple "execute procedure" command) and compiling create an overhead which can be avoided by the use of a stored procedure.
MySQL, like other RDBMS, has a query cache. But this avoid only compiling, and only if the query is exactly the same than a previously executed query, which means the cache is not used if you execute 2 times the same query, with different values in a where clause, for example.
I see no reason for a stored procedure simply to query for all user details.
Stored procedures are functional code that you execute on the database server. I can think of three reasons why you'd use them:
To create an interface for users that hides the schema details from clients.
Performance. Extensive calculations on a large data set might be done more efficiently on the database server
Sometimes it can be difficult (or impossible, depending on your skill) to express what you think you need in a declarative, set-based language like SQL. That's when some people throw up their hands and write stored procs.
Only 1. would be justifiable from your question. I would recommend sticking with SQL.
UPDATE: The new information you provided still does not justify stored procedures in my opinion. A query that returns 14 records is routine.
We've got a weird problem with joining tables from SQL Server 2005 and MS Access 2003.
There's a big table on the server and a rather small table locally in Access. The tables are joined via 3 fields, one of them a datetime field (containing a day; idea is to fetch additional data (daily) from the big server table to add data to the local table).
Up until the weekend this ran fine every day. Since yesterday we experienced strange non-time-outs in Access with this query. Non-time-out means that the query runs forever with rather high network transfer, but no timeout occurs. Access doesn't even show the progress bar. Server trace tells us that the same query is exectuted over and over on the SQL server without error but without result either. We've narrowed it down to the problem seemingly being accessing server table with a big table and either JOIN or WHERE containing a date, but we're not really able to narrow it down. We rebuilt indices already and are currently restoring backup data, but maybe someone here has any pointers of things we could try.
Thanks, Mike.
If you join a local table in Access to a linked table in SQL Server, and the query isn't really trivial according to specific limitations of joins to linked data, it's very likely that Access will pull the whole table from SQL Server and perform the join locally against the entire set. It's a known problem.
This doesn't directly address the question you ask, but how far are you from having all the data in one place (SQL Server)? IMHO you can expect the same type of performance problems to haunt you as long as you have some data in each system.
If it were all in SQL Server a pass-through query would optimize and use available indexes, etc.
Thanks for your quick answer!
The actual query is really huge; you won't be happy with it :)
However, we've narrowed it down to a simple:
SELECT * FROM server_table INNER JOIN access_table ON server_table.date = local_table.date;
If the server_table is a big table (hard to say, we've got 1.5 million rows in it; test tables with 10 rows or so have worked) and the local_table is a table with a single cell containing a date. This runs forever. It's not only slow, It just does nothing besides - it seems - causing network traffic and no time out (this is what I find so strange; normally you get a timeout, but this just keeps on running).
We've just found KB article 828169; seems to be our problem, we'll look into that. Thanks for your help!
Use the DATEDIFF function to compare the two dates as follows:
' DATEDIFF returns 0 if dates are identical based on datepart parameter, in this case d
WHERE DATEDIFF(d,Column,OtherColumn) = 0
DATEDIFF is optimized for use with dates. Comparing the result of the CONVERT function on both sides of the equal (=) sign might result in a table scan if either of the dates is NULL.
Hope this helps,
Bill
Try another syntax ? Something like:
SELECT * FROM BigServerTable b WHERE b.DateFld in (SELECT DISTINCT s.DateFld FROM SmallLocalTable s)
The strange thing in your problem description is "Up until the weekend this ran fine every day".
That would mean the problem is really somewhere else.
Did you try creating a new blank Access db and importing everything from the old one ?
Or just refreshing all your links ?
Please post the query that is doing this, just because you have indexes doesn't mean that they will be used. If your WHERE or JOIN clause is not sargable then the index will not be used
take this for example
WHERE CONVERT(varchar(49),Column,113) = CONVERT(varchar(49),OtherColumn,113)
that will not use an index
or this
WHERE YEAR(Column) = 2008
Functions on the left side of the operator (meaning on the column itself) will make the optimizer do an index scan instead of a seek because it doesn't know the outcome of that function
We rebuilt indices already and are currently restoring backup data, but maybe someone here has any pointers of things we could try.
Access can kill many good things....have you looked into blocking at all
run
exec sp_who2
look at the BlkBy column and see who is blocking what
Just an idea, but in SQL Server you can attach your Access database and use the table there. You could then create a view on the server to do the join all in SQL Server. The solution proposed in the Knowledge Base article seems problematic to me, as it's a kludge (if LIKE works, then = ought to, also).
If my suggestion works, I'd say that it's a more robust solution in terms of maintainability.