Fetching Data from SQL Server Through Shiny Application on Date selection bases - sql-server

Hey Every One i am developing a Shiny Application, where we Extract a data from sql Server through ODBC Connector by selecting Date to and from in a Application. i am unable to identify where the issue is because if i execute the code independently on R studio i am able to extract the data from sql Server But then when the same code is executed in Shiny Environment i am unable to achieve the data on shiny here is the below Kindly Guide me on this Thank you.
# ---------------------ui Code -----------------------------
library(shiny)
shinyUI(pageWithSidebar(
headerPanel("Time Analytics"),
sidebarPanel(
dateRangeInput(inputId = "dateRange",
label = "Date range",
start = "2007-09-17",
max = Sys.Date()
)
),#sidebar Panel Ends
# 09-Main Panel ----
mainPanel(
tabsetPanel(id ="theTabs",
tabPanel("Summary", dataTableOutput("tabi"),textOutput("tabii"))
)
)#Main Panel Ends
))
#------------------Server ----------------------------------
library(shiny);library(sqldf)
library(plyr);library(RODBC)
library(ggplot2)
#Creating the connection
shinyServer(function(input, output, session){ # pass in a session argument
# prep data once and then pass around the program
passData <- reactive({
ch = odbcConnect("Test")
#qry <- "SELECT * FROM Nifty50"
#qry <- cat("SELECT * FROM Nifty50 WHERE Date >= ",as.date(input$dateRange[1])," AND Date <= ",input$dateRange[2])
qry <- paste("SELECT * FROM Nifty50 WHERE Date >= ",input$dateRange[1]," AND Date <= ",input$dateRange[2])
#paste("SELECT * FROM Nifty50 WHERE Date >= ",input$dateRange[1]," AND Date <= ",input$dateRange[2])
subset_Table <- sqlQuery(ch,qry)
odbcClose(ch)
subset_Table <- as.data.frame(subset_Table)
return(subset_Table)
})
output$tabi <- renderDataTable({
d<- as.data.frame(passData())
d
})
output$tabii <- renderText({
paste("Minimium Data :",input$dateRange[1], "Max Date:",input$dateRange[2])
})
# ----------------------------------------------------End
})
Here the task is i need to fetch the data from selected Table on the bases of Date to and From criteria, which will be the subset data as per the selected Date from shiny app.

Modify qry as follows:
qry <- paste("SELECT * FROM Nifty50 WHERE Date >= '", input$dateRange[1], "' AND Date <= '", input$dateRange[2], "'", sep = "")

Related

Dynamic SQL query with cell reference in Excel

I have a query connected to a pivot table and I want it to show results based on a date from cell (so the user can change it without editing the query). I've read one way to do it is creating a connection to the cell
And drilling down that query so its data that can be referenced by another query
I've read I have to reference that connected table (just a date) through
SQL1 = Text.Combine({SQL,Date}),
SQL2 = Text.Combine({SQL1, "‘" }),
Source = Sql.Database("servername", "db", [Query= SQL2])
So I think it should look like this:
let
SQL =
"select trim(codart) AS REF,
descart AS 'DESCRIPTION',
codalm AS WAREHOUSE,
descalm AS WAREHOUSEDESCRIP,
unidades AS UNITS,
entran AS 'IN',
salen AS 'OUT',
m.entran*1 + m.salen*-1 as MOVEMENT,
(select sum(m1.entran*1 + m1.salen*-1)
from MOVSTOCKS m1
where m1.codart = m.codart and m1.codalm =
m.codalm and m.fecdoc >= m1.fecdoc) as 'CUMULATIVE',
PRCMEDIO as 'VALUE',
FECDOC as 'DATE',
REFERENCIA as 'REF',
tipdoc as 'DOCUMENT'
from (select m.*,
sum(m.entran*1 - m.salen*-1) over (partition by m.codart, m.codalm order by fecdoc) as cumulative,
max(fecdoc) over (partition by m.codart, m.codalm) as max_fecdoc
from MOVSTOCKS m
where fecdoc <= ‘ ), m
where fecdoc = max_fecdoc
and (m.entran <> 0 or m.salen <> 0)
order by fecdoc",
SQL1 = Text.Combine({SQL,Date}),
SQL2 = Text.Combine({SQL1, "‘" }),
Source = Sql.Database("servername", "db", [Query= SQL2])
in
Source
The problem is... if that dynamic cell is formated as date, the power query shows this error:
Expression.Error: We cannot convert the value #date (2020, 9, 30) to type Text.
Details:
Value= 30/09/2020
Type=Type
And if I format date as text, then it shows this error (after changing privacy options):
DataSource.Error: Microsoft SQL: Incorrect syntax near '‘'.
Details:
DataSourceKind=SQL
DataSourcePath=servername;db
Message=Incorrect syntax near '‘'.
Number=102
Class=15
I want SQL to read this
where fecdoc <= ‘ ), m
as
where fecdoc <= 30/09/2020 ), m
Having in mind that date comes from an excel cell that can be changed by the user.
I'm new with SQL and M, I don't even know if what I'm trying makes any sense.
Any feedback will be helpful. Thank you very much
Instead of whatever you're trying to do with SQL1 and SQL1, in your query, replace
where fecdoc <= ‘ ), m
with
where fecdoc <= '" & Date & "' ), m
where Date is a reference to your date in string format.

Behavior of Dates when using SQLServer vs. odbc R packages to connect to the database

I am working on an application that tracks tasks and dates using a SQL database. My application currently utilizes the RSQLServer package to connect to the database, because I can get that working on Windows and Linux. I've noticed that it does not store dates consistently.
Here is my MWE (ish - database connections will have to be set up yourself).
library(DBI)
library(dplyr)
library(lubridate)
# --- Connect to the database via RSQLServer and odbc --------------------------
db <- "your SQL database"
server <- "your SQL server"
conn <- dbConnect(RSQLServer::SQLServer(), server = server, database = db,
properties = list(user = "", password = "",
useNTLMv2 = TRUE, domain = "")
)
conn2 <- dbConnect(odbc::odbc(), dsn = "")
# --- Create the test table ----------------------------------------------------
dplyr::db_drop_table(conn, "TestTable")
if (!dbExistsTable(conn, "TestTable")) {
TestProcessStr <- "
CREATE TABLE TestTable(
Process_ID INT NOT NULL IDENTITY(1,1),
Start_Dt DATE NOT NULL,
End_Dt DATE DEFAULT '9999-12-31',
Comment VARCHAR(30),
PRIMARY KEY( Process_ID )
);"
dbExecute(conn, TestProcessStr)
} else {
message("TestTable exists")
}
# --- Write to test table using different connections --------------------------
rowadd <- data_frame(Start_Dt = Sys.Date(), End_Dt = Sys.Date() + months(3),
Comment = "SQLServer, Date as Date")
write_res <- dbWriteTable(conn, name = "TestTable",
value = rowadd, append = T)
# Convert all dates to character
rowadd <- rowadd %>% mutate_if(is.Date, as.character) %>%
mutate(Comment = "SQLServer, Date as Char")
write_res <- dbWriteTable(conn, name = "TestTable",
value = rowadd, append = T)
rowadd <- data_frame(Start_Dt = Sys.Date(), End_Dt = Sys.Date() + months(3)) %>%
mutate(Comment = "ODBC, Date as Date")
write_res <- dbWriteTable(conn2, name = "TestTable",
value = rowadd, append = T)
# Convert all dates to character
rowadd <- rowadd %>% mutate_if(is.Date, as.character) %>%
mutate(Comment = "ODBC, Date as Character")
write_res <- dbWriteTable(conn2, name = "TestTable",
value = rowadd, append = T)
# View database status
ttab <- dbReadTable(conn, "TestTable")
ttab
# --- Disconnect ---------------------------------------------------------------
dbDisconnect(conn)
dbDisconnect(conn2)
This produces the following output on my machine:
> ttab
Process_ID Start_Dt End_Dt Comment
1 1 2018-01-14 2018-04-14 SQLServer, Date as Date
2 2 2018-01-15 2018-04-15 SQLServer, Date as Char
3 3 2018-01-15 2018-04-15 ODBC, Date as Date
4 4 2018-01-15 2018-04-15 ODBC, Date as Character
Why does RSQLServer treat Date objects differently than dates-as-character objects? odbc::odbc() doesn't display the same behavior - is there something different about the java-based methods used by RSQLServer?
Ideally, I'd like to not have to convert all dates to strings before each and every write operation. I can't switch to the odbc package because I can't get the DSN properly configured on Linux without getting my IT department involved.

Google analytics organic search loop problems

The loop functions completely different and I'm not sure if it is because of the google analytics package because very little is different between the code.
Doesn't WORK difference in print statement output shows results are not coming out correctly.
library(googleAnalyticsR)
library(tidyverse)
#settings
start_dat <- as.character(Sys.Date()-31)
end_dat <- as.character(Sys.Date()-1)
#Authorize Google Analytics R- this will open a webpage
#You must be logged into your Google Analytics account on your web browser
ga_auth()
account_sum <- ga_account_list()
#Add the start and end date to the date frame, as well as some columns to use to populate the metrics
account_sum$start_dat <- start_dat
account_sum$end_dat <- end_dat
## choose the v3 segment
segment_for_call <- "gaid::-5"
## make the v3 segment object in the v4 segment object:
seg_ob <- segment_ga4("OrganicTraffic", segment_id = segment_for_call)
# cycle through the list of views, pull the data, and add it to the
#account_summary
for (i in 1:5){
view_id <- (Book1CSV[[1]][i])
views=view_id
ga_dat <- google_analytics_4(views,
date_range = c(start_dat, end_dat),
segments = seg_ob,
metrics = c("sessions", "pageviews"),
dimensions = c("year","segment"))
ga_dat <- summarise(ga_dat,
sessions = sum(sessions),
pageviews = sum(pageviews))
account_sum$sessions[i] <- ga_data$sessions
account_sum$pageviews[i] <- ga_data$pageviews
print(account_summary)
}
clean_sum <- select(account_sum,
ID = webPropertyId ,
Account = accountName,
Views = views,
Type = type,
Level = level,
'Start Date' = start_dat,
'End Date' = end_dat,
Sessions = sessions,
Pageviews = pageviews)
write.csv (ga_dat, "doesntwork.csv", row.names = TRUE)
THIS WORKS!!!!!!!!!!!!!!! Print statement prints code
library(googleAnalyticsR)
library(tidyverse)
#settings
start_date <- as.character(Sys.Date()-31)
end_date <- as.character(Sys.Date()-1)
metrics <- c("sessions", "pageviews")
dimensions <- "year"
#Authorize Google Analytics R- this will open a webpage
#You must be logged into your Google Analytics account on your web browser
ga_auth()
account_summary <- ga_account_list()
#Add the start and end date to the date frame, as well as some columns to use to populate the metrics
account_summary$start_date <- start_date
account_summary$end_date <- end_date
# cycle through the list of views, pull the data, and add it to the
#account_summary
for (i in 1:6){
view_id <- (Book1CSV[[1]][i])
ga_data <- google_analytics_4(viewId = view_id,
date_range = c(start_date,end_date),
metrics = metrics,
dimensions = dimensions)
# This query might return multiple rows (if it spans a year boundary), so
#collapse and clean up
ga_data <- summarise(ga_data,
sessions = sum(sessions),
pageviews = sum(pageviews))
#add the totals to the account summary
account_summary$sessions[i] <- ga_data$sessions
account_summary$pageviews[i] <- ga_data$pageviews
print(account_summary)
}
# Make a more compact set of data
clean_summary <- select(account_summary,
Account = accountName,
View = viewId,
Type = type,
Level = level,
'Start Date' = start_date,
'End Date' = end_date,
Sessions = sessions,
Pageviews = pageviews,
ID = webPropertyId)
select
write.csv (clean_summary, "worksfine.csv", row.names = FALSE)
Can't really understand whats going wrong here. Would like some detailed help. I have uploaded a file called Book1CSV as I couldn't get a try-catch statement that would work to catch the error. Google merchant store beta account was causing the crash.
This is an anti-R pattern, modifying an object in a loop. Its better to return a list of data.frames, then merge then at the end and create your summary.
So something like:
my_view_ids <- c(12345,233445,232434)
## fetch data, create a list of
all_data <- lapply(my_view_ids, function(id){
one_data <- tryCatch(google_analytics_4(viewId = id,
date_range = c(start_date,end_date),
metrics = metrics,
dimensions = dimensions),
error = function(ex){message(ex); NULL})
one_data$viewId <- id
one_data
})
## Reduce the list of data.frames to one data.frame
all_dataframe <- Reduce(rbind, all_data)
### make your summary etc.

T-SQL Temp Tables and Stored Procedures from R using RevoScaleR

In the example below, I was able to get the query to work with one exception. When I use q in place of source.query during the RxSqlServerData step, I get the error rxCompleteClusterJob Execution halted.
The first goal is to use a stored procedure in place of a longer query. Is this possible?
The second goal would be to create and call upon a #TEMPORARY table within the stored procedure. I'm wondering if that is possible, as well?
library (RODBC)
library (RevoScaleR)
sqlConnString <- "Driver=SQL Server;Server=SAMPLE_SERVER; Database=SAMPLE_DATABASE;Trusted_Connection=True"
sqlWait <- TRUE
sqlConsoleOutput <- FALSE
sql_share_directory <- paste("D:\\RWork\\AllShare\\", Sys.getenv("USERNAME"), sep = "")
sqlCompute <- RxInSqlServer(connectionString = sqlConnString, wait = sqlWait, consoleOutput = sqlConsoleOutput)
rxSetComputeContext(sqlCompute)
#This Sample Query Works
source.query <- paste("SELECT CASE WHEN [Order Date Key] = [Picked Date Key]",
"THEN 1 ELSE 0 END AS SameDayFulfillment,",
"[City Key] AS city, [STOCK ITEM KEY] AS item,",
"[PICKER KEY] AS picker, [QUANTITY] AS quantity",
"FROM [WideWorldImportersDW].[FACT].[ORDER]",
"WHERE [WWI ORDER ID] >= 63968")
#This Query Does Not
q <- paste("EXEC [dbo].[SAMPLE_STORED_PROCEDURE]")
inDataSource <- RxSqlServerData(sqlQuery=q, connectionString=sqlConnString, rowsPerRead=500)
order.logit.rx <- rxLogit(SameDayFulfillment ~ city + item + picker + quantity, data = inDataSource)
order.logit.rx
Currently, only T-SQL SELECT statements are allowed as input data-set, not stored procedures.

How to pass data.frame for UPDATE with R DBI

With RODBC, there were functions like sqlUpdate(channel, dat, ...) that allowed you pass dat = data.frame(...) instead of having to construct your own SQL string.
However, with R's DBI, all I see are functions like dbSendQuery(conn, statement, ...) which only take a string statement and gives no opportunity to specify a data.frame directly.
So how to UPDATE using a data.frame with DBI?
Really late, my answer, but maybe still helpful...
There is no single function (I know) in the DBI/odbc package but you can replicate the update behavior using a prepared update statement (which should work faster than RODBC's sqlUpdate since it sends the parameter values as a batch to the SQL server:
library(DBI)
library(odbc)
con <- dbConnect(odbc::odbc(), driver="{SQL Server Native Client 11.0}", server="dbserver.domain.com\\default,1234", Trusted_Connection = "yes", database = "test") # assumes Microsoft SQL Server
dbWriteTable(con, "iris", iris, row.names = TRUE) # create and populate a table (adding the row names as a separate columns used as row ID)
update <- dbSendQuery(con, 'update iris set "Sepal.Length"=?, "Sepal.Width"=?, "Petal.Length"=?, "Petal.Width"=?, "Species"=? WHERE row_names=?')
# create a modified version of `iris`
iris2 <- iris
iris2$Sepal.Length <- 5
iris2$Petal.Width[2] <- 1
iris2$row_names <- rownames(iris) # use the row names as unique row ID
dbBind(update, iris2) # send the updated data
dbClearResult(update) # release the prepared statement
# now read the modified data - you will see the updates did work
data1 <- dbReadTable(con, "iris")
dbDisconnect(con)
This works only if you have a primary key which I created in the above example by using the row names which are a unique number increased by one for each row...
For more information about the odbc package I have used in the DBI dbConnect statement see: https://github.com/rstats-db/odbc
Building on R Yoda's answer, I made myself the helper function below. This allows using a dataframe to specify update conditions.
While I built this to run transaction updates (i.e. single rows), it can in theory update multiple rows passing a condition. However, that's not the same as updating multiple rows using an input dataframe. Maybe somebody else can build on this...
dbUpdateCustom = function(x, key_cols, con, schema_name, table_name) {
if (nrow(x) != 1) stop("Input dataframe must be exactly 1 row")
if (!all(key_cols %in% colnames(x))) stop("All columns specified in 'key_cols' must be present in 'x'")
# Build the update string --------------------------------------------------
df_key <- dplyr::select(x, one_of(key_cols))
df_upt <- dplyr::select(x, -one_of(key_cols))
set_str <- purrr::map_chr(colnames(df_upt), ~glue::glue_sql('{`.x`} = {x[[.x]]}', .con = con))
set_str <- paste(set_str, collapse = ", ")
where_str <- purrr::map_chr(colnames(df_key), ~glue::glue_sql("{`.x`} = {x[[.x]]}", .con = con))
where_str <- paste(where_str, collapse = " AND ")
update_str <- glue::glue('UPDATE {schema_name}.{table_name} SET {set_str} WHERE {where_str}')
# Execute ------------------------------------------------------------------
query_res <- DBI::dbSendQuery(con, update_str)
DBI::dbClearResult(query_res)
return (invisible(TRUE))
}
Where
x: 1-row dataframe that contains 1+ key columns, and 1+ update columns.
key_cols: character vector, of 1 or more column names that are the keys (i.e. used in the WHERE clause)
Here is a little helper function I put together using REPLACE INTO to update a table using DBI, replacing old duplicate entries with the new values. It's basic and for my own needs, but should be easy to modify. All you need to pass to the function is the connection, table name, and dataframe. Note that the table must have a PRIMARY KEY column. I've also included a simple working example.
row_to_list <- function(Y) suppressWarnings(split(Y, f = row(Y)))
sql_val <- function(y){
if(!is.numeric(y)){
return(paste0("'",y,"'"))
}else{
if(is.na(y)){
return("NULL")
}else{
return(as.character(y))
}
}
}
to_sql_row <- function(x) paste0("(",paste(do.call("c", lapply(x, FUN = sql_val)), collapse = ", "),")")
bracket <- function(x) paste0("`",x,"`")
to_sql_string <- function(x) paste0("(",paste(sapply(x, FUN = bracket), collapse = ", "),")")
replace_into_table <- function(con, table_name, new_data){
#new_data <- data.table(new_data)
cols <- to_sql_string(names(new_data))
vals <- paste(lapply(row_to_list(new_data), FUN = to_sql_row), collapse = ", ")
query <- paste("REPLACE INTO", table_name, cols, "VALUES", vals)
rs <- dbExecute(con, query)
return(rs)
}
tb <- data.frame("id" = letters[1:20], "A" = 1:20, "B" = seq(.1,2,.1)) # sample data
dbWriteTable(con, "test_table", tb) # create table
dbExecute(con, "ALTER TABLE test_table ADD PRIMARY KEY (id)") # set primary key
new_data <- data.frame("id" = letters[19:23], "A" = 1:5, "B" = seq(101,105)) # new data
new_data[4,2] <- NA # add some NA values
new_data[5,3] <- NA
table_name <- "test_table"
replace_into_table(con, "test_table", new_data)
result <- dbReadTable(con, "test_table")

Resources