SQL Server version 2017. SSMS version 18.
I used 'Data export/Import' wizard of SSMS.
This table has two rows.
C1 / C2 / C3 are column name and NULL is null value.
C1 | C2 | C3
1 | NULL | Test
2 | Apple, Banana | Test3
When I export CSV without quotation.
CSV format can not use because Data include comma; 'Apple, Banana'
1,,Test
2,Apple, Banana,Test3
When I export CSV with quotation.
Also CSV format can not use because I can't distinguish "" is NULL or empty Stirng.
"1","","Test"
"2","Apple, Banana","Test3"
So, I hope When the data is null value.
CSV format like this.
"1",NULL,"Test"
"2","Apple, Banana","Test3"
How should I use do that? Is there only way using IFNULL function on Query?
Related
I have a data table which has a column as Acctno what is expected shows in separate column
|Acctno | expected_output|
|ABC:BKS:1023049101 | 1023049101 |
|ABC:UWR:19048234582 | 19048234582 |
|ABC:UEW:1039481843 | 1039481843 |
I know in Oracle SQL which I used the below
select regexp_substr(acctno,'[^:]',1,3) as expected_output
from temp_mytable
but in Microsoft SQL Server I am getting an error that regexp_substr is not a built in function
How can I resolve this issue?
We can use PATINDEX with SUBSTRING here:
SELECT SUBSTRING(acctno, PATINDEX('%:[0-9]%', acctno) + 1, LEN(acctno)) AS expected_output
FROM temp_mytable;
Demo
Note that this answer assumes that the third component would always start with a digit, and that the first two components would not have any digits. If this were not true, then we would have to do more work.
Just another option if the desired value is the last portion of the string and there are not more than 4 segments.
Select *
,NewValue = parsename(replace(Acctno,':','.'),1)
from YourTable
I am completing a snowflake university workshop but I have run into a problem. The course has provided an AVRO file and asked us to insert the data into a Variant column table. However when I run the COPY INTO commamd I get this error:
Number of columns in file (11) does not match that of the corresponding table (1), use file format option error_on_column_count_mismatch=false to ignore this error File 'iot_files/iot_files_sample_output.avro', line 1, character 827 Row 1, column "IOT_AVRO_DATA"[11] If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.
These are the instructions given by the course:
CREATE OR REPLACE TABLE IOT_AVRO_DATA
(mycolumn VARIANT);
copy INTO IOT_AVRO_DATA
FROM #GOOGLE_BUCKET_SFHOL/iot_files/iot_files_sample_output.avro;
FILE_FORMAT = (type = AVRO);
It looks like there is a mismatch between the number of columns in the file and in the table.
Any help advice would be appreciated, tried reaching out to snowflake via the workshop but they have not responded.
Are you sure your AVRO file is not corrupted?
The following works fine for me:
Upload to my stage a sample avro file (userdata1.avro taken from here)
spanaite#(no warehouse)#SERGIU_DB.(no schema)>put file:///Users/spanaite/Downloads/userdata1.avro #~;
+----------------+-------------------+-------------+-------------+--------------------+--------------------+----------+---------+
| source | target | source_size | target_size | source_compression | target_compression | status | message |
|----------------+-------------------+-------------+-------------+--------------------+--------------------+----------+---------|
| userdata1.avro | userdata1.avro.gz | 93561 | 79248 | NONE | GZIP | UPLOADED | |
+----------------+-------------------+-------------+-------------+--------------------+--------------------+----------+---------+
1 Row(s) produced. Time Elapsed: 3.026s
spanaite#(no warehouse)#SERGIU_DB.(no schema)>
Create a table and load the avro file:
create or replace table test_avro(mycolumn VARIANT);
copy into test_avro from #~/userdata1.avro.gz file_format = (type = AVRO);
select * from test_avro;
Try with one of the sample files from the link I posted above.
I'm trying to append two columns from a dataframe to an existing SQL server table. The code runs but when I query the SQL table, the additional rows are not present. What am I missing?
import sqlalchemy
engine = sqlalchemy.create_engine("mssql+pyodbc://user:pw#host:port/dbname?driver=ODBC+Driver+13+for+SQL+Server")
df.to_sql(name='database.tablename', con=engine, if_exists='append', index=False)
You cannot use dot notation in the name= parameter. Just use name=tablename. The other parts are fine.
If you need to assign a non-default (dbo) schema, there is a schema= parameter for df.to_sql(). The prefix database. is redundant because you have already assigned dbname in the engine.
Tested with SQL Server 2017 (latest docker image on debian 10) and anaconda python 3.7.
Test code
SQL Server part (create an empty table)
use testdb;
go
if OBJECT_ID('testdb..test') is not null
drop table test;
create table test (
[Brand] varchar(max),
[Price] money
);
Python part
from pandas import DataFrame
import sqlalchemy
# check your driver string
# import pyodbc
# pyodbc.drivers() # ['ODBC Driver 17 for SQL Server']
# connect
eng = sqlalchemy.create_engine("mssql+pyodbc://myid:mypw#localhost:1433/testdb?driver=ODBC+Driver+17+for+SQL+Server")
df = DataFrame(
data={'Brand': ['A','B','C'],
'Price': [10.00, 20.00, 30.00]},
columns=['Brand', 'Price']
)
df.to_sql(name="test", schema="dbo", con=eng, if_exists="append", index=False)
Result
select * from [test]
| Brand | Price |
|-------|---------|
| A | 10.0000 |
| B | 20.0000 |
| C | 30.0000 |
First off I am have a constraint the my solution must work from SQL Server 2008 R2
The problem that I'm trying to solve is that Excel converts the text value '002E9' to 2.00E+09. The task is to pass the original value '002E9' as text into a CSV file.
I have been passed a SSIS solution by a developer that has a the conversion as a SQL function. They have used
SELECT FORMAT(CAST(2.00E+09 AS FLOAT),'0E0');
This is fine in 2012 and above but does not work in SQL Server 2008 R2.
Is there a simple alternative? I'm happy to abandon SQL for a SSIS script if that's the best advice.
FORMAT doesn't exist in SQL Server 2008; but it's use is best avoided any way; it's an awfully slow function.
You can use CONVERT and the style 0 though:
SELECT REPLACE(CONVERT(varchar(10),CAST(2.00E+09 AS float),0),'+','');
This won't, however, give exactly the same format, and would return '2e009'. Based on the fact that you use the value '0E0' for the FORMAT function though (which would return '2E9' for your example value), I assume this is permissible.
Based upon the post Larnu made I arrived at this (note the REPLICATE function for getting the correct format from the stripped down string):
DECLARE #INPUTS AS table
(input_val varchar(100))
INSERT INTO #INPUTS
VALUES
('00923'),('00234'),('00568'),('00123'),('2.00E+09' ),('2.00E+34' ),('00RT1'),('001TL')
SELECT input_val
,REPLACE(REPLACE(REPLACE(input_val,'+',''),'0',''),'.','') paired_value
,REPLICATE('0',5-LEN(REPLACE(REPLACE(REPLACE(input_val,'+',''),'0',''),'.','')))
+REPLACE(REPLACE(REPLACE(input_val,'+',''),'0',''),'.','')+';' Converted_value
FROM #INPUTS
The results:
+-----------+--------------+-----------------+
| input_val | paired_value | Converted_value |
+-----------+--------------+-----------------+
| 00923 | 923 | 00923; |
| 00234 | 234 | 00234; |
| 00568 | 568 | 00568; |
| 00123 | 123 | 00123; |
| 2.00E+09 | 2E9 | 002E9; |
| 2.00E+34 | 2E34 | 02E34; |
| 00RT1 | RT1 | 00RT1; |
| 001TL | 1TL | 001TL; |
+-----------+--------------+-----------------+
Confirms the approach.
Thanks Larnu.
I have an SSRS report that was pointed to SQL Server views, which pointed to Oracle tables. I edited the SSRS report Dataset so as to query directly from the Oracle db. It seems like a very simple change until I got this error message:
System.InvalidCastException: Specified cast is not valid.
With the following details...
Field ‘UOM_QTY’ and it also says at
Oracle.ManagedDataAccess.Client.OracleDataReader.GetDecimal(Int32 i).
The SELECT statement on that field is pretty simple:
, (DELV_RECEIPT.INV_LBS/ITEM_UOM_XREF.CONV_TO_LBS) AS UOM_QTY
Does anyone know what would cause the message, and how to resolve the error? My objective is use to use the ORACLE datasource instead of SQL SERVER.
Error 1
Severity Code Description Project File Line Suppression State
Warning [rsErrorReadingDataSetField] The dataset ‘dsIngredientCosts’ contains a definition for the Field ‘UOM_QTY’. The data extension returned an error during reading the field. System.InvalidCastException: Specified cast is not valid.
at Oracle.ManagedDataAccess.Client.OracleDataReader.GetDecimal(Int32 i)
at Oracle.ManagedDataAccess.Client.OracleDataReader.GetValue(Int32 i)
at Microsoft.ReportingServices.DataExtensions.DataReaderWrapper.GetValue(Int32 fieldIndex)
at Microsoft.ReportingServices.DataExtensions.MappingDataReader.GetFieldValue(Int32 aliasIndex) C:\Users\bl0040\Documents\Visual Studio 2015\Projects\SSRS\Project_ssrs2016\Subscription Reports\Feed Ingredient Weekly Price Avg.rdl 0
Error 2
Severity Code Description Project File Line Suppression State
Warning [rsMissingFieldInDataSet] The dataset ‘dsIngredientCosts’ contains a definition for the Field ‘UOM_QTY’. This field is missing from the returned result set from the data source. C:\Users\bl0040\Documents\Visual Studio 2015\Projects\SSRS\Project_ssrs2016\Subscription Reports\Feed Ingredient Weekly Price Avg.rdl 0
Source Tables:
+------------+---------------+-------------+---------------+-----------+
| Source | TABLE_NAME | COLUMN_NAME | DataSize | COLUMN_ID |
+------------+---------------+-------------+---------------+-----------+
| ORACLE | DELV_RECEIPT | INV_LBS | NUMBER (7,0) | 66 |
+------------+---------------+-------------+---------------+-----------+
| ORACLE | ITEM_UOM_XREF | CONV_TO_LBS | NUMBER (9,4) | 3 |
+------------+---------------+-------------+---------------+-----------+
| SQL SERVER | DELV_RECEIPT | INV_LBS | numeric (7,0) | 66 |
+------------+---------------+-------------+---------------+-----------+
| SQL SERVER | ITEM_UOM_XREF | CONV_TO_LBS | numeric (9,4) | 3 |
+------------+---------------+-------------+---------------+-----------+
The error went away after adding a datatype conversion statement to the data selection.
, CAST(DELV_RECEIPT.INV_LBS/ITEM_UOM_XREF.CONV_TO_LBS AS NUMERIC(9,4)) AS UOM_QTY
Can anyone provide some information on why the original query would be a problem and why the CAST would fix these errors? I tried casting the results because someone on Code Project forum said...
why don't you use typed datasets? you get such head aches just because
of not coding in a type-safe manner. you have a dataset designer in
the IDE which makes the life better, safer, easier and you don't use
it. I really can't understand.
Here is an approach to fix this error with an extension method instead of modifying the SQL-Query.
public static Decimal MyGetDecimal(this OracleDataReader reader, int i)
{
try
{
return reader.GetDecimal(i);
}
catch (System.InvalidCastException)
{
Oracle.ManagedDataAccess.Types.OracleDecimal hlp = reader.GetOracleDecimal(i);
Oracle.ManagedDataAccess.Types.OracleDecimal hlp2 = Oracle.ManagedDataAccess.Types.OracleDecimal.SetPrecision(hlp, 27);
return hlp2.Value;
}
}
Thank you for this but what happens if your query looks like:
SELECT x.* from x
and .GetDecimal appears nowhere?
Any suggestions in that case? I have created a function in ORACLE itself that rounds all values in a result set to avoid this for basic select statements but this seems wrong for loading updateable datasets...
Obviously this is an old-school approach to getting data.