OPENJSON unable to parse Chinese characters - sql-server

I'm trying to convert my JSON data into a table format in SQL Server.
Following are my JSON data:
[{
"emp_no": "001",
"emp_designation":"Data Admin",
"emp_name": "Peter",
"emp_name2": "彼特"
},
{
"emp_no": "002",
"emp_designation":"Software Engineer",
"emp_name": "Lee",
"emp_name2": "李"
}]
What I had tried are:
DECLARE #JSON NVARCHAR(MAX)
set #JSON='[{
"emp_no": "001",
"emp_designation":"Data Admin",
"emp_name": "Peter",
"emp_name2": "彼特"},
{
"emp_no": "002",
"emp_designation":"Software Engineer",
"emp_name": "Lee",
"emp_name2": "李"
}]'
--Method 1
SELECT * INTO #emp_temp FROM OPENJSON(#JSON)
WITH (emp_no varchar(20),
emp_designation varchar(50),
emp_name NVARCHAR(100),
emp_name2 NVARCHAR(100))
SELECT * FROM #Emp_temp
DROP TABLE #Emp_temp
--Method 2
SELECT
JSON_Value (EMP.VALUE, '$.emp_no') as emp_no,
JSON_Value (EMP.VALUE, '$.emp_designation') as emp_designation,
JSON_Value (EMP.VALUE, '$.emp_name') as emp_name,
JSON_Value (EMP.VALUE, '$.emp_name2') as emp_name2
INTO #Emp_temp2
FROM OPENJSON (#JSON) as EMP
SELECT * FROM #Emp_temp2
DROP TABLE #Emp_temp2
However, both temp table return me following result, with the Chinese characters remain as "???".
Temp table select result
emp_no emp_designation emp_name emp_name2
001 |Data Admin | Peter| ??
002 |Software Engineer| Lee | ?
Any idea how to preserve the original Chinese characters after parse the data into temp table?
Thanks.
*Edit:
I know it can work by putting a extra 'N' in front of the JSON
set #JSON=N'[
{ "emp_no": "001...
.....
But actually the JSON is a parameter in a Store Procedure, I cannot simply add a N like : set #JSON = 'N' + #JSON,
which this will jeopardize the format of the JSON data, and cause an error.
ALTER PROCEDURE [dbo].[SP_StoreEmpInfo]
#JSON NVARCHAR(max)
#JSON = 'N' + #JSON
/*Will cause invalid JSON format error */
SELECT
JSON_Value (EMP.VALUE, '$.emp_no') as.....

Try adding 'N' before your sql set to indicate that unicode characters are contained within like this:
DECLARE #JSON NVARCHAR(MAX)
set #JSON=N'[{
"emp_no": "001",
"emp_designation":"Data Admin",
"emp_name": "Peter",
"emp_name2": "彼特"},
{
"emp_no": "002",
"emp_designation":"Software Engineer",
"emp_name": "Lee",
"emp_name2": "李"
}]'
This question may assist in background:
What does N' stands for in a SQL script ? (the one used before characters in insert script)

Related

Extract values from JSON of Array in T-SQL

I have a JSON like this to process in SQL
{"RowIndex":[1,2], "Data":["a","b"]}
and i want to extract the data to show that as a table like this
RowIndex Data
1 a
2 b
I understand that i have to use OPENJSON, JSON_QUERY or JSON_VALUE but i cannot find a way to get what I want that not implies to write a query with many join like
select C1.value as RowIndex,
C2.value as Data,
From (select [key], value from OPENJSON(JSON_QUERY(#jsonstring, '$.RowIndex'))) C1
inner join (select [key], value from OPENJSON(JSON_QUERY(#jsonstring, '$.Data'))) C2 on C1.[key] = C2.[key]
Because if the arrays in the JSON grow the query will be unmaintenable and slow
One method, using a "couple" of OPENJSON clauses:
DECLARE #JSON nvarchar(MAX) = N'{"RowIndex":[1,2], "Data":["a","b"]}';
SELECT RI.[value] AS RowIndex,
D.[value] AS Data
FROM OPENJSON(#JSON)
WITH (RowIndex nvarchar(MAX) AS JSON,
Data nvarchar(MAX) AS JSON) J
CROSS APPLY OPENJSON(RowIndex) RI
CROSS APPLY OPENJSON(Data) D
WHERE RI.[key] = D.[key];
To elaborate on my comments though, it seems like you should be fixing the JSON design and have something like this:
[
{
"RowIndex": "1",
"Data": "a",
"Number": "1"
},
{
"RowIndex": "2",
"Data": "b",
"Number": "3"
}
]
Which can be far more easily queried:
DECLARE #JSON nvarchar(MAX) = N'[
{
"RowIndex": "1",
"Data": "a",
"Number": "1"
},
{
"RowIndex": "2",
"Data": "b",
"Number": "3"
}
]';
SELECT *
FROM OPENJSON(#JSON)
WITH (RowIndex int,
Data char(1),
Number int) OJ;

How to insert all array elements in one field in SQL Server?

I have a column named "RecordIds". I am importing data from json to sql server . I have this json file. but when i inserted all in table but in last three fields i got no value in database table. how can i solve this problem ?
{
"Id": 1,
"ImageName": "person1",
"PersonName": "C1-C4:stretch",
"GroupId": 1,
"Text": "Sam has complete C4 tetraplegia. He sits in his power wheelchair all day and is unable to move his arms.<br/><br/>The aim of Sam`s programme is to prevent contracture.<br/><br/>This programme is suitable for people with C1-C4 tetraplegia.",
"RecordIds": [511, 517, 569, 571, 688, 1037],
"DropdownIds": [1, 56, 58],
"TextTitleIds": [3, 5, 7]
}
here is my query
declare #details varchar(max)
select #details =
BulkColumn
from openrowset(BULK'D:\JSON\mmh\examples.json', single_blob) json
if(ISJSON(#details) = 1)
begin
print 'Valid Json'
insert into [dbo].[Examples]
select * from
openjson(#details, '$.example')
with(
id smallint '$.Id',
[ImageName] varchar(50) '$.ImageName',
[PersonName] varchar(50) '$.PersonName',
[GroupId] smallint '$.GroupId',
[Text] varchar(50) '$.Text',
[RecordIds] varchar(50) '$.RecordIds',
[DropdownIds] varchar(50) '$.DropdownIds',
[TextTitleIds] varchar(50) '$.TextTitleIds'
)
end
else
begin
print 'invalid Json'
end
i got this output
RecordIds DropdownIds TextTitleIds
NULL NULL NULL
You need to use AS JSON, eg
declare #json nvarchar(max) =
'{
"example":
{
"Id": 1,
"ImageName": "person1",
"PersonName": "C1-C4:stretch",
"GroupId": 1,
"Text": "Sam has complete C4 tetraplegia. He sits in his power wheelchair all day and is unable to move his arms.<br/><br/>The aim of Sam`s programme is to prevent contracture.<br/><br/>This programme is suitable for people with C1-C4 tetraplegia.",
"RecordIds": [511, 517, 569, 571, 688, 1037],
"DropdownIds": [1, 56, 58],
"TextTitleIds": [3, 5, 7]
}
}'
select * from
openjson(#json, '$.example')
with(
id smallint '$.Id',
[ImageName] varchar(50) '$.ImageName',
[PersonName] varchar(50) '$.PersonName',
[GroupId] smallint '$.GroupId',
[Text] varchar(50) '$.Text',
[RecordIds] nvarchar(MAX) '$.RecordIds' as json,
[DropdownIds] nvarchar(MAX) '$.DropdownIds' as json,
[TextTitleIds] nvarchar(MAX) '$.TextTitleIds' as json
)

How can I read below Json column stored in SQL Server using openjson?

Declare #ResponseText nvarchar(4000)
set #responseText ='{
"submissions": [
{
"xml_id":"id_x5d94851726b470.68571510",
"fields": [
{"fieldvalue":"customerEmail#xyzdomain.com","fieldid":"57282490"},
{"fieldvalue":"123","fieldid":"57282423"},
{"fieldvalue":"12345-678900","fieldid":"57282500"},
{"fieldvalue":"Test Message here ","fieldid":"57282564"}
]
}
]
}'
SELECT *
FROM OPENJSON (#ResponseText, '$.submissions') WITH (
ID NVARCHAR(100) '$.xml_id',
$.fields.field NVARCHAR(100) ...
)
etc rest of all the record? I got "NULL" for the rest fields under fields array
You can try it like this:
Declare #ResponseText nvarchar(4000)
set #responseText ='{
"submissions": [
{
"xml_id":"id_x5d94851726b470.68571510",
"fields": [
{"fieldvalue":"customerEmail#xyzdomain.com","fieldid":"57282490"},
{"fieldvalue":"123","fieldid":"57282423"},
{"fieldvalue":"12345-678900","fieldid":"57282500"},
{"fieldvalue":"Test Message here ","fieldid":"57282564"}
]
}
]
}'
--The query
SELECT A.ID
,B.*
FROM OPENJSON (#ResponseText, '$.submissions')
WITH (ID NVARCHAR(100) '$.xml_id'
,fields NVARCHAR(MAX) AS JSON) A
OUTER APPLY OPENJSON(a.fields)
WITH(fieldvalue NVARCHAR(150)
,fieldid BIGINT) B;
The result
ID fieldvalue fieldid
id_x5d94851726b470.68571510 customerEmail#xyzdomain.com 57282490
id_x5d94851726b470.68571510 123 57282423
id_x5d94851726b470.68571510 12345-678900 57282500
id_x5d94851726b470.68571510 Test Message here 57282564
The idea in short:
You started correctly using the WITH-clause to read the xml_id. The property fields is nothing else than another element on the same level. But we return it AS JSON. This will allow to add another APPLY OPENJSON(), pass in the fragment we got from $.fields and use another WITH-clause to get the two properties of the objects within the array.

Querying Json whose root is an array of objects in SQL Server

I have a column in SQL table that has json value like below:
[
{"address":{"value":"A9"},
"value":{"type":11,"value":"John"}},
{"address":{"value":"A10"},
"value":{"type":11,"value":"Doe"}}]
MSDN Examples for JSON_VALUE or JSON_QUERY require a json object at root. How can I query above to return rows that have "address" as A9 and "value" as John? I'm using SQL Azure.
Something like this:
declare #json nvarchar(max) = '[
{"address":{"value":"A9"},
"value":{"type":11,"value":"John"}},
{"address":{"value":"A10"},
"value":{"type":11,"value":"Doe"}}]'
select a.*
from openjson(#json) r
cross apply openjson(r.value)
with (
address nvarchar(200) '$.address.value',
name nvarchar(200) '$.value.value'
) a
where address = N'A9'
and name = N'John'
outputs
address name
------- -----
A9 John
(1 row affected)
It may not be entirely relevant to the OP's post as the usage is different, however it is possible to retrieve arbitrary items from a root-level unnamed JSON array e.g.
declare #json nvarchar(max) = '[
{"address":
{"value":"A9"},
"value":
{"type":11,"value":"John"}
},
{"address":
{"value":"A10"},
"value":
{"type":11,"value":"Doe"}
}
]'
select
JSON_VALUE(
JSON_QUERY(#json, '$[0]'),
'$.address.value') as 'First address.value',
JSON_VALUE(
JSON_QUERY(#json, '$[1]'),
'$.address.value') as 'Second address.value'
Output :
First address.value Second address.value
A9 A10

To split records in one column into multiple columns

I am not sure whether this is achievable in sql server or not. I have a table ADDRESS and its metadata is given below. SQ_ID is my unique key column
SQ_ID INT
ADDRESS_LINE_1 varchar(255)
ADDRESS_LINE_2 varchar(255)
ADDRESS_LINE_3 varchar(255)
ADDRESS_LINE_4 varchar(255)
REGION varchar(255)
POSTOCODE varchar(255)
COUNTRY_CODE varchar(255)
Now I have loaded around 2 millions of records into this table and the problem is that all the address details have been loaded into ADDRESS LINE 1 column in this table. Now I am trying to find a way to split this. Below given is some of the sample set of addresses.
So I want to split the data in Address line 1 in such a way that
value in addressLine1 in the given record should be populated to
addressline1 in the same table value in addressLine2 in the given
record should be populated to addressline2 in the same table value in
addressLine3 in the given record should be populated to addressline3
in the same table value in primaryTown in the given record should be
populated to addressline4 in the same table value in provinceOrState
in the given record should be populated to region in the same table
value in isoAlpha2Code in the given record should be populated to
COUNTRY_CODE in the same table value in zipOrPostalCode in the given
record should be populated to POSTOCODE in the same table
My Expected output is given below
Also I am attaching set of sample address values for testing
{'addressLine1': '67 xxxx Road', 'primaryTown': 'HOxxxCHxxCH',
'zipOrPostalCode': 'RM11 1EX', 'addressCountry': {'isoAlpha2Code':
'GB'}}
{'primaryTown': 'MünXXer', 'addressCountry': {'isoAlpha2Code': 'DE'}}
{'addressLine1': '28 THE EXCAC', 'primaryTown': 'PERTH',
'provinceOrState': 'WA', 'addressCountry': {'isoAlpha2Code': 'AU'}}
{'addressLine1': '28 THE ESPLANADE', 'primaryTown': 'PERTH',
'provinceOrState': 'WA', 'addressCountry': {'isoAlpha2Code': 'AU'}}
{'addressLine1': '76 XXX STREET', 'primaryTown': 'MAXDFOT',
'provinceOrState': 'NSW', 'addressCountry': {'isoAlpha2Code': 'AU'}}
{'addressLine1': 'UNIT 56', 'addressLine2': '22 XDFR STREET',
'primaryTown': 'MANLY VALE', 'provinceOrState': 'NSW',
'addressCountry': {'isoAlpha2Code': 'AU'}}
{'addressLine1': 'BjoXCDSaret 15', 'addressLine2': 'Jppsdeheim',
'addressLine3': '', 'primaryTown': 'AKERdwfUS', 'addressCountry':
{'isoAlpha2Code': 'NO'}}
You can't parse JSON by splitting. SQL Server 2016 and later supports JSON though, so you can extract the data you want using functions like JSON_VALUE.
You don't have to do that though, unless you want to index and filter those values. It may make sense to extract the country or zip code if you intend to filter or group results using them. Address lines on the other hand can't be queried and might as well remain in the JSON string. You can always extract them when necessary with JSON_VALUE.
Puttin primaryTown in AddressLine4 looks like a mistake though. That is a significant attribute that could be used in querying. It should go to its own field, or not extracted at all.
SQL Server 2016 and later
In any case, you can parse a JSON value with JSON_VALUE , eg :
declare #myTable table(json varchar(max))
insert into #myTable
values
('{"addressLine1": "BjoXCDSaret 15", "addressLine2": "Jppsdeheim","addressLine3": "", "primaryTown": "AKERdwfUS", "addressCountry":
{"isoAlpha2Code": "NO"}}'),
('{"addressLine1": "67 xxxx Road", "primaryTown": "HOxxxCHxxCH",
"zipOrPostalCode": "RM11 1EX", "addressCountry": {"isoAlpha2Code":
"GB"}}'),
('{"primaryTown": "MünXXer", "addressCountry": {"isoAlpha2Code": "DE"}}'),
('{"addressLine1": "28 THE EXCAC", "primaryTown": "PERTH",
"provinceOrState": "WA", "addressCountry": {"isoAlpha2Code": "AU"}}'),
('{"addressLine1": "28 THE ESPLANADE", "primaryTown": "PERTH",
"provinceOrState": "WA", "addressCountry": {"isoAlpha2Code": "AU"}}'),
('{"addressLine1": "76 XXX STREET", "primaryTown": "MAXDFOT",
"provinceOrState": "NSW", "addressCountry": {"isoAlpha2Code": "AU"}}'),
('{"addressLine1": "UNIT 56", "addressLine2": "22 XDFR STREET",
"primaryTown": "MANLY VALE", "provinceOrState": "NSW",
"addressCountry": {"isoAlpha2Code": "AU"}}')
select
JSON_VALUE(json,'$.zipOrPostalCode') as ZipCode,
JSON_VALUE(json,'$.primaryTown') as primaryTown,
JSON_VALUE(json,'$.addressCountry.isoAlpha2Code') as CountryCode
from #mytable
You can use the same expressions to create persisted computed columns on the table and index them, thus speeding queries that need to filter by country or zip code.
Older versions
In older versions, perhaps the simplest solution would be to create a SQLCLR function that uses JSON.NET to parse the data and return the values.
Another option is to convert the JSON string to XML using string replacements and use XPATH to retrieve values. This can get really tricky though as the replacements depend on the expected data, are sensitive to whitespace and can easily break when dealing with nested objects.
For example, this flat JSON object :
declare #json varchar(max)='{"addressLine1": "BjoXCDSaret 15", "addressLine2": "Jppsdeheim", "addressLine3": "", "primaryTown": "AKERdwfUS"}'
Can be converted to XML with a few replacements. Notice that I fixed whitespace differences to ensure there's always a space between tokens. Otherwise I'd have to replace both "," and ", ".
select
cast(
replace(
replace(
replace(
replace(#json,'{"','<it '),
'": "',' ="'),
'", "','" '),
'}',' />')
as xml)
The result is :
<it addressLine1="BjoXCDSaret 15" addressLine2="Jppsdeheim" addressLine3="" primaryTown="AKERdwfUS" />
This can now be queried with .value :
select
cast(replace(replace(replace(replace(#json,'{"','<it '),'": "',' ="'),'", "','" '),'}',' />') as xml)
.value('(/it/#primaryTown)[1]','varchar(20)')
This will return :
AKERdwfUS
This breaks with the nested addressCountry object though. If you know what the JSON text contains, you could cheat and replace specific attributes, not just separators, eg :
declare #json varchar(max)='{"addressLine1": "UNIT 56", "addressLine2": "22 XDFR STREET", "primaryTown": "MANLY VALE", "provinceOrState": "NSW", "addressCountry": {"isoAlpha2Code": "AU"}}'
select
cast(
replace(
replace(
replace(
replace(
replace(
replace(#json,'", "addressCountry": {"','"><addressCountry '),
'}}','/></it>'),
'{"','<it '),
'": "',' ="'),
'", "','" ')
,'}',' />')
as xml).value('(/it/addressCountry/#isoAlpha2Code)[1]','varchar(20)')
This returns AU.
That's some serious cheating though that can only work through trial and error. In this case, the addressCountry attribute and the following separators are converted to an element. }} is expected to appear only at the end of the string, so it gets special treatment.
Use a client script
It's probably easier to use a small .NET program to read the data using JSON.NET and extract the desired values. 2M rows aren't a lot so parsing the data once in a while won't be a big problem

Resources