I have a table in PostgreSQL that has array column like abc_id varchar[](1) and it has data in it like {A,B,C}.
I want to create that table in Synapse SQL how can I create array columns is it possible?
There is no explicit array type ibn Synapse SQL. We consider the array type data as complex type. As mentioned in the comment, consider using JSON as the main base.
The below image will explain the scheme of array and nested structure.
The following code block is the sample JSON example of implementing nested structures, which is a complex type and acceptable in Synapse SQL.
{
"id": "66532691-ab20-11ea-8b1d-936b3ec64e54",
"context": {
"data": {
"eventTime": "2020-06-10T13:43:34.553Z",
"samplingRate": "100.0",
"isSynthetic": "false"
},
"session": {
"isFirst": "false",
"id": "38619c14-7a23-4687-8268-95862c5326b1"
},
"custom": {
"dimensions": [
{
"customerInfo": {
"ProfileType": "ExpertUser",
"RoomName": "",
"CustomerName": "diamond",
"UserName": "XXXX#yahoo.com"
}
},
{
"customerInfo": {
"ProfileType": "Novice",
"RoomName": "",
"CustomerName": "topaz",
"UserName": "XXXX#outlook.com"
}
}
]
}
}
}
Related
I have a collection in my MongoDB that contains one field that is an array.
Refer to the data above, the field 'Courses' is an array.
The JSON format of the data is like this:
{
"_id": {
"$oid": "60eb59b98a970a20865142e8"
},
"Name": "Sadia",
"Age": 24,
"Institute": "IBA",
"Courses": [{
"Name": "ITP",
"Grade": "A-"
}, {
"Name": "OOP",
"Grade": "A-"
}]
}
I am aware that there is a way in case its an object, but could not find a way on how to read this data using Talend since it contains an array.
I am having trouble with querying a MongoDB collection with an array inside.
Here is the structure of my collection that I am querying. This is one record:
{
"_id": "abc123def4567890",
"profile_id": "abc123def4567890",
"image_count": 2,
"images": [
{
"image_id": "ABC123456789",
"image_url": "images/something.jpg",
"geo_loc": "-0.1234,11.234567890",
"title": "A Title",
"shot_time": "01:23:33",
"shot_date": "11/22/2222",
"shot_type": "scenery",
"conditions": "cloudy",
"iso": 16,
"f": 2.4,
"ss": "1/545",
"focal": 6.0,
"equipment": "",
"instructions": "",
"upload_date": 1234567890,
"update_date": 1234567890
},
{
"image_id": "ABC123456789",
"image_url": "images/something.jpg",
"geo_loc": "-0.1234,11.234567890",
"title": "A Title",
"shot_time": "01:23:33",
"shot_date": "11/22/2222",
"shot_type": "portrait",
"conditions": "cloudy",
"iso": "16",
"f": "2.4",
"ss": "1/545",
"focal": "6.0",
"equipment": "",
"instructions": "",
"upload_date": 1234567890,
"update_date": 1234567890
}
]
}
Forgive the formatting, I didn't know how else to show this.
As you can see, it's a profile with a series of images within an array called 'images' and there are 2 images. Each of the 'images' array items contain an object of attributes for the image (url, title, type, etc).
All I want to do is to return the object element whose attributes match certain criteria:
Select object from images which has shot_type = "scenery"
I tried to make it as simple as possible so i started with:
find( { "images.shot_type": "scenery" } )
This returns the entire record and both the images within. So I tried projection but I could not isolate the single object within the array (in this case object at position 0) and return it.
I think the answer lies with projection but I am unsure.
I have gone through the MongoDB documents for hours now and can't find inspiration. I have read about $elemMatch, $, and the other array operators, nothing seems to allow you to single out an array item based on data within. I have been through this page too https://docs.mongodb.com/manual/tutorial/query-arrays/ Still can't work it out.
Can anyone provide help?
Have I made an error by using '$push' to populate my images field (making it an array) instead of using '$set' which would have made it into an embedded document? Would this have made a difference?
Using aggregation:
db.collection.aggregate({
$project: {
_id: 0,
"result": {
$filter: {
input: "$images",
as: "img",
cond: {
$eq: [
"$$img.shot_type",
"scenery"
]
}
}
}
}
})
Playground
You can use $elemMatch in this way (simplified query):
db.collection.find({
"profile_id": "1",
},
{
"images": {
"$elemMatch": {
"shot_type": 1
}
}
})
You can use two objects into find query. The first will filter all document and will only get those whose profile_id is 1. You can omit this stage and use only { } if you wnat to search into the entire collection.
Then, the other object uses $elemMatch to get only the element whose shot_type is 1.
Check an example here
I need to create a JSON schema for data that comes as an array directly within the root object, unnamed. An MWE for this kind of JSON would be:
{
[
{
"veggieName": "potato",
"veggieLike": true
},
{
"veggieName": "broccoli",
"veggieLike": false
}
]
}
I have seen examples for schemas which validate such an array which is not nested in an object. I have also seen examples which work when the array is named, for example
{
vegetables : [
{
"veggieName": "potato",
"veggieLike": true
},
{
"veggieName": "broccoli",
"veggieLike": false
}
]
}
This second example can be validated by the schema
{
"$id": "https://example.com/arrays.schema.json",
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "A representation of a person, company, organization, or place",
"type": "object",
"properties": {
"vegetables": {
"type": "array",
"items": { "$ref": "#/definitions/veggie" }
}
},
"definitions": {
"veggie": {
"type": "object",
"required": [ "veggieName", "veggieLike" ],
"properties": {
"veggieName": {
"type": "string",
"description": "The name of the vegetable."
},
"veggieLike": {
"type": "boolean",
"description": "Do I like this vegetable?"
}
}
}
}
}
But the problem is, as soon as the name "vegetables" is removed, I was not able to find a way to define a valid schema. How do I properly represent my data structure in a schema?
(MWEs derived from http://json-schema.org/learn/miscellaneous-examples.html).
The schema you are looking for is the following:
{
"$id":"https://example.com/arrays.schema.json",
"$schema":"http://json-schema.org/draft-07/schema#",
"description":"A representation of a person, company, organization, or place",
"type":"array",
"items":{
"type":"object",
"required":[
"veggieName",
"veggieLike"
],
"properties":{
"veggieName":{
"type":"string",
"description":"The name of the vegetable."
},
"veggieLike":{
"type":"boolean",
"description":"Do I like this vegetable?"
}
}
}
}
You also need to modify your base array instance, your original one (the "unnamed" array) was not valid JSON:
[
{
"veggieName":"potato",
"veggieLike":true
},
{
"veggieName":"broccoli",
"veggieLike":false
}
]
Unlike XML, where you are allowed a single root node per document only, in JSON you can have either a type or an array as a root type.
Imagine the following collection of city records:
{
"city": "London",
"inhabitants": [
{
"id": "34543534",
"user": {
"name": "Jonathan Deer",
"email": "john#btinternet.com"
}
},
{
"id": "0454534",
"user": {
"name": "Tanya Patel",
"email": "tanya#btinternet.com"
}
},
{
"id": "4345345",
"user": {
"name": "Catherine King",
"email": "catherine#gmail.com"
}
}
]
}
{
"city": "Manchester",
"inhabitants": [
{
"id": "980003",
"user": {
"name": "Benjamin Thaw",
"email": "benny#btinternet.com"
}
},
{
"id": "734488",
"user": {
"name": "Craig Longstone",
"email": "craig#gmail.com"
}
},
{
"id": "4400093",
"user": {
"name": "Arnold Greentree",
"email": "arnold#btinternet.com"
}
},
]
},
What I'm trying to do is loop through each inhabitants array of each city, and see if any of the people there has an email address containing btinternet.com in it. For those users I want to sent a new flag isBT: true and for everyone else (e.g., gmail.com users) isBT: false:
"user": {
"name": "Tanya Patel",
"email": "tanya#btinternet.com"
"isBT" true
}
I tried the following queries - first query sets all of them to isBT: false while the second one searches for "btinternet.com" in email address and sets isBT: true:
db.city.update({ "inhabitants.user.email": {$exists: true}}, {$set: { "inhabitants.$.user.isBT": false}}, {multi: true})
db.city.update({ "inhabitants.user.email": {$regex: "btinternet.com"}}, {$set: { "inhabitants.$.user.isBT": true}}, {multi: true})
The problem is that when I execute the second query, there are several inhabitants records that are left with isBT: false even though they contain the necessary "btinternet.com" email address. It almost seems like only the first user record that matches the criteria gets updated... Is there a way to update ALL user records for all "inhabitants" arrays?
I looked at using the positional operator $[], but our DB is on version 2.6.3 but this operator was introduced only in 3.6...
The short answer is "no".
The long answer is "no, because your MongoDB version doesn't support such an operation". You'll need to either...
1. retrieve all matching documents and perform a full array update through server-side processing of the data (e.g. use the MongoDB cursor.forEach()),
2. extend your match for "inhabitants.user.isBT": true (use
$elemMatch) and repeatedly perform the update query until the
number of modified documents is 0 (i.e. there are no more array
elements to update), or
3. update your MongoDB version and any
server-side code that relies on features of the current version that
have changed between 2.6 and 3.6.
Any solution to this problem will require more effort than a single query. There's no getting around it. It's a tough pill to swallow, but there really isn't a nice alternative.
Considering the below bad model, as I am totally new to this.
{
"uid": "some-id",
"database": {
"name": "nameOfDatabase",
"collection": [
{
"name": "nameOfCollection",
"fields": {
"0": "field_1",
"1": "field_2"
}
},
{
"name": "nameOfAnotherCollection",
"fields": {
"0": "field_1"
}
}
]
}
}
I have the collection name (i.e database.collection.name) and I have a few fields to add to it or delete from it (there are some already existing ones under database.collection.fields, I want to add new ones or delete exiting ones).
In short how do I update/delete "fields", when I have the database name and the collection name.
I cannot figure out how to use positional operator $ in this context.
Using mongoose update as
Model.update(conditions, updates, options, callback);
I don't know what are correct conditions and correct updates parameters.
So far I have unsuccessfully used the below for model.update
conditions = {
"uid": req.body.uid,
"database.name": "test",
"database.collection":{ $elemMatch:{"name":req.body.collection.name}}
};
updates = {
$set: {
"fields": req.body.collection.fields
}
};
---------------------------------------------------------
conditions = {
"uid": req.body.uid,
"database.name": "test",
"database.collection.$.name":req.body.collection.name
};
updates = {
$addToSet: {
"fields": req.body.collection.fields
}
};
I tried a lot more but none did work, as I am totally new.
I am getting confused between $push, $set, $addToSet, what to use what not to?, how to?
The original schema is supposed to be as show below, but running queries on it is getting harder n harder.
{
"uid": "some-id",
"database": [
{ //array of database objects
"name": "nameOfDatabase",
"collection": [ //array of collection objects inside respective databases
{
"name": "nameOfCollection",
"fields": { //fields inside a this particular collection
"0": "field_1",
"1": "field_2"
}
}
]
}
]
}