Related
I am creating an indexer that takes a document, runs the KeyPhraseExtractionSkill and outputs it back to the index.
For many documents, this works out of the box. But for those records which are over 50,000, this does not work. OK, no problem; this is clearly stated in the docs.
What the docs suggest is so use the Text Split Skill. What I've done is use the Text Split skill, split the original document into pages, pass all pages to the KeyPhraseExtractionSkill. Then we need to merge them back, as we'd end up with an array of arrays of strings. Unfortunately, it seems that the Merge Skill does not accept an array of arrays, just an array.
https://i.imgur.com/dBD4qgb.png <- Link to the skillset hierarchy.
This is the error reported by Azure:
Required skill input was not of the expected type 'StringCollection'. Name: 'itemsToInsert', Source: '/document/content/pages/*/keyPhrases'. Expression language parsing issues:
What I want to achieve in the end of the day is to run the KeyPhraseExtractionSkill for text which is larger than 50,000 to add it back to the index eventually.
JSON for skillset
"#odata.context": "https://-----------.search.windows.net/$metadata#skillsets/$entity",
"#odata.etag": "\"0x8D957466A2C1E47\"",
"name": "devalbertcollectionfilesskillset2",
"description": null,
"skills": [
{
"#odata.type": "#Microsoft.Skills.Text.SplitSkill",
"name": "SplitSkill",
"description": null,
"context": "/document/content",
"defaultLanguageCode": "en",
"textSplitMode": "pages",
"maximumPageLength": 1000,
"inputs": [
{
"name": "text",
"source": "/document/content"
}
],
"outputs": [
{
"name": "textItems",
"targetName": "pages"
}
]
},
{
"#odata.type": "#Microsoft.Skills.Text.EntityRecognitionSkill",
"name": "EntityRecognitionSkill",
"description": null,
"context": "/document/content/pages/*",
"categories": [
"person",
"quantity",
"organization",
"url",
"email",
"location",
"datetime"
],
"defaultLanguageCode": "en",
"minimumPrecision": null,
"includeTypelessEntities": null,
"inputs": [
{
"name": "text",
"source": "/document/content/pages/*"
}
],
"outputs": [
{
"name": "persons",
"targetName": "people"
},
{
"name": "organizations",
"targetName": "organizations"
},
{
"name": "entities",
"targetName": "entities"
},
{
"name": "locations",
"targetName": "locations"
}
]
},
{
"#odata.type": "#Microsoft.Skills.Text.KeyPhraseExtractionSkill",
"name": "KeyPhraseExtractionSkill",
"description": null,
"context": "/document/content/pages/*",
"defaultLanguageCode": "en",
"maxKeyPhraseCount": null,
"modelVersion": null,
"inputs": [
{
"name": "text",
"source": "/document/content/pages/*"
}
],
"outputs": [
{
"name": "keyPhrases",
"targetName": "keyPhrases"
}
]
},
{
"#odata.type": "#Microsoft.Skills.Text.MergeSkill",
"name": "Merge Skill - keyPhrases",
"description": null,
"context": "/document",
"insertPreTag": " ",
"insertPostTag": " ",
"inputs": [
{
"name": "itemsToInsert",
"source": "/document/content/pages/*/keyPhrases"
}
],
"outputs": [
{
"name": "mergedText",
"targetName": "keyPhrases"
}
]
}
],
"cognitiveServices": {
"#odata.type": "#Microsoft.Azure.Search.CognitiveServicesByKey",
"key": "------",
"description": "/subscriptions/13abe1c6-d700-4f8f-916a-8d3bc17bb41e/resourceGroups/mde-dev-rg/providers/Microsoft.CognitiveServices/accounts/mde-dev-cognitive"
},
"knowledgeStore": null,
"encryptionKey": null
}```
Please let me know if there is anything else that I can add to improve the question. Thanks!
[1]: https://i.stack.imgur.com/GNf7F.png
You don't have to merge the key phrase outputs to insert them to the index.
Assuming your index already has a field called mykeyphrases of type Collection(Edm.String), to populate it with the key phrase outputs, add this indexer output field mapping:
"outputFieldMappings": [
...
{
"sourceFieldName": "/document/content/pages/*/keyPhrases/*",
"targetFieldName": "mykeyphrases"
},
...
]
The /* at the end of sourceFieldName is important to flattening the array of arrays of strings. This will also work as the skill input if you want to pass an array of strings to another skill for other enrichments.
This is my first attempt at parsing nested JSON with Ruby. I need to go through the JSON to pull out specific values for "_id", "name", and "type" for instance. I then need to create a reference table so that I can refer to each "_id" and associated information. I also need to combine information from multiple JSON responses. I've been able to get basic information and have tried a few things I've found online. I just need a little assistance with a starting point. If anyone has any ideas of where to start with this I'd really appreciate it.
Devices JSON response hash. Each device starts with _id.
{
"api": "1.0",
"error": null,
"id": "60b5d4c3077862123cfa4443",
"result": {
"devices": [
{
"_id": "123456787786211fd31f3dd",
"batteryPowered": true,
"category": "door_lock",
"deviceTypeId": "144_1_1",
"firmware": [
{
"id": "us.144.1_1.0",
"version": "2.6"
}
],
"gatewayId": "1234567807786214fbc6bd4e",
"info": {
"firmware.stack": "3.28",
"hardware": "0",
"manufacturer": "Kwikset",
"model": "912",
"protocol": "zwave",
"zwave.node": "2",
"zwave.smartstart": "no"
},
"name": "Garage Door",
"parentDeviceId": "",
"persistent": false,
"reachable": false,
"ready": true,
"roomId": "1234567807786211fd31f3eb",
"security": "middle",
"status": "idle",
"subcategory": "",
"type": "doorlock"
},
{
"_id": "1234567897786211fd31f3ed",
"batteryPowered": true,
"category": "door_lock",
"deviceTypeId": "59_1_1129",
"firmware": [
{
"id": "us.59.18064.0",
"version": "3.3"
},
{
"id": "us.59.18065.1",
"version": "11.0"
}
],
"gatewayId": "1234567897786214fbc6bd4e",
"info": {
"firmware.stack": "6.3",
"hardware": "3",
"manufacturer": "Schlage",
"model": "BE469ZP",
"protocol": "zwave",
"zwave.node": "3",
"zwave.smartstart": "no"
},
"name": "Front Door",
"parentDeviceId": "",
"persistent": false,
"reachable": true,
"ready": true,
"roomId": "1234567807786211fd31f3ec",
"security": "high",
"status": "idle",
"subcategory": "",
"type": "doorlock"
},
{
"_id": "1234567897786211fd31f40a",
"batteryPowered": false,
"category": "switch",
"deviceTypeId": "57_20562_12344",
"firmware": [
{
"id": "us.57.29240.0",
"version": "5.25"
}
],
"gatewayId": "1234567807786214fbc6bd4e",
"info": {
"firmware.stack": "4.54",
"hardware": "255",
"manufacturer": "Honeywell",
"model": "ZW4103/39337",
"protocol": "zwave",
"zwave.node": "4",
"zwave.smartstart": "no"
},
"name": "Lamp Switch",
"parentDeviceId": "",
"persistent": false,
"reachable": true,
"ready": true,
"roomId": "1234567807786211fd31f416",
"security": "no",
"status": "idle",
"subcategory": "interior_plugin",
"type": "switch.outlet"
},
{
"_id": "1234567b07786211fd31f40e",
"batteryPowered": false,
"category": "dimmable_light",
"deviceTypeId": "57_20548_12339",
"firmware": [
{
"id": "us.57.29747.0",
"version": "5.21"
}
],
"gatewayId": "1234567d07786214fbc6bd4e",
"info": {
"firmware.stack": "4.34",
"hardware": "255",
"manufacturer": "Honeywell",
"model": "39339/ZW3107",
"protocol": "zwave",
"zwave.node": "5",
"zwave.smartstart": "no"
},
"name": "Lamp Dimmer",
"parentDeviceId": "",
"persistent": false,
"reachable": true,
"ready": true,
"roomId": "1234567807786211fd31f416",
"security": "no",
"status": "idle",
"subcategory": "dimmable_plugged",
"type": "dimmer.outlet"
}
]
}
}
There is then also a JSON response that lists the functions for each device in the same format above. However instead of "devices"=> it is "items"=> and the beach function is the _id key again.
I'd like to combine function _id tags and descriptions with the device JSON, so I can create a way to send my script "unlock door lock 1" and it subs the number with the _id of the device and the function _id.
You can start with a very rough navigator function like this:
def find_device(data, name, index)
# Filter through the device list...
data['result']['devices'].select do |device|
# ...for matching names.
device.name == name
end[index] # Take indexed entry
end
Where now you can do find_device(data, 'door_lock', 0) to dig up that entry.
Converting "door lock 1" to [ 'door_lock', 0 ] should be pretty trivial:
def to_location(str)
# Split off the name component(s) and index number
*name, index = str.split(/\s+/)
# Reassemble with underscores and -1 to account for 0-index
[ name.join('_'), index.to_i - 1 ]
end
I am making a discord webhook, and I want to know all the different things I can add to it.
As far as styling it goes; I only have a color and a message. (My code is not included besides that, just the code for the color and the message)
.setTitle('Biscuit AIO Cooked!')
.setColor('#84e1f4');
If anyone can link me a site or reply to me with all of the things I can add to the webhook, please do so :)
Read about message embeds and embed visualizer.
At last one of the features of a webhook is that you can send several embeds at once message. So code will looks like this.
{
"content": "this `supports` __a__ **subset** *of* ~~markdown~~ 😃 ```js\nfunction foo(bar) {\n console.log(bar);\n}\n\nfoo(1);```",
"embeds": [
{
"title": "title ~~(did you know you can have markdown here too?)~~",
"description": "this supports [named links](https://discordapp.com) on top of the previously shown subset of markdown. ```\nyes, even code blocks```",
"url": "https://discordapp.com",
"color": 11038012,
"timestamp": "2020-07-03T15:05:41.392Z",
"footer": {
"icon_url": "https://cdn.discordapp.com/embed/avatars/0.png",
"text": "footer text"
},
"thumbnail": {
"url": "https://cdn.discordapp.com/embed/avatars/0.png"
},
"image": {
"url": "https://cdn.discordapp.com/embed/avatars/0.png"
},
"author": {
"name": "author name",
"url": "https://discordapp.com",
"icon_url": "https://cdn.discordapp.com/embed/avatars/0.png"
},
"fields": [
{
"name": "🤔",
"value": "some of these properties have certain limits..."
},
{
"name": "😱",
"value": "try exceeding some of them!"
},
{
"name": "🙄",
"value": "an informative error should show up, and this view will remain as-is until all issues are fixed"
},
{
"name": "<:thonkang:219069250692841473>",
"value": "these last two",
"inline": true
},
{
"name": "<:thonkang:219069250692841473>",
"value": "are inline fields",
"inline": true
}
]
},
{
"title": "title ~~(did you know you can have markdown here too?)~~",
"description": "this supports [named links](https://discordapp.com) on top of the previously shown subset of markdown. ```\nyes, even code blocks```",
"url": "https://discordapp.com",
"color": 11038012,
"timestamp": "2020-07-03T15:05:41.392Z",
"footer": {
"icon_url": "https://cdn.discordapp.com/embed/avatars/0.png",
"text": "footer text"
},
"thumbnail": {
"url": "https://cdn.discordapp.com/embed/avatars/0.png"
},
"image": {
"url": "https://cdn.discordapp.com/embed/avatars/0.png"
},
"author": {
"name": "author name",
"url": "https://discordapp.com",
"icon_url": "https://cdn.discordapp.com/embed/avatars/0.png"
},
"fields": [
{
"name": "🤔",
"value": "some of these properties have certain limits..."
},
{
"name": "😱",
"value": "try exceeding some of them!"
},
{
"name": "🙄",
"value": "an informative error should show up, and this view will remain as-is until all issues are fixed"
},
{
"name": "<:thonkang:219069250692841473>",
"value": "these last two",
"inline": true
},
{
"name": "<:thonkang:219069250692841473>",
"value": "are inline fields",
"inline": true
}
]
}
]
}
This is my first time using Discord embeds, let alone, making a webhook. Having a little bit of trouble trying to figure out what I'm doing wrong. Hopefully someone can help!
When I try to send an embed with a message with this code:
{
"username": "Official Discord Servers",
"avatar_url": "https://imgur.com/a/Cl3zspb",
"embeds": [{
"description": "Here is a list of official Discord servers maintained by our State. Note that most of these servers may restrict full access to those who are members of their respective departments:",
"color": 16007990,
"fields": [
{
"name": "State of Atoll",
"value": "[Discord Server](https://discord.gg/5KKgb2z)"
},
{
"name": "Atoll Law Enforcement Training Academy",
"value": "[Discord Server](https://discord.gg/GJnbavz)"
},
{
"name": "Atoll Department of Homeland Security",
"value": "[Discord Server](https://discord.gg/acnPFa7)"
},
{
"name": "Palm County Sheriff's Office",
"value": "[Discord Server](https://discord.gg/uYFANDs)"
},
{
"name": "Atoll State Police",
"value": "[Discord Server](https://discord.gg/WNHKtra)"
},
{
"name": "Atoll Department of Corrections",
"value": "[Discord Server](https://discord.gg/VbFew9s)"
},
{
"name": "Atoll National Guard",
"value": "[Discord Server](https://discord.gg/6zE9HyW)"
}]
}
I get this error:
{
"code": 50006,
"message": "Cannot send an empty message"
}
You need to send "content" as a key value pair containing the content of the message being sent.
"content": "Just testing webhooks"
It requires header Content-Type with value application/json. Also try setting content field.
Your issue is invalid JSON, so try using JSON validator
The JSON referred in the question cannot be parsed, it is not properly formed, it is missing closing brackets. Please find last set of }] brackets in my answer below, try this,
{
"username": "Official Discord Servers",
"avatar_url": "https://imgur.com/a/Cl3zspb",
"embeds": [{
"description": "Here is a list of official Discord servers maintained by our State. Note that most of these servers may restrict full access to those who are members of their respective departments:",
"color": 16007990,
"fields": [
{
"name": "State of Atoll",
"value": "[Discord Server](https://discord.gg/5KKgb2z)"
},
{
"name": "Atoll Law Enforcement Training Academy",
"value": "[Discord Server](https://discord.gg/GJnbavz)"
},
{
"name": "Atoll Department of Homeland Security",
"value": "[Discord Server](https://discord.gg/acnPFa7)"
},
{
"name": "Palm County Sheriff's Office",
"value": "[Discord Server](https://discord.gg/uYFANDs)"
},
{
"name": "Atoll State Police",
"value": "[Discord Server](https://discord.gg/WNHKtra)"
},
{
"name": "Atoll Department of Corrections",
"value": "[Discord Server](https://discord.gg/VbFew9s)"
},
{
"name": "Atoll National Guard",
"value": "[Discord Server](https://discord.gg/6zE9HyW)"
}]
}]
}
In my case, I was sending a plain text directly instead of JSON in the request. Then when I changed the message to the below it worked.
{content: "This is the message"}
If you are sending embeds, you can still keep the content set to null.
I have this schema for a json response
{
"title": "Products",
"description": "schema for products",
"type": "array",
"properties": {
"id": {
"description": "id of a product",
"type": "integer"
},
"name": {
"description": "name of the product",
"type": "string"
},
"created_at": {
"description": "record created_at",
"type": "string",
"format": "date-time"
},
"updated_at": {
"description": "record updated_at",
"type": "string",
"format": "date-time"
}
},
"required": ["id", "name"]
}
and I want to match this schema with this json
[{
"id": 1,
"name": "Cricket Ball"
}, {
"id": 2,
"name": "Soccer Ball"
}, {
"id": 3,
"name": "football ball"
}, {
"id": 4,
"name": "Basketball ball"
}, {
"id": 5,
"name": "Table Tennis ball"
}, {
"id": 6,
"name": "Tennis ball"
}]
This schema matches the response but it also matches the schema in which the required field is this
"required": ["ids", "names"]
I think the schema is validated against the array and the objects in the array are not validated.
The way you have it set up now, your properties key refers to the array itself, not to each item, and is being ignored (because arrays don't have properties, they just have items). You need to use the items key to validate each item in the array, like so:
{
"title": "Products",
"description": "schema for products",
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"description": "id of a product",
"type": "integer"
},
"name": {
"description": "name of the product",
"type": "string"
},
"created_at": {
"description": "record created_at",
"type": "string",
"format": "date-time"
},
"updated_at": {
"description": "record updated_at",
"type": "string",
"format": "date-time"
}
},
"required": ["id", "name"]
}
}
try map
new_array = response.map{ |k| { 'id': k['properties']['id']['description'], 'name': k['properties']['name']['description'] } }