Is there a way to split a snowpark dataframe column based on a string?
Here is what I have tried so far
from snowflake.snowpark import Session
from snowflake.snowpark import functions as SF
connection_parameters = {
"account": "myaccount",
"user": "myuser",
}
session = Session.builder.configs(connection_parameters).create()
dh_session = session.table('tableName')
dh = dh_session.select(SF.to_timestamp(SF.col("timestamp")).as_("timestamp"),SF.col("name"))
# Split the name column by string delimiter '-AA' and get the first part
dh.select(SF.split("name",SF.lit("-AA").get_value(0)).as_("test")).show()
However I get an error message
AttributeError: 'Column' object has no attribute 'getItem'
Thanks
Instead of using get_value, try using get like so,
from snowflake.snowpark import Session
from snowflake.snowpark import functions as SF
connection_parameters = {
"account": "myaccount",
"user": "myuser",
}
session = Session.builder.configs(connection_parameters).create()
dh_session = session.table('tableName')
dh = dh_session.select(SF.to_timestamp(SF.col("timestamp")).as_("timestamp"),SF.col("name"))
# Split the name column by string delimiter '-AA' and get the first part
dh = dh.select(SF.split("name",SF.lit("-AA")).as_("split_name"))
dh.select(SF.get(dh.split_name, SF.lit(0)).as_("name[0]")).show()
Example of first using a filter then select with sample dataframe. Note this uses a 'like' match which may or may not be an accurate regex for your data.
from snowflake.snowpark import Session
from snowflake.snowpark import functions as SF
connection_parameters = {
"account": "myaccount",
"user": "myuser",
}
session = Session.builder.configs(connection_parameters).create()
##
## Create a dummy dataframe
##
dh = session.create_dataframe([["morning-CC", 2], ["hello-AA", 4],
["bye-BB", 7], ["another-AA", 3]], schema=["name", "col2"])
dh.filter("name like '%-AA%'").select(col("name"),
SF.split(col("name"),SF.lit("-"))[0].alias("test")).show()
"NAME"
"TEST"
hello-AA
"hello"
another-AA
"another"
Related
I need to be able to do the following in Postman
Parse through a JSON response
Retrieve specific values: some nested, some not
(optional) Concatenate the final result
I can currently perform 1 and I can accomplish 2 for one specific value with the help of this explanation link
What I can't do is retrieve two values. Below is the sample response I am working with
{
"part": "9FH74T00",
"summaries": [
{
"id": "A1AGQF32TR"
}
]
}
I was able to modify the JS from the link above to extract the nested id value
var response = JSON.parse(responseBody);
var ids = response.summaries.map(function(summary) {
return summary.id;
});
console.log(ids);
How can I expand on the JS to also retrieve the part value and ideally output that in the console as part = id or 9FH74T00 = A1AGQF32TR
#lucas-nguyen's seems a valid response/comment to what you propose. What is not well explained is the model. summaries is an array so would you get multiple ids? The fact that you use a map makes me thing there will be multiple ids.
part = id1,id2,id3
or
part = id1
part = id2
part = id3
I've created a sample request with your payload and two ids in summaries. I've console.log 3 approaches:
single part and single id
single part and ids in a list
single part and ids in tuples
So with the example response:
{
"part": "9FH74T00",
"summaries": [
{
"id": "A1AGQF32TR"
},
{
"id": "SECONDID"
}
]
}
You would get the following output:
Option 1:
9FH74T00 = A1AGQF32TR
Option 2, list of ids:
9FH74T00 = A1AGQF32TR,SECONDID
Option 3, list of ids:
9FH74T00 = A1AGQF32TR
9FH74T00 = SECONDID
If for some reason you can't open the example, this is my test tab:
const response = pm.response.json().data;
// Just concatenate single part and single id
console.log("\nOption 1:")
console.log(`${response.part} = ${response.summaries[0].id}`)
// Co
console.log("\nOption 2, list of ids:")
const ids = response.summaries.map( x => x.id)
console.log(`${response.part} = ${ids.join(",")}`)
console.log("\nOption 3, list of ids:")
ids.forEach( x => console.log(`${response.part} = ${x}`))
Example:
API A:
{
"customer":[
{
"name":"Jane",
"phone":"9999999",
"email":"jane#test.com"
},
{
"name":"John",
"phone":"8888888",
"email":"john#test.com"
},
{
"name":"Joe",
"phone":"7777777",
"email":"Joe#test.com"
}
]
}
Using the JSON extractor, I want to get the names of all the customers
so: Jane, John, Joe
How do I get these values and turn them into an array
[{"name":"Jane", "name":"John", "name":"Joe"}]
And pass it onto the next API?
Note: That it has to be dynamic so API A could show different 2 names or 1 name or more and needs to be adjusted into the array
First of all your [{"name":"Jane", "name":"John", "name":"Joe"}] is not a valid JSON, you can check it yourself:
so I strongly doubt that this is the string you need to generate.
So if you really need to construct this value you can do something like:
Add JSR223 PostProcessor as a child of the request which returns this "customers" data
Put the following code into "Script" area:
def response = new groovy.json.JsonSlurper().parse(prev.getResponseData())
def payload = new StringBuilder()
payload.append('[{')
0.upto(response.customer.size - 1, { index ->
payload.append('"name": "').append(response.customer[index].name).append('"')
if (index != response.customer.size - 1) {
payload.append(',')
}
})
payload.append('}]')
vars.put('payload', payload as String)
Refer the generated value as ${payload} where required
Demo:
More information:
JsonSlurper
Apache Groovy - Parsing and producing JSON
Apache Groovy - Why and How You Should Use It
I'm a Python newbie and I'm trying to write a script to extract json keys by passing the keys dinamically, reading them from a csv.
First of all this is my first post and I'm sorry if my questions are banals and if the code is incomplete but it's just a pseudo code to understand the problem (I hope not to complicate it...)
The following partial code retrieves the values from three key (group, user and id or username) but I'd like to load the objects and key from a csv to make them dinamicals.
Input json
{
"fullname": "The Full Name",
"group": {
"user": {
"id": 1,
"username": "John Doe"
},
"location": {
"x": "1234567",
"y": "9876543"
}
},
"color": {
"code": "ffffff",
"type" : "plastic"
}
}
Python code...
...
url = urlopen(jsonFile)
data = json.loads(url.read())
id = (data["group"]["user"]["id"])
username = (data["group"]["user"]["username"])
...
File.csv loaded into an array. Each line contains one or more keys.
fullname;
group,user,id;
group,user,username;
group,location,x;
group,location,y;
color,code;
The questions are: can I use a variable containing the object or key to be extract?
And how can I specify how many keys there are in the keys array to put them into the data([ ][ ]...) using only one line?
Something like this pseudo code:
...
url = urlopen(jsonFile)
data = json.loads(url.read())
...
keys = line.split(',')
...
# using keys[] to identify the objects and keys
value = (data[keys[0]][keys[1]][keys[2]])
...
But the line value = (data[keys[0]][keys[1]][keys[2]]) should have the exact number of the keys per line read from the csv.
Or I must to make some "if" lines like these?:
...
if len(keys) == 3:
value = (data[keys[0]][keys[1]][keys[2]])
if len(keys) == 2:
value = (data[keys[0]][keys[1]])
...
Many thanks!
I'm not sure I completely understand your question, but I would suggest you to try and play with pandas. It might be as easy as this:
import pandas as pd
df = pd.read_json(<yourJsonFile>, orient='columns')
name = df.fullname[0]
group_user = df.group.user
group_location = df.group.location
color_type = df.color.type
color_code = df.color.code
(Where group_user and group_location will be python dictionaries).
I want to extract below json and use values accordingly.
I/p JSON:-
{
"status": "Success",
"message": "User created successfully",
"id": [
131188,
131191
]
}
Here I want values of id field. I used JSON Extractor and gave expression as $.id which gives me [131188,131191] in a variable. Now I want to use individual values out of this array i.e. 131188 and 131191.
Any Idea how to do it?
Update : I don't want to use 2 JSON Extractors.
Just add [*] to your JSON path expression as below
$.id[*]
This will create a jmeter variable for each value.Note that you should use -1 in the match numbers field.
You could use a json extractor and a "JSR223 PostProcessor" with groovy language. An example:
import groovy.json.JsonSlurper
//String jsonString = vars.get("jsonFromExtractor")
String jsonString = '''
{
"status": "Success",
"message": "User created successfully",
"id": [
131188,
131191
]
}
'''
log.info("jsonString:" + jsonString)
def json = new JsonSlurper().parseText( jsonString )
String idValue1 = json.get("id").get(0)
String idValue2 = json.get("id").get(1)
log.info("idValue1:" + idValue1)
log.info("idValue2:" + idValue2)
I hope this helps
How can I pass an object like a list or dictionary as an argument in behave .feature file so I can use that argument in my python function step? See an example of what I'm trying to achieve below:
Feature:
Scenario: Given the inputs below
Given a "<Dictionary>" and "<List>"
When we insert "<Dictionary>" and "<List>"
Then we confirm the result in the database
Examples: Input Variables
|Input1 |Input2 |
|Dictionary(json) |List |
You can provide the data as json, and parse it using json.loads in the steps.
Note, to use Examples: we need a Scenario Outline instead of a
Scenario.
# features/testing_objects.feature
Feature: Testing objects
Scenario Outline: Given the inputs below
Given a <Dictionary> and <List>
When we insert them
Then we confirm the result in the database
Examples: Input Variables
|Dictionary |List |
|{"name": "Fred", "age":2} |[1,2,"three"]|
Parse it using json.loads in the steps:
# features/steps/steps.py
import json
from behave import given, when, then
#given('a {dictionary} and {a_list}')
def given_dict_and_list(context, dictionary, a_list):
context.dictionary = json.loads(dictionary)
context.a_list = json.loads(a_list)
#when('we insert them')
def insert_data(context):
print('inserting dictionary', context.dictionary)
print('inserting list', context.a_list)
#then('we confirm the result in the database')
def confirm(context):
print('checking dictionary', context.dictionary)
print('checking list', context.a_list)
Instead of using Examples: you could also use a multi-line string literal
and then access each object in a separate step, via context.text.
Feature: String literal JSON
Scenario:
Given a dictionary
"""
{
"name": "Fred",
"age": 2
}
"""
And a list
"""
[1, 2, "three"]
"""
Then we can check the dictionary
And check the list
#given('a dictionary')
def given_a_dictionary(context):
context.dictionary = json.loads(context.text)
#given('a list')
def given_a_list(context):
context.a_list = json.loads(context.text)
#then('we can check the dictionary')
def check_the_dictionary(context):
assert context.dictionary == {
'name': 'Fred',
'age': 2
}
#then('check the list')
def check_the_list(context):
assert context.a_list == [1, 2, 'three']