hi below my code example. Is there a way to reduce the number of lines of code and save time?
example.. using function.. i need any other method. please help
enter image description here
Yes. You can get the list by name from globals(). See the example below:
list1 = []
list2 = []
msg = {
'type': 2
}
def addToList(listNumber, msg):
globals()["list" + str(listNumber)].append(msg)
for key, val in msg.items():
if key == 'type':
addToList(val, msg)
print(list1) #[]
print(list2) #[{'type': 2}]
Related
I am writing a Spark 3 UDF to mask an attribute in an Array field.
My data (in parquet, but shown in a JSON format):
{"conditions":{"list":[{"element":{"code":"1234","category":"ABC"}},{"element":{"code":"4550","category":"EDC"}}]}}
case class:
case class MyClass(conditions: Seq[MyItem])
case class MyItem(code: String, category: String)
Spark code:
val data = Seq(MyClass(conditions = Seq(MyItem("1234", "ABC"), MyItem("4550", "EDC"))))
import spark.implicits._
val rdd = spark.sparkContext.parallelize(data)
val ds = rdd.toDF().as[MyClass]
val maskedConditions: Column = updateArray.apply(col("conditions"))
ds.withColumn("conditions", maskedConditions)
.select("conditions")
.show(2)
Tried the following UDF function.
UDF code:
def updateArray = udf((arr: Seq[MyItem]) => {
for (i <- 0 to arr.size - 1) {
// Line 3
val a = arr(i).asInstanceOf[org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema]
val a = arr(i)
println(a.getAs[MyItem](0))
// TODO: How to make code = "XXXX" here
// a.code = "XXXX"
}
arr
})
Goal:
I need to set 'code' field value in each array item to "XXXX" in a UDF.
Issue:
I am unable to modify the array fields.
Also I get the following error if remove the line 3 in the UDF (cast to GenericRowWithSchema).
Error:
Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to MyItem
Question: How to capture Array of Structs in a function and how to return a modified array of items?
Welcome to Stackoverflow!
There is a small json linting error in your data: I assumed that you wanted to close the [] square brackets of the list array. So, for this example I used the following data (which is the same as yours):
{"conditions":{"list":[{"element":{"code":"1234","category":"ABC"}},{"element":{"code":"4550","category":"EDC"}}]}}
You don't need UDFs for this: a simple map operation will be sufficient! The following code does what you want:
import spark.implicits._
import org.apache.spark.sql.Encoders
case class MyItem(code: String, category: String)
case class MyElement(element: MyItem)
case class MyList(list: Seq[MyElement])
case class MyClass(conditions: MyList)
val df = spark.read.json("./someData.json").as[MyClass]
val transformedDF = df.map{
case (MyClass(MyList(list))) => MyClass(MyList(list.map{
case (MyElement(item)) => MyElement(MyItem(code = "XXXX", item.category))
}))
}
transformedDF.show(false)
+--------------------------------+
|conditions |
+--------------------------------+
|[[[[XXXX, ABC]], [[XXXX, EDC]]]]|
+--------------------------------+
As you see, we're doing some simple pattern matching on the case classes we've defined and successfully renaming all of the code fields' values to "XXXX". If you want to get a json back, you can call the to_json function like so:
transformedDF.select(to_json($"conditions")).show(false)
+----------------------------------------------------------------------------------------------------+
|structstojson(conditions) |
+----------------------------------------------------------------------------------------------------+
|{"list":[{"element":{"code":"XXXX","category":"ABC"}},{"element":{"code":"XXXX","category":"EDC"}}]}|
+----------------------------------------------------------------------------------------------------+
Finally a very small remark about the data. If you have any control over how the data gets made, I would add the following suggestions:
The conditions JSON object seems to have no function in here, since it just contains a single array called list. Consider making the conditions object the array, which would allow you to discard the list name. That would simpify your structure
The element object does nothing, except containing a single item. Consider removing 1 level of abstraction there too.
With these suggestions, your data would contain the same information but look something like:
{"conditions":[{"code":"1234","category":"ABC"},{"code":"4550","category":"EDC"}]}
With these suggestions, you would also remove the need of the MyElement and the MyList case classes! But very often we're not in control over what data we receive so this is just a small disclaimer :)
Hope this helps!
EDIT: After your addition of simplified data according to the above suggestions, the task gets even easier. Again, you only need a map operation here:
import spark.implicits._
import org.apache.spark.sql.Encoders
case class MyItem(code: String, category: String)
case class MyClass(conditions: Seq[MyItem])
val data = Seq(MyClass(conditions = Seq(MyItem("1234", "ABC"), MyItem("4550", "EDC"))))
val df = data.toDF.as[MyClass]
val transformedDF = df.map{
case MyClass(conditions) => MyClass(conditions.map{
item => MyItem("XXXX", item.category)
})
}
transformedDF.show(false)
+--------------------------+
|conditions |
+--------------------------+
|[[XXXX, ABC], [XXXX, EDC]]|
+--------------------------+
I am able to find a simple solution with Spark 3.1+ as new features are added in this new Spark version.
Updated code:
val data = Seq(
MyClass(conditions = Seq(MyItem("1234", "ABC"), MyItem("234", "KBC"))),
MyClass(conditions = Seq(MyItem("4550", "DTC"), MyItem("900", "RDT")))
)
import spark.implicits._
val ds = data.toDF()
val updatedDS = ds.withColumn(
"conditions",
transform(
col("conditions"),
x => x.withField("code", updateArray(x.getField("code")))))
updatedDS.show()
UDF:
def updateArray = udf((oldVal: String) => {
if(oldVal.contains("1234"))
"XXX"
else
oldVal
})
I just wanna get all the records from the transaction variable and save it to an array.i tried and all I am getting is this constant error. please help me, I just wanna all models(records) to be saved on an array.
Type '()' cannot conform to 'View'
#State private var transactions: [Transactions] = [Transactions]()
ForEach(transactions, id: \.self) { transaction in
timexxx[0] = transaction.timexx ?? "0"
Text(timexxx[0] ?? "0")
}
enter image description here
Like what #multiverse has suggested
ForEach loop expects some sort of View but you are giving it or attempting to give an "array" (it only wants View)
Here is an updated code where you give the ForEach what it wants and you append to your timexxx array
ForEach(Array(transactions.enumerated()), id: \.offset) { (offset, transaction) in
Text(transaction.timexx ?? "\(offset)")
.onAppear {
timexxx[offset] = transaction.timexx ?? "\(offset)"
}
}
Update
for your question
"how do I do this with a simple "For" loop ? let's say I wanna do this operation in a simple class."
This is how it's done.
I removed the view Text.
for (i, transaction) in transactions.enumerated() {
timexxx[i] = transaction.timexx ?? "0"
}
Ok , so this is an error i faced as well, when i was learning SwiftUI( i am still a beginner), so now we need to understand , what does this error actually means, in this case the ForEach loop expects some sort of View but you are giving it or attempting to give an "array" (it only wants View)....
If you want values to be transferred to an array just simply create a function and do it .....
say you have a class and inside of which you do
#Published var song = [Song]()
then what you do is inside a function like loadData()
objects is the array whose elements you want transferred and most likely those elements belong to a Struct like Song here(if not its even simpler just use what ever type it has like Int, String etc.), this way all your elements will get transferred to song from objects
func loadData() {
song = objects.map {
artist in
Song(album: artist.album, artistImage: artist.artistImage)
}
}
Here i add the simplest possible way to transfer from one array to other
var objects = [1,2,3,4,5]
var song = [Int]()
func loadData() {
song = objects.map { $0 }
}
loadData()
print(song)
//[1,2,3,4,5]
Trying to filter selective index. Fieldid values changes with every build, What do not change is fieldName": "TX.Sessionval.cost". Need to filter out the whole stringval and save into variable
fieldName": "TX.Sessionval.cost"
[
{
"Fieldid": "Fieldid/112",
"fieldName": "TX.Sessionval.cost",
"stringval": "jklah-dw-4c8d-8320-das313s3ASsda|000725N8WuUrfwAS7alP|banker_name"
}
]
tried def slurper = new JsonSlurper() def result = slurper.parseText(response.getResponseBodyContent()) def newf = result.findAll { it.contains("TX.Sessionval.cost") } getting groovy.lang.MissingMethodException: No signature of method: java.util.LinkedHashMap$Entry.contains()
in Postman locator works fine
postman.setEnvironmentVariable("Stdid", jsonData.newFields[112].stringValue)
Your JSON in your question is wrong. Extrapolating from your working
code, this should work:
result.newFields.find{ it.fieldName == "TX.Sessionval.cost" }?.stringValue
This works with me result.newFields.findAll { Map map -> map.get("fieldName").contains("TX.Sessionval.cost") }?.stringValue
I'm a Python newbie and I'm trying to write a script to extract json keys by passing the keys dinamically, reading them from a csv.
First of all this is my first post and I'm sorry if my questions are banals and if the code is incomplete but it's just a pseudo code to understand the problem (I hope not to complicate it...)
The following partial code retrieves the values from three key (group, user and id or username) but I'd like to load the objects and key from a csv to make them dinamicals.
Input json
{
"fullname": "The Full Name",
"group": {
"user": {
"id": 1,
"username": "John Doe"
},
"location": {
"x": "1234567",
"y": "9876543"
}
},
"color": {
"code": "ffffff",
"type" : "plastic"
}
}
Python code...
...
url = urlopen(jsonFile)
data = json.loads(url.read())
id = (data["group"]["user"]["id"])
username = (data["group"]["user"]["username"])
...
File.csv loaded into an array. Each line contains one or more keys.
fullname;
group,user,id;
group,user,username;
group,location,x;
group,location,y;
color,code;
The questions are: can I use a variable containing the object or key to be extract?
And how can I specify how many keys there are in the keys array to put them into the data([ ][ ]...) using only one line?
Something like this pseudo code:
...
url = urlopen(jsonFile)
data = json.loads(url.read())
...
keys = line.split(',')
...
# using keys[] to identify the objects and keys
value = (data[keys[0]][keys[1]][keys[2]])
...
But the line value = (data[keys[0]][keys[1]][keys[2]]) should have the exact number of the keys per line read from the csv.
Or I must to make some "if" lines like these?:
...
if len(keys) == 3:
value = (data[keys[0]][keys[1]][keys[2]])
if len(keys) == 2:
value = (data[keys[0]][keys[1]])
...
Many thanks!
I'm not sure I completely understand your question, but I would suggest you to try and play with pandas. It might be as easy as this:
import pandas as pd
df = pd.read_json(<yourJsonFile>, orient='columns')
name = df.fullname[0]
group_user = df.group.user
group_location = df.group.location
color_type = df.color.type
color_code = df.color.code
(Where group_user and group_location will be python dictionaries).
1.I have this array and i want to extract only the "rate" values.
I want y Array to look like this
array = [6,355.1675 , 4,826.3112 , 5,429.8488]
[{
code = USD;
description = "United States Dollar";
rate = "6,355.1675";
"rate_float" = "6355.1675";
symbol = "$";
}, {
code = GBP;
description = "British Pound Sterling";
rate = "4,826.3112";
"rate_float" = "4826.3112";
symbol = "£";
}, {
code = EUR;
description = Euro;
rate = "5,429.8488";
"rate_float" = "5429.8488";
symbol = "€";
}]
It looks like you have JSON that contains an array of dictionaries.
Write some code that:
Decodes the JSON into native objects. At the simplest it would be converted to an array of dictionaries.
You could then write code that loops through the array, extracts the rate value for each array entry, and prints it.
As others have said, you could do this with a map statement, but if you're new to programming then the for loop approach is easier to understand.
If you array is called array, then you can do it this way:
array.map { print ($0["rate"] }