I have to the code:
socket.emit("login",obj, new Ack {
override def call(args: AnyRef*): Unit = {
println(args)
}
})
Console output
WrappedArray({"uid":989,"APILevel":5,"status":"ok"})
How to convert args from WrappedArray to JSON?
There are many json libraries. For example you can take a look at Scala json parsers performance, to see the usage and performance. I'll demonstrate how that can be done using play-json. We need to first create a case class that represents your data model:
case class Model(uid: Int, APILevel: Int, status: String)
Now we need to create a formatter, on the companion object:
object Model {
implicit val format: Format[Model] = Json.format[Model]
}
To create a JsValue from it, you can:
val input: String = "{\"uid\":989,\"APILevel\":5,\"status\":\"ok\"}"
val json: JsValue = Json.parse(input)
and to convert it to the model:
val model: Model = json.as[Model]
A complete running example can be found at Scastie. Just don't forget to add play-json as a dependency, by adding the following to your build.sbt:
resolvers += "play-json" at "https://mvnrepository.com/artifact/com.typesafe.play/play-json"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.9.1"
Related
I am writing a Spark 3 UDF to mask an attribute in an Array field.
My data (in parquet, but shown in a JSON format):
{"conditions":{"list":[{"element":{"code":"1234","category":"ABC"}},{"element":{"code":"4550","category":"EDC"}}]}}
case class:
case class MyClass(conditions: Seq[MyItem])
case class MyItem(code: String, category: String)
Spark code:
val data = Seq(MyClass(conditions = Seq(MyItem("1234", "ABC"), MyItem("4550", "EDC"))))
import spark.implicits._
val rdd = spark.sparkContext.parallelize(data)
val ds = rdd.toDF().as[MyClass]
val maskedConditions: Column = updateArray.apply(col("conditions"))
ds.withColumn("conditions", maskedConditions)
.select("conditions")
.show(2)
Tried the following UDF function.
UDF code:
def updateArray = udf((arr: Seq[MyItem]) => {
for (i <- 0 to arr.size - 1) {
// Line 3
val a = arr(i).asInstanceOf[org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema]
val a = arr(i)
println(a.getAs[MyItem](0))
// TODO: How to make code = "XXXX" here
// a.code = "XXXX"
}
arr
})
Goal:
I need to set 'code' field value in each array item to "XXXX" in a UDF.
Issue:
I am unable to modify the array fields.
Also I get the following error if remove the line 3 in the UDF (cast to GenericRowWithSchema).
Error:
Caused by: java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to MyItem
Question: How to capture Array of Structs in a function and how to return a modified array of items?
Welcome to Stackoverflow!
There is a small json linting error in your data: I assumed that you wanted to close the [] square brackets of the list array. So, for this example I used the following data (which is the same as yours):
{"conditions":{"list":[{"element":{"code":"1234","category":"ABC"}},{"element":{"code":"4550","category":"EDC"}}]}}
You don't need UDFs for this: a simple map operation will be sufficient! The following code does what you want:
import spark.implicits._
import org.apache.spark.sql.Encoders
case class MyItem(code: String, category: String)
case class MyElement(element: MyItem)
case class MyList(list: Seq[MyElement])
case class MyClass(conditions: MyList)
val df = spark.read.json("./someData.json").as[MyClass]
val transformedDF = df.map{
case (MyClass(MyList(list))) => MyClass(MyList(list.map{
case (MyElement(item)) => MyElement(MyItem(code = "XXXX", item.category))
}))
}
transformedDF.show(false)
+--------------------------------+
|conditions |
+--------------------------------+
|[[[[XXXX, ABC]], [[XXXX, EDC]]]]|
+--------------------------------+
As you see, we're doing some simple pattern matching on the case classes we've defined and successfully renaming all of the code fields' values to "XXXX". If you want to get a json back, you can call the to_json function like so:
transformedDF.select(to_json($"conditions")).show(false)
+----------------------------------------------------------------------------------------------------+
|structstojson(conditions) |
+----------------------------------------------------------------------------------------------------+
|{"list":[{"element":{"code":"XXXX","category":"ABC"}},{"element":{"code":"XXXX","category":"EDC"}}]}|
+----------------------------------------------------------------------------------------------------+
Finally a very small remark about the data. If you have any control over how the data gets made, I would add the following suggestions:
The conditions JSON object seems to have no function in here, since it just contains a single array called list. Consider making the conditions object the array, which would allow you to discard the list name. That would simpify your structure
The element object does nothing, except containing a single item. Consider removing 1 level of abstraction there too.
With these suggestions, your data would contain the same information but look something like:
{"conditions":[{"code":"1234","category":"ABC"},{"code":"4550","category":"EDC"}]}
With these suggestions, you would also remove the need of the MyElement and the MyList case classes! But very often we're not in control over what data we receive so this is just a small disclaimer :)
Hope this helps!
EDIT: After your addition of simplified data according to the above suggestions, the task gets even easier. Again, you only need a map operation here:
import spark.implicits._
import org.apache.spark.sql.Encoders
case class MyItem(code: String, category: String)
case class MyClass(conditions: Seq[MyItem])
val data = Seq(MyClass(conditions = Seq(MyItem("1234", "ABC"), MyItem("4550", "EDC"))))
val df = data.toDF.as[MyClass]
val transformedDF = df.map{
case MyClass(conditions) => MyClass(conditions.map{
item => MyItem("XXXX", item.category)
})
}
transformedDF.show(false)
+--------------------------+
|conditions |
+--------------------------+
|[[XXXX, ABC], [XXXX, EDC]]|
+--------------------------+
I am able to find a simple solution with Spark 3.1+ as new features are added in this new Spark version.
Updated code:
val data = Seq(
MyClass(conditions = Seq(MyItem("1234", "ABC"), MyItem("234", "KBC"))),
MyClass(conditions = Seq(MyItem("4550", "DTC"), MyItem("900", "RDT")))
)
import spark.implicits._
val ds = data.toDF()
val updatedDS = ds.withColumn(
"conditions",
transform(
col("conditions"),
x => x.withField("code", updateArray(x.getField("code")))))
updatedDS.show()
UDF:
def updateArray = udf((oldVal: String) => {
if(oldVal.contains("1234"))
"XXX"
else
oldVal
})
I want to get a list of String in ArrayList<Array<String>> by using stream() and map in Kotlin.
Each Array<String> of my arrayList has 3 values and I want to get the first index value and the last index value of each array.
This is my code:
fun main(args: Array<String>) {
val result: List<String>
val obj1 = arrayOf("fruit", "Mangue", "Africa")
val obj2 = arrayOf("Milk", "Soja", "Europ")
val obj3 = arrayOf("Meat", "cochon","Asia")
val myArrayList: ArrayList<Array<String>> = ArrayList<Array<String>>(3)
val myList: MutableList<Array<String>> = mutableListOf<Array<String>>()
myList.add(obj1)
myList.add(obj2)
myList.add(obj3)
myArrayList.addAll(myList)
result = myArrayList.stream().map{it -> ("${it[0]}-${it[2]}")}.toList()
println("ArrayList of objects :")
println(myArrayList)
println("my list of String result")
println(result)
}
I want to have this list of String in my result:
[fruit-africa,milk-Europ,Meat-Asia]
Also, when I print myArrayList, I have a bad result:
ArrayList of objects :
[[Ljava.lang.String;#5caf905d, [Ljava.lang.String;#27716f4, [Ljava.lang.String;#8efb846]
How can I do it, please?
Your Question
When I run your code, this is the output I see:
ArrayList of objects :
[[Ljava.lang.String;#5b480cf9, [Ljava.lang.String;#6f496d9f, [Ljava.lang.String;#723279cf]
my list of String result
[fruit-Africa, Milk-Europ, Meat-Asia]
And in your question, you have:
I want to have this list of String in my result:
[fruit-africa,milk-Europ,Meat-Asia]
So, it looks like you already have the output you want for result. The only difference from the actual output is the lack of a space after each comma. If you don't want that space, then use joinToString to customize the output:
println(result.joinToString(",", "[", "]"))
As for:
[[Ljava.lang.String;#5b480cf9, [Ljava.lang.String;#6f496d9f, [Ljava.lang.String;#723279cf]
You see that output because arrays don't override the toString() function, and therefore use the default implementation. In Kotlin, you can use contentToString() to get similar output as you see when printing a List.
println(myArrayList.joinToString(", ", "[", "]") { it.contentToString() })
So, here is the updated code with the above changes:
fun main(args: Array<String>) {
val result: List<String>
val obj1 = arrayOf("fruit", "Mangue", "Africa")
val obj2 = arrayOf("Milk", "Soja", "Europ")
val obj3 = arrayOf("Meat", "cochon","Asia")
val myArrayList: ArrayList<Array<String>> = ArrayList<Array<String>>(3)
val myList: MutableList<Array<String>> = mutableListOf<Array<String>>()
myList.add(obj1)
myList.add(obj2)
myList.add(obj3)
myArrayList.addAll(myList)
result = myArrayList.stream().map{it -> ("${it[0]}-${it[2]}")}.toList()
println("ArrayList of objects :")
println(myArrayList.joinToString(", ", "[", "]") { it.contentToString() })
println("my list of String result")
println(result.joinToString(",", "[", "]"))
}
Which gives this output:
ArrayList of objects :
[[fruit, Mangue, Africa], [Milk, Soja, Europ], [Meat, cochon, Asia]]
my list of String result
[fruit-Africa,Milk-Europ,Meat-Asia]
Potential Improvements
With all that said, there are a few things you can simplify in your code:
This is a minor point, but since you don't use the args parameter you can actually omit it.
Your myList is not necessary; you can add your arrays directly to myArrayList.
Given the small number of elements in each array, and the small number of arrays, you can actually create the List<Array<String>> and populate it with a single listOf.
For variable types, you should prefer using List, the interface, rather than ArrayList, the implementation. This is known as "programming to an interface". Preferring List also means better use of listOf and mutableListOf, which are the idiomatic ways of creating lists in Kotlin.
You should prefer using List over arrays. In other words, create a List<List<String>> instead of a List<Array<String>>.
Lists do override the toString() method, providing readable output. Also, lists have better API support and work better with generics.
You don't need to use stream(). Kotlin provides many extension functions for arrays and Iterables, one of those being map which returns a List. Yes, these transformation functions are eagerly evaluated, unlike with streams, but given you're only performing one transfomration this doesn't really matter (in fact, the stream is likely less performant).
See kotlin.collections for the available built-in extension functions.
Given you want the first and last elements of each array, I would use first() and last().
Here is the simplified code (I added explicit types to make it clearer what the variables reference):
fun main() {
val lists: List<List<String>> = listOf(
listOf("Fruit", "Mangue", "Africa"),
listOf("Milk", "Soja", "Europe"),
listOf("Meat", "Cochon", "Asia")
)
println("List of lists of strings:")
println(lists)
val result: List<String> = lists.map { "${it.first()}-${it.last()}" }
println("Result:")
println(result)
}
Output:
List of Arrays:
[[Fruit, Mangue, Africa], [Milk, Soja, Europe], [Meat, Cochon, Asia]]
Result:
[Fruit-Africa, Milk-Europe, Meat-Asia]
From our code, we call some service and get back stringified JSON as a result. The stringified JSON is of an array of "SomeItem", which just has four fields in it - 3 Longs and 1 String
Ex:
[
{"id":33,"count":40000,"someOtherCount":0,"someString":"stuffHere"},
{"id":35,"count":23000,"someOtherCount":0,"someString":"blah"},
...
]
I've been using the play API to read values out using implicit Writes / Reads. But I'm having trouble getting it to work for Arrays.
For example, I've been try to parse the value out of the response, and then convert it to the SomeItem case class array, but it's failing:
val sanityCheckValue: JsValue: Json.parse(response.body)
val Array[SomeItem] = Json.fromJson(sanityCheckValue)
I have
implicit val someItemReads = Json.reads[SomeItem]
But it looks like it's not working. I've tried to set up a Json.reads[Array[SomeItem]] as well, but no luck.
Should this be working? Any tips on how to get this to work?
import play.api.libs.json._
case class SomeItem(id: Long, count: Long, someOtherCount: Long, someString: String)
object SomeItem {
implicit val format = Json.format[SomeItem]
}
object PlayJson {
def main(args: Array[String]): Unit = {
val strJson =
"""
|[
| {"id":33,"count":40000,"someOtherCount":0,"someString":"stuffHere"},
| {"id":35,"count":23000,"someOtherCount":0,"someString":"blah"}
|]
""".stripMargin
val listOfSomeItems: Array[SomeItem] = Json.parse(strJson).as[Array[SomeItem]]
listOfSomeItems.foreach(println)
}
}
object EventConsumer {
def main(args: Array[String]): Unit = {
val env = ExecutionEnvironment.getExecutionEnvironment
val data1 = env.readTextFile("file:////some_events.txt");
// Define the data source
data1 .map (new myMapFunction)
}
class myMapFunction extends MapFunction[String,Unit]
{
override def map(in: String): Unit = {
println(in)
}
}
}
Really stuck with this compilation error for a long time, any help please.
Error:(27, 15) could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[String]
flatMap { _.split("\n")}.filter(_.nonEmpty).map (new myMapFunction)
Error:(24, 15) not enough arguments for method map: (implicit evidence$2: org.apache.flink.api.common.typeinfo.TypeInformation[Unit], implicit evidence$3: scala.reflect.ClassTag[Unit])org.apache.flink.api.scala.DataSet[Unit].
Unspecified value parameters evidence$2, evidence$3.
data1.map (new myMapFunction)
^
^
When using Flink's Scala DataSet API it is necessary to add the following import to your code: import org.apache.flink.api.scala._.
When using Flink's Scala DataStream API you have to import import org.apache.flink.streaming.api.scala._.
The reason is that the package object contains a function which generates the missing TypeInformation instances.
I know there are more elaborate ways to achieve this in Java, but Groovy should have a concise way to do the same as per http://groovy.codehaus.org/Looping
Class Currency.groovy
class Currency {
String name
double rate
}
CurrencyController
def select(){
List<Currency> selectedCurrencies = Currency.getAll(params.currencies)
selectedCurrencies.eachWithIndex { obj, i -> obj.rate = update(obj.name)};
[selectedCurrencies:selectedCurrencies]
}
def update(String sym){
return sym
}
The above code throws:
No signature of method: currencychecker.CurrencyController$_$tt__select_closure12.doCall() is applicable for argument types: (currencychecker.Currency)
Thanks to #dmahapatro, the issue was that I was using an iterator variable obj[i], even though obj itself is the iterated object. The rest is correct!
I experimented with selectCurrencies.each as well instead of selectCurrencies.eachWithIndex however the right one in this case is eachWithIndex