Adding file names from array in dataframe column in spark scala - arrays

Question

val hadoopConf = new Configuration()
val fs = FileSystem.get(hadoopConf)
val status = fs.listStatus(new Path("/home/Test/")).map(_.getPath().toString)
val df = spark.read.format("json").load(status : _*)
How to add the file name in a new column in df?
I tried:
val dfWithCol = df.withColumn("filename",input_file_name())
But it adds the same file name in all the columns?
Can anyone suggest a better approach?

Answer 1

It is expected behaviour because your json file having more than one record in it.
Spark adds the filenames for each record and you want to check all the unique filenames then do distinct on filename column
//to get unique filenames
df.select("filename").distinct().show()
Example:
#source data
hadoop fs -cat /user/shu/json/*.json
{"id":1,"name":"a"}
{"id":1,"name":"a"}
val hadoopConf = new Configuration()
val fs = FileSystem.get(hadoopConf)
val status = fs.listStatus(new Path("/user/shu/json")).map(_.getPath().toString)
val df = spark.read.format("json").load(status : _*)
df.withColumn("filename",input_file_name()).show(false)
//unique filenames for each record
+---+----+----------------------------------------------------------------------------+
|id |name|input |
+---+----+----------------------------------------------------------------------------+
|1 |a |hdfs://nn:8020/user/shu/json/i.json |
|1 |a |hdfs://nn:8020/user/shu/json/i1.json |
+---+----+----------------------------------------------------------------------------+
in the above example you can see unique filenames for each record (as i have 1 record in each json file).