I have transactions as a DataFrame array<string>:
transactions: org.apache.spark.sql.DataFrame = [collect_set(b): array<string>]
I want to change it to RDD[Array[string]], but when I am changing it to RDD, it's getting changed to org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]:
val sam: RDD[Array[String]] = transactions.rdd
<console>:42: error: type mismatch;
found : org.apache.spark.rdd.RDD[org.apache.spark.sql.Row]
required: org.apache.spark.rdd.RDD[Array[String]]
val sam: RDD[Array[String]] = transactions.rdd
transactions.rdd will return RDD[Row], as it is in message.
You can manually convert Row to Array:
val sam = transactions.rdd.map(x => x.getList(0).toArray.map(_.toString))
More Spark 2.0 style it would be:
val sam = transactions.select("columnName").as[Array[String]].rdd
Replace columnName with proper column name from DataFrame - probably you should rename collect_set(b) to more user-friendly name
Dataframe is actually a array[Row] so whenever you run collect on a dataframe it will create a array[Row] and when you are converting it RDD it becomes a RDD[Row].
So if you want RDD[Array[String]] you can do it this way:
val sam = transactions.rdd.map(x => x.toString().stripPrefix("[").stripSuffix("]").split(fieldSeperator))
Related
I am trying to read a table from SqlServer which has one column: travel_CDE of datatype: binary(8).
This is how the source data looks like:
select * from sourcetable=>
location type_code family travel_CDE
Asia Landlocked Terrain 0xD9F21933D5346766
Below is my read statement:
val dataframe = spark.read.format("jdbc")
.option("url", s"jdbc:sqlserver://url:port;DatabaseName=databasename")
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
.option("user", "username")
.option("password", "password")
.option("dbtable", tablename)
.option("partitionColumn", partitionColumn)
.option("numPartitions", numPartitions)
.option("lowerBound", 1)
.option("upperBound", upperBound)
.option("fetchsize", 100)
.load()
When I print the schema of the dataframe, I see that the column is being read in binary datatype:
scala> dataframe.printSchema()
root
|-- location: string (nullable = true)
|-- type_code: string (nullable = true)
|-- family: string (nullable = true)
|-- travel_CDE: binary (nullable = true)
But when I read the data inside of my spark dataframe, I see data being represented in a different format for the column travel_CDE.
Example:
scala> dataframe.select("travel_CDE").take(1)
res11: Array[org.apache.spark.sql.Row] = Array([[B#a4a0ce])
So I thought spark reading data in its own format and I took that column out and re-applied a schema of binary datatype as below.
import org.apache.spark.sql.types.{StructType, StructField, BinaryType}
val schema = StructType(Array(StructField("bintype", BinaryType, true)))
val bincolDF = dataframe.select("travel_CDE")
val bindColtypeDF = spark.createDataFrame(bincolDF.rdd, schema)
But even after applying BinaryType on that column, I still see the same format of data as earlier.
scala> bindtype.take(1)
res9: Array[org.apache.spark.sql.Row] = Array([[B#1d48ff1])
I am saving this dataframe into big query and I see the same format (wrong data format) there as well.
Below is how I am doing it.
dataframe.write.format("bigquery").option("table", s"Bigquery_table_name").mode("overwrite").save()
Could anyone let me know what should I do to read the source data properly in a way that Spark reads it in the same format.
Is it even possible to do it while reading the data or should I convert the column after reading data into dataframe ?
Any help is much appreciated.
val hadoopConf = new Configuration()
val fs = FileSystem.get(hadoopConf)
val status = fs.listStatus(new Path("/home/Test/")).map(_.getPath().toString)
val df = spark.read.format("json").load(status : _*)
How to add the file name in a new column in df?
I tried:
val dfWithCol = df.withColumn("filename",input_file_name())
But it adds the same file name in all the columns?
Can anyone suggest a better approach?
It is expected behaviour because your json file having more than one record in it.
Spark adds the filenames for each record and you want to check all the unique filenames then do distinct on filename column
//to get unique filenames
df.select("filename").distinct().show()
Example:
#source data
hadoop fs -cat /user/shu/json/*.json
{"id":1,"name":"a"}
{"id":1,"name":"a"}
val hadoopConf = new Configuration()
val fs = FileSystem.get(hadoopConf)
val status = fs.listStatus(new Path("/user/shu/json")).map(_.getPath().toString)
val df = spark.read.format("json").load(status : _*)
df.withColumn("filename",input_file_name()).show(false)
//unique filenames for each record
+---+----+----------------------------------------------------------------------------+
|id |name|input |
+---+----+----------------------------------------------------------------------------+
|1 |a |hdfs://nn:8020/user/shu/json/i.json |
|1 |a |hdfs://nn:8020/user/shu/json/i1.json |
+---+----+----------------------------------------------------------------------------+
in the above example you can see unique filenames for each record (as i have 1 record in each json file).
My table data has 5 columns and 5288 rows. I am trying to read that data into a CSV file adding column names. The code for that looks like this :
cursor = conn.cursor()
cursor.execute('Select * FROM classic.dbo.sample3')
rows = cursor.fetchall()
print ("The data has been fetched")
dataframe = pd.DataFrame(rows, columns =['s_name', 't_tid','b_id', 'name', 'summary'])
dataframe.to_csv('data.csv', index = None)
The data looks like this
s_sname t_tid b_id name summary
---------------------------------------------------------------------------
db1 001 100 careie hello this is john speaking blah blah blah
It looks like above but has 5288 such rows.
When I try to execute my code mentioned above it throws an error saying :
ValueError: Shape of passed values is (5288, 1), indices imply (5288, 5)
I do not understand what wrong I am doing.
Use this.
dataframe = pd.read_sql('Select * FROM classic.dbo.sample3',con=conn)
dataframe.to_csv('data.csv', index = None)
I have a dataframe and would like to rename the columns based on a dictionary with multiple values per key. The dictionary key has the desired column name, and the values hold possible old column names. The column names have no pattern.
import pandas as pd
column_dict = {'a':['col_a','col_1'], 'b':['col_b','col_2'], 'c':'col_c','col_3']}
df = pd.DataFrame([(1,2.,'Hello'), (2,3.,"World")], columns=['col_1', 'col_2', 'col_3'])
Function to replace text with key
def replace_names(text, dict):
for key in dict:
text = text.replace(dict[key],key)
return text
replace_names(df.columns.values,column_dict)
Gives an error when called on column names
AttributeError: 'numpy.ndarray' object has no attribute 'replace'
Is there another way to do this?
You can use df.rename(columns=...) if you supply a dict which maps old column names to new column names:
import pandas as pd
column_dict = {'a':['col_a','col_1'], 'b':['col_b','col_2'], 'c':['col_c','col_3']}
df = pd.DataFrame([(1,2.,'Hello'), (2,3.,"World")], columns=['col_1', 'col_2', 'col_3'])
col_map = {col:key for key, cols in column_dict.items() for col in cols}
df = df.rename(columns=col_map)
yields
a b c
0 1 2.0 Hello
1 2 3.0 World
I am having a JDBC connection with Apache Spark and PostgreSQL and I want to insert some data into my database. When I use append mode I need to specify id for each DataFrame.Row. Is there any way for Spark to create primary keys?
Scala:
If all you need is unique numbers you can use zipWithUniqueId and recreate DataFrame. First some imports and dummy data:
import sqlContext.implicits._
import org.apache.spark.sql.Row
import org.apache.spark.sql.types.{StructType, StructField, LongType}
val df = sc.parallelize(Seq(
("a", -1.0), ("b", -2.0), ("c", -3.0))).toDF("foo", "bar")
Extract schema for further usage:
val schema = df.schema
Add id field:
val rows = df.rdd.zipWithUniqueId.map{
case (r: Row, id: Long) => Row.fromSeq(id +: r.toSeq)}
Create DataFrame:
val dfWithPK = sqlContext.createDataFrame(
rows, StructType(StructField("id", LongType, false) +: schema.fields))
The same thing in Python:
from pyspark.sql import Row
from pyspark.sql.types import StructField, StructType, LongType
row = Row("foo", "bar")
row_with_index = Row(*["id"] + df.columns)
df = sc.parallelize([row("a", -1.0), row("b", -2.0), row("c", -3.0)]).toDF()
def make_row(columns):
def _make_row(row, uid):
row_dict = row.asDict()
return row_with_index(*[uid] + [row_dict.get(c) for c in columns])
return _make_row
f = make_row(df.columns)
df_with_pk = (df.rdd
.zipWithUniqueId()
.map(lambda x: f(*x))
.toDF(StructType([StructField("id", LongType(), False)] + df.schema.fields)))
If you prefer consecutive number your can replace zipWithUniqueId with zipWithIndex but it is a little bit more expensive.
Directly with DataFrame API:
(universal Scala, Python, Java, R with pretty much the same syntax)
Previously I've missed monotonicallyIncreasingId function which should work just fine as long as you don't require consecutive numbers:
import org.apache.spark.sql.functions.monotonicallyIncreasingId
df.withColumn("id", monotonicallyIncreasingId).show()
// +---+----+-----------+
// |foo| bar| id|
// +---+----+-----------+
// | a|-1.0|17179869184|
// | b|-2.0|42949672960|
// | c|-3.0|60129542144|
// +---+----+-----------+
While useful monotonicallyIncreasingId is non-deterministic. Not only ids may be different from execution to execution but without additional tricks cannot be used to identify rows when subsequent operations contain filters.
Note:
It is also possible to use rowNumber window function:
from pyspark.sql.window import Window
from pyspark.sql.functions import rowNumber
w = Window().orderBy()
df.withColumn("id", rowNumber().over(w)).show()
Unfortunately:
WARN Window: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
So unless you have a natural way to partition your data and ensure uniqueness is not particularly useful at this moment.
from pyspark.sql.functions import monotonically_increasing_id
df.withColumn("id", monotonically_increasing_id()).show()
Note that the 2nd argument of df.withColumn is monotonically_increasing_id() not monotonically_increasing_id .
I found the following solution to be relatively straightforward for the case where zipWithIndex() is the desired behavior, i.e. for those desirng consecutive integers.
In this case, we're using pyspark and relying on dictionary comprehension to map the original row object to a new dictionary which fits a new schema including the unique index.
# read the initial dataframe without index
dfNoIndex = sqlContext.read.parquet(dataframePath)
# Need to zip together with a unique integer
# First create a new schema with uuid field appended
newSchema = StructType([StructField("uuid", IntegerType(), False)]
+ dfNoIndex.schema.fields)
# zip with the index, map it to a dictionary which includes new field
df = dfNoIndex.rdd.zipWithIndex()\
.map(lambda (row, id): {k:v
for k, v
in row.asDict().items() + [("uuid", id)]})\
.toDF(newSchema)
For anyone else who doesn't require integer types, concatenating the values of several columns whose combinations are unique across the data can be a simple alternative. You have to handle nulls since concat/concat_ws won't do that for you. You can also hash the output if the concatenated values are long:
import pyspark.sql.functions as sf
unique_id_sub_cols = ["a", "b", "c"]
df = df.withColumn(
"UniqueId",
sf.md5(
sf.concat_ws(
"-",
*[
sf.when(sf.col(sub_col).isNull(), sf.lit("Missing")).otherwise(
sf.col(sub_col)
)
for sub_col in unique_id_sub_cols
]
)
),
)