I have a list stored in database as varchar. When I read the list value, it is read as string. I'm trying to convert string to a valid list
Here are 2 example lists I have
str_db_bins = [10,20,30,40] (This is string)
The desired output is
str_db_bins_list = [10,20,30,40] (This is list)
I want to convert this to list. When I use split method I get [[,1,0...]] which is not the desired output
2nd situtation is
str_db_lov = ['id','num','val'].
When I use split method, again I get weird output
The desired output is to read this as list
str_db_lov_list = ['id', 'num', 'val']
How do I deal with these situations
Thanks
Pari
str_db_bins_list = eval(str_db_bins)
str_db_lov_list = eval(str_db_lov)
Related
I want to extract users and items features (latent factors) from the result of collaborative filtering using ALS in Spark. The code I have so far:
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
import org.apache.spark.mllib.recommendation.Rating
// Load and parse the data
val data = sc.textFile("myhdfs/inputdirectory/als.data")
val ratings = data.map(_.split(',') match { case Array(user, item, rate) =>
Rating(user.toInt, item.toInt, rate.toDouble)
})
// Build the recommendation model using ALS
val rank = 10
val numIterations = 10
val model = ALS.train(ratings, rank, numIterations, 0.01)
// extract users latent factors
val users = model.userFeatures
// extract items latent factors
val items = model.productFeatures
// save to HDFS
users.saveAsTextFile("myhdfs/outputdirectory/users") // does not work as expected
items.saveAsTextFile("myhdfs/outputdirectory/items") // does not work as expected
However, what gets written to HDFS is not what I expect. I expected each line to have a tuple (userId, Array_of_doubles). Instead I see the following:
[myname#host dir]$ hadoop fs -cat myhdfs/outputdirectory/users/*
(1,[D#3c3137b5)
(3,[D#505d9755)
(4,[D#241a409a)
(2,[D#c8c56dd)
.
.
It is dumping the hash value of the array instead of the entire array. I did the following to print the desired values:
for (user <- users) {
val (userId, lf) = user
val str = "user:" + userId + "\t" + lf.mkString(" ")
println(str)
}
This does print what I want but I can't then write to HDFS (this prints on the console).
What should I do to get the complete array written to HDFS properly?
Spark version is 1.2.1.
#JohnTitusJungao is right and also the following lines works as expected :
users.saveAsTextFile("myhdfs/outputdirectory/users")
items.saveAsTextFile("myhdfs/outputdirectory/items")
And this is the reason, userFeatures returns an RDD[(Int,Array[Double])]. The array values are denoted by the symbols you see in the output e.g. [D#3c3137b5 , D for double, followed by # and hex code which is created using the Java toString method for this type of objects. More on that here.
val users: RDD[(Int, Array[Double])] = model.userFeatures
To solve that you'll need to make the array as a string :
val users: RDD[(Int, String)] = model.userFeatures.mapValues(_.mkString(","))
The same goes for items.
I have an array loaded in, and been playing around in the REPL but can't seem to get this to work.
My array looks like this:
record_id|string|FALSE|1|
offer_id|decimal|FALSE|1|1,1
decision_id|decimal|FALSE|1|1,1
offer_type_cd|integer|FALSE|1|1,1
promo_id|decimal|FALSE|1|1,1
pymt_method_type_cd|decimal|FALSE|1|1,1
cs_result_id|decimal|FALSE|1|1,1
cs_result_usage_type_cd|decimal|FALSE|1|1,1
rate_index_type_cd|decimal|FALSE|1|1,1
sub_product_id|decimal|FALSE|1|1,1
campaign_id|decimal|FALSE|1|1,1
When I run my command:
for(i <- 0 until schema.length){
val convert = schema(i).toString;
convert.split('|').drop(2);
println(convert);
}
It won't drop anything. It also is not splitting it on the |
Strings are immutable, and so split and drop don't mutate the string - they return a new one.
You need to capture the result in a new val
val split = convert.split('|').drop(2);
println(split.mkString(" "));
Consider also defining a lambda function for mapping each item in the array, where intermediate results are passed on with the function,
val res = schema.map(s => s.toString.split('|').drop(2))
I'm really fresh to Python and need help reading information from txt file. I have a large C++ app need to duplicate it in Python. Sadly I have no idea where to start. I've been reading and watching some tutorials, but little help from them and I'm running out of time.
So my task is:
I have a shopping list with:
-Name of the item, price and age.
I also need to create two searches.
Search whether the item is in the shop (comparing strings).
if name of the item is == to the input name.
Search by age. Once the program finds the items, then it needs to print the list according to the price - from the lowest price to the highest.
For example you input age 15 - 30, the program prints out appropriate
items and sorts them by the price.
Any help would be nice. At least from where I could start.
Thank you.
EDITED
So far, I have this code:
class data:
price = 0
agefrom = 0
ageto = 0
name = ''
# File reading
def reading():
with open('toys.txt') as fd:
toyslist = []
lines = fd.readlines()
for line in lines:
information = line.split()
print(information)
"""information2 = {
'price': int(information[1])
'ageftom': int(information[2])
'ageto': int(information[3])
#'name': information[4]
}"""
information2 = data()
information2.price = int(information[0])
information2.agefrom = int(information[1])
information2.ageto = int(information[2])
information2.name = information[3]
toyslist.append(information2)
return toyslist
information = reading()
I HAVE A PROBLEM WITH THIS PART. I want to compare the user's input with the item information in the txt file.
n_search = raw_input(" Please enter the toy you're looking for: ")
def name_search(information):
for data in information:
if data.name == n_search:
print ("We have this toy.")
else:
print ("Sorry, but we don't have this toy.")
If you want to fins something in a list it's generally as straightforward as:
if "apple" in ["tuna", "pencil", "apple"]
However, in your case, the list to search is a list of lists so you need to "project" it somehow. List comprehension is often the easiest to reason about, a sort of for loop in a for loop.
if "apple" in [name for name,price,age in [["tuna",230.0,3],["apple",0.50,1],["pencil",1.50,2]]]
From here you want to start looking at filters whereby you provide a function that determines whether an entry is matched or not. you can roll your own in a for loop or use something more functional like 'itertools'.
Sorting on a list is also easy, just use 'sorted(my_list)' supplying a comparator function if you need it.
Examples as per your comment...
class ShoppingListItem:
def __init__(self,name,price,age):
self.name=name
self.price=price
self.age=age
or
from collections import namedtuple
sli = namedtuple("ShoppingListItem",['name','age','price'])
Can someone please help me with this. I saved my data into Parse.com into column with type array (example: ["11:30","12:45,"13:02"], just some list of some times as string). I have tried to get this data with swift:
var take: NSMutableArray!
var query = PFQuery(className: "test")
query.getObjectInBackgroundWithId("QZ6Y8Oljc5"){
(testData: PFObject!, error: NSError!) -> Void in
if (error == nil){
take = testData["workday"]
println(take)
}
else{
println(error)
}
}
the problem is that i get only json array type:
(
"11:30",
"12:45,
"13:02"
)
How can I convert it into NSArray so it could be like:
var myArray = ["11:30","12:45,"13:02"]
Thank you for any suggestions because I tried every method I found here, but without any results.
The problem with JSON data is that it is it's own array that has to be sifted and groomed. normally people would end up using huge nested IF statements which ends up looking messy. Luckily, someone created a code that sifts through JSON data and gives you back usable types (Int, Arrays, Strings) by use of a massive switch table.
https://github.com/SwiftyJSON/SwiftyJSON
Look it up, it should help. Once you have it implemented you can call it by typing..
let json = JSON(Data : JSONData)
then to sift through, you use substrings.. (depending on the data, you match it with a string or int)
let firstIndex = json["workday"]
//Int
let firstIndexOfWorkDay = json["workday"][0]
//String
let firstIndexOfWorkDay = json["workday"]["time"]
and so on... however, you will need to cast it once you singled out the data
let firstIndexOfWorkDay = json["workday"][0].valueOfFloat
//printing it would give 11:30
although personally I use ".description" .. since sometimes when I sift through all the array, its a mix of types.
let firstIndexOfWorkDay = json["workday"][0].description
println(firstIndexOfWorkDay)
//would literally give "11:30" including the quotation marks
then I use string methods to trim the quotations then cast it to whatever type I need. But it's up to your creativity once you figure out how it works
I have data inside a data attribute, like so:
<div class="dashboard-module" data-rows="new Date(2013,10,04),12,"OR"##new Date(2013,10,17),2,"OR"##new Date(2013,10,09),2,"CA""></div>
Im trying to split this string up and use it in the data.addRow function:
rows = el.data('rows');
rowsarray = rows.split('##');
// Error: Row given with size different than 3 (the number of columns in the table).
$.each(rowsarray, function(index, value) {
data.addRow( [value] );
});
// the following works
data.addRow([new Date(2013,10,04),12,"OR"]);
data.addRow([new Date(2013,10,09),2,"CA"]);
data.addRow([new Date(2013,12,12),14,"AL"]);
I guess the commas inside the new date are being counted as different parts of the array?
I'm assuming that the double-quotes inside your data-rows attribute are escaped (otherwise the HTML is malforned).
When you call rowsarray = rows.split('##');, you are getting an array of strings, like this:
[
'new Date(2013,10,04),12,"OR"',
'new Date(2013,10,17),2,"OR"',
'new Date(2013,10,09),2,"CA"'
]
not an array of arrays. If you want to store your data in an HTML attribute, your best bet is to use a JSON-compatible format. The problem then becomes storing dates, since Date objects are not JSON-compatible, but that is easy to work around. Store your data like this instead:
[["Date(2013,10,04)",12,"OR"],["Date(2013,10,17)",2,"OR"],["Date(2013,10,09)",2,"CA"]]
I did two things with the data-rows attribute: first, I changed the dates from a format like new Date(2013,10,17) to a string like "Date(2013,10,17)". Second, I converted the string to a JSON string representation of an array of arrays (which uses the standard javascript array brackets [ and ]). Note that JSON requires the use of double-quotes for all internal strings, so you must either escape all internal strings to use with the data-rows attribute, or use single-quotes around the data-rows attribute string (eg: data-rows='<string>').
You can then parse that string for entry into your DataTable:
rows = JSON.parse(el.data('rows'));
// convert date strings to Date objects
for (var i = 0; i < rows.length; i++) {
var dateStr = rows[i][0];
var dateArr = dateStr.substring(5, dateStr.length - 1).split(',');
rows[i][0] = new Date(dateArr[0], dateArr[1], dateArr[2]);
}
data.addRows(rows);