Selenium Webdriver - passing bulk data with excel sheet by header name- more than 50 fields of form - selenium-webdriver

I am looking for some solution where i want to pass 100s of records to the form where i am having more than 50 fields. I did some research for the testNG data providers but it looks like that it returns only strings so i feel that it will not be feasible to go with data providers as if its not good to pass 50 string arguments to specific function. Also i did some research to read excel file and i get two ways that either i can go with the jxl or Apache poi but with that also i am not able to read the data by the column header as if i can not go with the row and column number of approach as i have so many fields that i need to work with. The reason behind that is that in future is one field has added to single form that its going to be rework and again its not feasible.
enter image description here
I have been following this link:
http://www.softwaretestinghelp.com/selenium-framework-design-selenium-tutorial-21/
for reading data column wise but any how i am not getting the records based on the column header. Do we have any other way to achieve this.
Thanks

"testNG data providers but it looks like that it returns only strings" - incorrect. It allows you to return a multidimensional array of type Object. What kind of object you create is your own code. You may choose to read from the excel, encapsulate all the fields in one object (your own pojo) or multiple objects and then the method argument can have just those object types declared and not the 50 strings.
Both jxl and poi are libraries to interact with excel. If you want to have specific interaction with excel, like reading based on header, then you need to write code for that - it doesn't come out of the box.
If you are concerned about addition of one more column , then build your indices first by reading the header column, then put it in a relevant data structure and then go about reading your data.

I finally achieved that with the help of apache poi. I created on centralized function that is returning the hashmap having title as an index.
Here is that function:
Here is my main test function:
#Test(dataProvider="dpCreateNewCust")
public void createNewCustomer(List<Map<String, String>> sheetList){
try{
//Step 2. Login
UtilityMethods.SignIn();
for(Map<String, String> map : sheetList){
//Step 3. New Customer
if(map.get("Testcase").equals("Yes"))
{
//Process with excel data
ProcessNewCustomer(map);
}
}
}
catch(InterruptedException e)
{
System.out.println ("Login Exception Raised: <br> The exception get caught" + e);
}
}
//My data provider
#DataProvider(name = "dpCreateNewCust")
public Object[][] dpCreateNewCust(){
XLSfilename = System.getProperty("user.dir")+"//src//watts//XLSFiles//testcust.xlsx";
List<Map<String, String>> arrayObject = UtilityMethods.getXLSData(XLSfilename,Sheetname));
return new Object[][] { {arrayObject } };
}
//----GetXLSData Method in UtilityMethods Class :
public static List<Map<String, String>> getXLSData(String filename, String sheetname)
{
List<String> titleList = new ArrayList<String>();
List<Map<String, String>> sheetList = new ArrayList<Map<String, String>>();
try {
FileInputStream file = new FileInputStream(filename);
//Get the workbook instance for XLS file
XSSFWorkbook XLSbook = new XSSFWorkbook(file);
//Get first sheet from the workbook
//HSSFSheet sheet = workbook.getSheetAt(0);
WorkSheet = XLSbook.getSheet(sheetname);
//Iterate through each rows from first sheet
int i = 0;
Iterator<Row> rowIterator = WorkSheet.iterator();
while(rowIterator.hasNext()) {
Row row = rowIterator.next();
//For each row, iterate through each columns
Iterator<Cell> cellIterator = row.cellIterator();
int j = 0;
Map<String, String> valueMap = new HashMap<>();
while(cellIterator.hasNext()) {
Cell cell = cellIterator.next();
if(i==0){
titleList.add(cell.getStringCellValue());
}
else
{
String cellval = "";
switch(cell.getCellType()) {
case Cell.CELL_TYPE_BOOLEAN:
cellval = cell.getBooleanCellValue()+"";
break;
case Cell.CELL_TYPE_NUMERIC:
cellval = String.valueOf(cell.getNumericCellValue())+"";
break;
case Cell.CELL_TYPE_STRING:
cellval = cell.getStringCellValue();
break;
default:
break;
}
if(cellval!="")
{
valueMap.put(titleList.get(j), cellval); valueMap.put("ResultRow",String.valueOf(row.getRowNum()));
valueMap.put("ResultCol",String.valueOf(0));
}
}
j++;
}
if(i!=0 && !valueMap.isEmpty()){
//System.out.println(valueMap);
sheetList.add(valueMap);
}
i++;
}
//System.out.println(sheetList); System.exit(0);
file.close();
XLSbook.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return sheetList;
}

Related

WebAPI EF update 30,000 rows of data is very slow

I am trying to have my asp.net WebAPI web service read a .csv and update a database using Entity Framework. The .csv file is about 20,000-30,000 rows.
As of now I am using a TextfieldParser to read the .csv, each row of the .csv file I create a new object, then add object to the EF context.
Once it's done adding all rows to the context, then I call db.SaveChanges();
Watching the console I noticed it calls an update statement for each row... which takes a long time. Is there a better more efficient way to accomplish this?
if (filetype == "xxx")
{
using (TextFieldParser csvReader = new TextFieldParser(downloadFolder + fileName))
{
csvReader.SetDelimiters(new string[] { "," });
csvReader.HasFieldsEnclosedInQuotes = true;
int rowCount = 1;
while (!csvReader.EndOfData)
{
string[] fieldData = csvReader.ReadFields();
//skip header row
if (rowCount != 1)
{
var t = new GMI_adatpos
{
PACCT = fieldData[3]
};
db.GMI_adatpos.Add(t);
}
rowCount++;
}
}
}
db.SaveChanges();
This issue is very common,
In your case, we can split it into two category:
Add vs AddRange Performance
Write & database round-trip
Add vs AddRange Performance
The Add method will try to detect change every time you add a new record while the AddRange only does it once. Detecting changes every time can take several minutes.
This issue is very easy to fix, simply create a list, add the entity to this list instead and use AddRange with the list at the end.
List<GMI_adatpo> list = new List<GMI_adatpo>();
if (filetype == "xxx")
{
using (TextFieldParser csvReader = new TextFieldParser(downloadFolder + fileName))
{
csvReader.SetDelimiters(new string[] { "," });
csvReader.HasFieldsEnclosedInQuotes = true;
int rowCount = 1;
while (!csvReader.EndOfData)
{
string[] fieldData = csvReader.ReadFields();
//skip header row
if (rowCount != 1)
{
var t = new GMI_adatpos
{
PACCT = fieldData[3]
};
list.Add(t);
}
rowCount++;
}
}
}
db.GMI_adatpos.AddRange(list)
db.SaveChanges();
Write & Database round-trip
Everytime you save a record, you perform a database round-trip. So if you insert average 30,000 record, you perform 30,000 database round-trip which is insane!
Disclaimer: I'm the owner of the project Entity Framework Extensions
This library allows to perform:
BulkSaveChanges
BulkInsert
BulkUpdate
BulkDelete
BulkMerge
You can either call BulkSaveChanges instead of SaveChanges or create a list to insert and use directly BulkInsert instead for even more performance.
BulkSaveChanges Solution (Way faster than SaveChanges)
db.GMI_adatpos.AddRange(list)
db.SaveChanges();
BulkInsert Solution (Fastest than BulkSaveChanges but do not save related entities)
db.BulkInsert(list);
Because the number of items added to the DbContext is very high, ram space is gradually filled, and operation is very slow. Therefore is better that after a few records (ex 100), calling SaveChanges Methods and renew DbContext.
if (filetype == "xxx")
{
using (TextFieldParser csvReader = new TextFieldParser(downloadFolder + fileName))
{
csvReader.SetDelimiters(new string[] { "," });
csvReader.HasFieldsEnclosedInQuotes = true;
int rowCount = 1;
while (!csvReader.EndOfData)
{
if(rowCount%100 == 0)
{
db.Dispose();
db.SaveChanges();
db = new AppDbContext();//Your DbContext
}
string[] fieldData = csvReader.ReadFields();
//skip header row
if (rowCount != 1)
{
var t = new GMI_adatpos
{
PACCT = fieldData[3]
};
db.GMI_adatpos.Add(t);
}
rowCount++;
}
}
}

Entity Framework : Create a model from Dictionary<TKey,TValue> to be mapped to a database table

Earlier I had a table named ApplicationConfiguration which simply had [Key],[Value] columns to store some config data. This was queried straight away using SQL queries.
Now I intend to make use of Entity Framework (EF) Code First approach to query this table. The specialty of this table is that the table will have only a fixed number of rows in its lifetime. Only the Value column can be updated.
So as per the code first approach, we have to first write our POCO classes with its properties that will be mapped to columns in the underlying table. However, I wish to have a Dictionary<> structure to represent these configuration KV pairs. My concern is, will EF be able to fire update queries against any updation to the the value of a particular pair.
Also since I am using Code First approach, I would want some seed data(i.e the fixed number of rows and its initial content) to the added after the table itself is created on the fly when the application is first executed.
If Dictionary<> cannot be used, please suggest some alternative. Thanks in advance.
Coded this way:
public class ApplicationConfiguration
{
public int Id { get; set; }
public string Key { get; set; }
public int Value { get; set; } // should be string, but I'm lazy
}
class Context : DbContext
{
internal class ContextInitializer : DropCreateDatabaseIfModelChanges<Context>
{
protected override void Seed(Context context)
{
var defaults = new List<ApplicationConfiguration>
{
new ApplicationConfiguration {Key = "Top", Value = 5},
new ApplicationConfiguration {Key = "Bottom", Value = 7},
new ApplicationConfiguration {Key = "Left", Value = 1},
new ApplicationConfiguration {Key = "Right", Value = 3}
};
// foreach (var c in defaults)
// context.ConfigurationMap.Add(c.Key, c); // by design, no IReadOnlyDictionary.Add
foreach (var c in defaults)
context.ApplicationConfigurations.Add(c);
base.Seed(context);
}
}
public Context()
{
Database.SetInitializer(new ContextInitializer());
}
private IDbSet<ApplicationConfiguration> ApplicationConfigurations
{
get { return Set<ApplicationConfiguration>(); }
}
public IReadOnlyDictionary<string, ApplicationConfiguration> ConfigurationMap
{
get { return ApplicationConfigurations.ToDictionary(kvp => kvp.Key, kvp => kvp); }
}
}
Used this way:
using (var context = new Context())
{
ReadConfigurationOnly(context.ConfigurationMap);
}
using (var context = new Context())
{
ModifyConfiguration(context.ConfigurationMap);
context.SaveChanges();
}
static void ReadConfigurationOnly(IReadOnlyDictionary<string, ApplicationConfiguration> configuration)
{
foreach (var k in configuration.Keys)
Console.WriteLine("{0} = {1}", k, configuration[k].Value);
}
static void ModifyConfiguration(IReadOnlyDictionary<string, ApplicationConfiguration> configuration)
{
foreach (var k in configuration.Keys)
configuration[k].Value++; // this is why I was lazy, using an int for a string
}
So, I wrote it up this way — using an int Value property rather than a string — just so I could run the "Used this way" code over and over, and see the database update each time, without having to come up with some other way to change Value in an interesting way.
It's not quite as nifty here to use a IReadOnlyDictionary<string, ApplicatonConfiguration> instead of a IReadOnlyDictionary<string, string>, the way we'd really like, but that's more than made up for by the fact that we can easily modify our collection values without resorting to a clumsier Set method taking a dictionary as input. The drawback, of course, is that we have to settle for configuration[key].Value = "new value" rather than configuration[key] = "new value", but — as I say — I think it's worth it.
EDIT
Dang! I wrote this code up specifically to answer this question, but I think I like it so much, I'm going to add it to my bag of tricks ... this would fit in really well when my company goes from local databases to Azure instances in the cloud, and the current app.config has to go into the database.
Now all I need is a ContextInitializer taking a System.Configuration.ConfigurationManager as a ctor parameter in order to seed a new database from an existing app.config ...
I don't think you can map a table directly to a Dictionary; you will probably have to write your own wrapper to fill a dictionary from the table and update it back to the DB. Entities are each a row of a given table... Something like this (untested):
public Dictionary<string, string> GetDictionary()
{
Dictionary<string, string> dic = new Dictionary<string, string>();
using (var db = new Context())
{
var configs = db.ApplicationConfiguration.Select();
foreach (var entry in configs)
{
dic.Add(config.Key, config.Value);
}
}
return dic;
}
public void SaveConfig(Dictionary<string, string> dic)
{
using (var db = new Context())
{
foreach (KeyValuePair kvp in dic)
{
if (!db.ApplicationConfiguration.First(a => a.Key == kvp.Key).Value == kvp.Value)
{
var ac = new ApplicationConfiguration();
ac.Key = kvp.Key;
ac.Value = kvp.Value;
db.Entry(ac).State = EntityState.Modified;
}
}
db.SaveChanges();
}
}
For your second question, you want to use the Seed() method to add initial values to the database. See here for an example implementation.

Multi-dimension queryable HashMap Array

I have a CSV file that I'd like to store as a Java object. I would like the name of the columns to be the first dimension of the array, and key pair values the second dimension of the array. I've tried different solutions (mostly LinkedHashMaps) but none seem to work properly.
The CSV looks like this:
TimeStamp;Column 1;Column 2;Column3
1385733406;Value1;Value12;Value13
1385733409;Value21;Value22;Value23
1385733411;Value31;Value32;Value33
I would like the array to look something like this:
["Column 1"]
["1385733406","Value1"]
["1385733409","Value21"]
["1385733411","Value31"]
["Column 2"]
["1385733406","Value2"]
["1385733409","Value22"]
["1385733411","Value32"]
["Column 3"]
["1385733406","Value2"]
["1385733409","Value22"]
["1385733411","Value33"]
This way, I would be able to query the object and retrieve all the key pair values from a given column, for instance, all the data from Column 1. Using HashMaps doesn't seem to work because they require two arguments, and doing this doesn't seem to be the proper way. This is the code I could come up with so far, which I don't think is the right track but it's all I could come up with. I'm using OpenJDK 1.7
public class CsvCollection {
public Map<String,Map<String,Integer>> Results = new LinkedHashMap<String, Map<String,Integer>>();
public CsvCollection(){
}
public void parseCsvResultFile(File csvFile){
CSVReader reader = null;
List myEntries = null;
try {
reader = new CSVReader(new FileReader(csvFile.getAbsolutePath()), ';');
} catch (FileNotFoundException e) {
System.out.println("Error opening [], aborting parsing");
}
try {
myEntries = reader.readAll();
} catch (IOException e) {
System.out.println("Error reading content of CSV file (but file was opened)");
}
for(String header: (String[]) myEntries.get(0)){
Results.put(header, null);
// What now?
}
}
}
You can make 2 change as follows to implements the function you needed.
1) Change the following code
public Map<String,Map<String,Integer>> Results = new LinkedHashMap<String, Map<String,Integer>>();
to
public Map<String, List<String[]>> Results = new LinkedHashMap<String, List<String[]>>();
This change is made because for a specify column, like Column 1, it has 3 rows with Timestamp and corresponding Column value. You need to use a List to store them.
2) Change the following for-loop
for(String header: (String[]) myEntries.get(0)){
Results.put(header, null);
// What now?
}
to
String[] headerColumns = (String[]) myEntries.get(0);
// First column is TimeStamp, skip it
for (int i = 1; i < headerColumns.length; i++) {
List<String[]> list = new ArrayList<String[]>();
for (int rowIndex = 1; rowIndex < myEntries.size(); rowIndex++) {
String[] row = (String[]) myEntries.get(rowIndex);
list.add(new String[] { row[0], row[i] });
}
Results.put(headerColumns[i], list);
}
With the above 2 changes, if you print the Results (type of Map< String, List< String[] > >) in console using the following code,
for(Map.Entry<String, List<String[]>> entry : Results.entrySet())
{
System.out.printf("[%s]\n",entry.getKey());
for(String[] array : entry.getValue())
{
System.out.println(Arrays.toString(array));
}
}
you will get the result you need:
[Column 1]
[1385733406, Value1]
[1385733409, Value21]
[1385733411, Value31]
[Column 2]
[1385733406, Value12]
[1385733409, Value22]
[1385733411, Value32]
[Column3]
[1385733406, Value13]
[1385733409, Value23]
[1385733411, Value33]
Note: the above example is executed using the content from a CSV file below:
TimeStamp;Column 1;Column 2;Column3
1385733406;Value1;Value12;Value13
1385733409;Value21;Value22;Value23
1385733411;Value31;Value32;Value33

Why does this code - adding wordnet synonyms to index - fail?

I am writing this code as part of my CustomAnalyzer:
public class CustomAnalyzer extends Analyzer {
SynonymMap mySynonymMap = null;
CustomAnalyzer() throws IOException {
SynonymMap.Builder builder = new SynonymMap.Builder(true);
FileReader fr = new FileReader("/home/watsonuser/Downloads/wordnetSynonyms.txt");
BufferedReader br = new BufferedReader(fr);
String line = "";
while ((line = br.readLine()) != null) {
String[] synset = line.split(",");
for(String syn: synset)
builder.add(new CharsRef(synset[0]), new CharsRef(syn), true);
}
br.close();
fr.close();
try {
mySynonymMap = builder.build();
} catch (IOException e) {
System.out.println("Unable to build synonymMap");
e.printStackTrace();
}
}
public TokenStream tokenStream(String fieldName, Reader reader) {
TokenStream result = new PorterStemFilter(new SynonymFilter(
(new StopFilter(true,new LowerCaseFilter
(new StandardFilter(new StandardTokenizer
(Version.LUCENE_36,reader)
)
),StopAnalyzer.ENGLISH_STOP_WORDS_SET)), mySynonymMap, true)
);
}
}
Now, if I use the same CustomAnalyzer as part of my querying, then if I enter the query as
myFieldName: manager
it expands the query with synonyms for manager.
But, I want the synonyms to be part of only my index and I don't want my query to be expanded with synonyms.
So, when I removed the SynonymFilter from my CustomAnalyzer only when querying the index, the query remains as
myFieldName: manager
but, it fails to retrieve documents that have the synonyms of manager.
How do we solve this problem?
If you do not have your synonym builder during Query processing then the only term it will match is what you mapped to during indexing. And you are not showing that part here.
The best way to troubleshoot this is to look at Admin/Core/Analysis screen (in Solr 4+) and put your text in. It will show what happens with the text after each stage in indexing and queries is run.
You don't even need to run reindexer. You can just define a bunch of different types you are trying to figure out and then run the analysis of the sample sentences directly against those types.

Dynamically add check box and records from database in javaFX 2.2 Table View

I am writing a client-server java FX application with a table View. I have a database in the server side and I want from the Client to load the table Columns and table Records from the db to a table View dynamically. So far I have found many hints, on how to do this successfully. The thing is that I want to add to the table a column 'select' which is a check box. Below is my code.
private void AddToTableRecordsFromDB(TabPane tp){
tableview = (TableView) tp.lookup("#table");
ObservableList<Object> data = null;
try {
String[] columnNames = (String[]) Login.Login.in.readObject();
ArrayList<ArrayList> al = (ArrayList<ArrayList>) Login.Login.in.readObject();
/**********************************
* TABLE COLUMN ADDED DYNAMICALLY *
**********************************/
TableColumn select = new TableColumn("Select");
select.setCellValueFactory(new PropertyValueFactory("invited"));
select.setCellFactory(new Callback<TableColumn<ObservableValue, Boolean>, TableCell<ObservableValue, Boolean>>() {
public TableCell<ObservableValue, Boolean> call(TableColumn<ObservableValue, Boolean> p) {
return new CheckBoxTableCell<ObservableValue, Boolean>();
}
});
tableview.getColumns().add(select);
for(int i=0 ; i<columnNames.length; i++){
//use non property style for making dynamic table
final int j = i;
TableColumn col;
col = new TableColumn(columnNames[i]);
col.setCellValueFactory(new Callback<CellDataFeatures<ObservableList,String>,ObservableValue<String>>(){
#Override
public ObservableValue<String> call(CellDataFeatures<ObservableList, String> param) {
return new SimpleStringProperty(param.getValue().get(j).toString());
}
});
tableview.getColumns().add(col);
}
/********************************
* Data added to ObservableList *
********************************/
data = FXCollections.observableArrayList();
for(int i=0 ; i<al.size(); i++){
ObservableList<ArrayList> row = FXCollections.observableArrayList(al.get(i));
data.add(row);
}
//FINALLY ADDED TO TableView
tableview.setItems(data);
} catch (IOException | ClassNotFoundException ex) {
Logger.getLogger(Developer_Main.class.getName()).log(Level.SEVERE, null, ex);
}
}
I took the CheckBoxTableCell class from the JavaFX2.0 Ensemble.
The database loads successfully in to the table view and also my 'select' column is created, but I can not see any check boxes in the rows.
Any help please ?
You say you are using "non property style" to add columns dynamically, and your table view made of list of list items. I guess there is no getInvited() method in those data structure model. However by setting select.setCellValueFactory(new PropertyValueFactory("invited")); the table column will look for that method. Set cell value factory with valid value.
EDIT: I didn't test but can you try the code below.
select.setCellValueFactory(new Callback<TableColumn.CellDataFeatures<ObservableList, String>, ObservableValue<String>>() {
public ObservableValue<String> call(TableColumn.CellDataFeatures<ObservableList, String> p) {
return new SimpleStringProperty(p.getValue().get(0).toString());
}
});
Note the get(0). Namely it would be better if readObject() returns at least 1 item. Another note is CheckBoxTableCell needs a Callback which returns ObservableProperty<Boolean> and binds bidirectionally so I think it is better to implement your own cell factory containing checkbox, regarding to your data model.

Resources