I'm benchmarking Ignite and running the following tests:
Get 5K keys out of 100K/1M (3K are matched) using
cache.getAll(keys)
Query out of 100K/1M using:
List keys = cache.query(new ScanQuery(
(k, p) -> p.getSalary() > 900000)
Cache.Entry::getKey).getAll();
I have tuned Durable Memory as suggested here and I'm using separate ssd for storagePath and walPath but the results are not so good:
case 1 using 100K - 4,761ms
case 1 using 1M - 4,979
case 2 using 100K - 250ms
case 2 using 1M - 2,207ms
my object size is 1k. This is it:
public class MyPerson implements Serializable
{
/** Resume text (create LUCENE-based TEXT index for this field). */
#QueryTextField
private String resume;
/** MyPerson ID (indexed). */
#QuerySqlField(index = true)
private Long id;
/** Organization ID (indexed). */
#QuerySqlField(index = true)
private Long orgId;
/** First name (not-indexed). */
#QuerySqlField
private String firstName;
/** Last name (not indexed). */
#QuerySqlField
private String lastName;
/** Salary (indexed). */
#QuerySqlField(index = true)
private double salary;
public void setResume(String resume) {
this.resume = resume;
}
public Long getId() {
return id;
}
public MyPerson(Long id){
this.id = id;
}
public double getSalary() {
return salary;
}
public void setSalary(double salary) {
this.salary = salary;
}
}
How can I improve my results?
Thanks,
Alon
Related
In Apache Flink, I am not able to see the output in std out, but my job is running successfully and data is coming
As you are running your job on a cluster, DataStreams are printed to the stdout of the TaskManager process. This TaskManager stdout is directed to an .out file in the ./log/ directory of the Flink root directory. I believe this is here you have seen your output.
I don't know if it is possible to change the stdout of TaskManagers, however, a quick and dirty solution could be to write the output to a socket :
output.writeToSocket(outputHost, outputPort, new SimpleStringSchema())
public static void main(String[] args) throws Exception {
// the host and the port to connect to
final String hostname = "192.168.1.73";
final int port = 9000;
final StreamExecutionEnvironment env = StreamExecutionEnvironment.createRemoteEnvironment("192.168.1.68", 6123);
// get input data by connecting to the socket
DataStream<String> text = env.socketTextStream(hostname, port, "\n");
// parse the data, group it, window it, and aggregate the counts
DataStream<WordWithCount> windowCounts = text
.flatMap(new FlatMapFunction<String, WordWithCount>() {
public void flatMap(String value, Collector<WordWithCount> out) {
for (String word : value.split("\\s")) {
out.collect(new WordWithCount(word, 1L));
}
}
})
.keyBy("word").timeWindow(Time.seconds(5))
.reduce(new ReduceFunction<WordWithCount>() {
public WordWithCount reduce(WordWithCount a, WordWithCount b) {
return new WordWithCount(a.word, a.count + b.count);
}
});
// print the results with a single thread, rather than in parallel
windowCounts.print().setParallelism(1);
env.execute("Socket Window WordCount");
}
public static class WordWithCount {
public String word;
public long count;
public WordWithCount() {
}
public WordWithCount(String word, long count) {
this.word = word;
this.count = count;
}
#Override
public String toString() {
return word + " : " + count;
}
}
I am new in using TPL in .Net applications. While creating a simple console application to achieve some parallel tasks those are dynamically created, I am stuck with some issues.
Problem here is that when 10 tasks are created and run, although the console is showing all the 10 tasks, when writing those into a log file after putting a delay between consoling and logging, the log file misses some of the items randomly.
Below is my sample code (This is just a skeleton of my actual code)
class Program
{
public static int datacount = 10;
static void Main(string[] args)
{
List<Task> tasks = new List<Task>();
var s1 = DateTime.Now;
var transList = GenerateTransactionList();
foreach (var transaction in transList)
{
Transactions transactionNew = new Transactions();
transactionNew = transaction;
tasks.Add(Task.Factory.StartNew(() => serialMethod(transactionNew)));
}
Task.WhenAll(tasks).Wait();
Console.WriteLine("Completed!!!");
}
private static List<Transactions> GenerateTransactionList()
{
Random r = new Random();
List<Transactions> transactionList = new List<Transactions>();
for (int i = 1; i <= datacount; ++i)
{
Transactions tr = new Transactions();
tr.ID = 0;
tr.Amount = r.Next(1, 10);
tr.Created_By = "Iteration" + i;
tr.Notes = "Iteration" + i;
tr.Created_On = DateTime.Now;
transactionList.Add(tr);
}
return transactionList;
}
private static async Task<string> serialMethod(Transactions tlist)
{
Console.WriteLine("Started Serial Iteration" + tlist.Notes);
try
{
Console.WriteLine("Finished Serial Iteration" + tlist.Notes);
Thread.Sleep(10000);//doing some time consuming process
WriteLog("Parallel2", DateTime.Now, DateTime.Now, tlist.Notes);
return "Success";
}
catch (Exception ex)
{
Console.WriteLine("serialmethod" + ex.Message);
return "Failure";
}
}
public static void WriteLog(string type,
DateTime startTime, DateTime endTime,
string dataSet)
{
try
{
string logFolderPath = AppDomain.CurrentDomain.BaseDirectory + #"\Logs";
if (!Directory.Exists(logFolderPath))
Directory.CreateDirectory(logFolderPath);
string logFilePath = logFolderPath + #"\Log_" + DateTime.Today.ToString("yyyy.MM.dd") + ".csv";
string line = string.Empty;
if (!File.Exists(logFilePath))
{
line = #"""Type"",""Start Time"",""End Time"",""Duration"",""Iteration""";
writeLineToFile(logFilePath, line);
}
string duration = (endTime - startTime).ToString();
line = "\"" + type + "\"," +
"\"" + startTime.ToString("MM/dd/yyyy hh:mm:ss tt") + "\"," +
"\"" + endTime.ToString("MM/dd/yyyy hh:mm:ss tt") + "\"," +
"\"" + duration + "\"," +
"\"" + dataSet + "\"";
writeLineToFile(logFilePath, line);
}
catch (Exception)
{
//do nothing
}
}
private static void writeLineToFile(string fileName, string line)
{
using (var writer = new StreamWriter(fileName, true))
{
writer.WriteLine(line);
}
}
}
class Transactions
{
public int ID { get; set; }
public decimal Amount { get; set; }
public int Points { get; set; }
public string Notes { get; set; }
public string Created_By { get; set; }
public DateTime Created_On { get; set; }
}
Do you have any idea why this is happening. I have tried using ConcurrentBag instead of list. But that too is not helping. Please guide and let me know if I am missing anything or my implementation is completely wrong.
There a re a bunch of error-prone lines in your code:
You're overriding the reference for transaction in your foreach loop
You're using StartNew method instead of Tas.Run
You're using blocking WaitAll instead of await WhenAll, so you do block one thread in your application for no reason
You can simply switch to Parallel.Foreach instead of foreach
And most important: you're writing to the same file from different threads simultaneously, so they are basically interrupting each other. Either use some blocking to write the file (which cannot be done in parallel) or use some library for logging, like NLog or whatever, so it will handle logging for you
Your threads can run into situation when some of them trying to create file when other already done that, so move out the creation logic for file into one place (which the libraries like NLog will do for you properly)
Try to use object initializers instead of setting one property after another:
var tr = new Transactions
{
ID = 0,
Amount = r.Next(1, 10),
Created_By = "Iteration" + i,
Notes = "Iteration" + i,
Created_On = DateTime.Now
}
I'm writing a program but I'm unable to call a few of the methods I made. The errors are as follows:
-method reportMenu(String) in the type CommissionReportSystem is not applicable for the arguments ()
-Cannot make a static reference to the non-static method getSalesData() from the type CommissionReportSystem
-The method computeTotalSales() is undefined for the type CommissionReportSystem
-The method computeSalesCommission(double) in the type CommissionReportSystem is not applicable for the arguments ()
-The method showAgentCommission(double) in the type CommissionReportSystem is not applicable for the arguments ()
I've tried a lot of fixes but nothing seems to be sticking and I'm unsure of how to proceed. I've included the relevant parts of the code below. I would appreciate any tips on how to fix any of these. Thank you!
import java.io.*;
import java.text.*;
import java.util.*;
public class CommissionReportSystem {
private static final String String = null;
public static void main(String[] args) {
getSalesData ();
computeTotalSales ();
computeSalesCommission ();
showAgentCommission ();
shutdown ();
}
String [] getSalesData (){
String [] data = new String [2];
String ticketsSold = "";
String ticketPrice = "";
String buffer = new String ();
data[0] = buffer;
data[1] = buffer;
BufferedReader br = null;
try {
br = new BufferedReader (new InputStreamReader(System.in));
System.out.print ("Enter tickets sold:");
buffer = br.readLine ();
ticketsSold = buffer;
System.out.print ("Enter ticket price:");
buffer = br.readLine ();
ticketPrice = buffer;
} catch (Exception e) {
System.out.println ("Invalid entry");
}
data [0] = ticketsSold;
data [1] = ticketPrice;
return data;
}
public static double totalSales (String ticketsSold, String ticketPrice){
int ticketsSoldNum = Integer.parseInt(ticketsSold);
double ticketPriceNum = Double.parseDouble(ticketPrice);
double totalSalesNum = ticketsSoldNum * ticketPriceNum;
return totalSalesNum;}
public static final double computeSalesCommission (double totalSalesNum){
final double rate1 = 0.025;
final double rate2 = 0.0375;
final double rate3 = 0.0425;
final double salesLimit1 = 2000;
final double salesLimit2 = 4000;
final double agentCommission= 0;
if (totalSalesNum <= 2000) {
agentCommission = rate1 * totalSalesNum;
}
else if (totalSalesNum <= 4000){
agentCommission = rate2 * totalSalesNum;
}
else (totalSalesNum > 4000){
agentCommission = rate3 * totalSalesNum;
}
return agentCommission;
}
public static void showAgentCommission (double agentCommission){
System.out.format ("Congratulation agent Cindy Smith, your current daily commission:" + agentCommission);
}
public static void shutdown (){
System.out.format ("Thank you for your time! Have a great day!");
}
public static void handleInvalidData (){
}
}
1) getSalesData() is an instance method. If you want to call an instance method, create an object of the class and call method using that. Else you have to make the method static. Remember one the thing you cannot access the instance variables inside static method.
2) There is no method computeTotalSales() in your class.
3) computeSalesCommission() requires an argument of type double. You have called it without any argument.
4) The last comment is also valid for showAgentCommission().
I have WPF application with PcapDotNet DLL's that measure my machine Interface Rate.
This is the Model:
public class Interface
{
public PacketDevice PacketDevice { get { return livePacketDevice; } }
private DateTime _lastTimestamp;
private double _bitsPerSecond;
private double _packetsPerSecond;
private DateTime _lastTimestamp;
private static List<Interface> _machineInterfaces; // list of all machine interfaces
public void Start(Interface inf)
{
OpenAdapterForStatistics(inf.PacketDevice);
}
public void OpenAdapterForStatistics(PacketDevice selectedOutputDevice)
{
if (selectedOutputDevice != null)
{
using (PacketCommunicator statCommunicator = selectedOutputDevice.Open(100, PacketDeviceOpenAttributes.Promiscuous, 1000)) //open the output adapter
{
try
{
statCommunicator.Mode = PacketCommunicatorMode.Statistics; //put the interface in statstics mode
statCommunicator.ReceiveStatistics(0, StatisticsHandler); //start the main loop
}
catch (Exception)
{ }
}
}
}
private void StatisticsHandler(PacketSampleStatistics statistics)
{
DateTime currentTimestamp = statistics.Timestamp; //current sample time
DateTime previousTimestamp = _lastTimestamp; //previous sample time
_lastTimestamp = currentTimestamp; //set _lastTimestamp for the next iteration
if (previousTimestamp == DateTime.MinValue) //if there wasn't a previous sample than skip this iteration (it's the first iteration)
return;
double delayInSeconds = (currentTimestamp - previousTimestamp).TotalSeconds; //calculate the delay from the last sample
_bitsPerSecond = statistics.AcceptedBytes * 8 / delayInSeconds; //calculate bits per second
_packetsPerSecond = statistics.AcceptedPackets / delayInSeconds; //calculate packets per second
if (NewPointEventHandler != null)
NewPointEventHandler(_bitsPerSecond);
double value = _packetsPerSecond;
}
As you can see Start method start to measure the Interface rate and put the values into 2 fields:
_bitsPerSecond and _packetsPerSecond.
So after the application start i have this field:
List<Interface> _machineInterfaces;
That read all my machine interfaces.
After that i start my Start method:
private void StartStatistics()
{
int index = listview.SelectedIndex; // select the selected interface from my `ListView` list.
Interface inf = new Interface();
ThreadStart tStarter = delegate
{
inf.Start(Interface.MachineInterfaces[index]); // send the selected interface
};
Thread thread = new Thread(tStarter);
thread.IsBackground = true;
thread.Start();
statisticsTimer.Start(); // start my timer
}
OK so now this is my Question:
This is my Timer Tick Event:
public RadObservableCollection<double> mbitPerSecondValue { get; private set; }
If my BitsPerSecond Class Interface member define as regular and not Static it's value is always zero:
private void statisticsTimer_Tick(object sender, EventArgs e)
{
int index = listview.SelectedIndex;
double bps = Interface.MachineInterfaces[index].BitsPerSecond; // always zero !
mbitPerSecondValue.Add(bps);
}
In case BitsPerSecond define as static all good:
private void statisticsTimer_Tick(object sender, EventArgs e)
{
int index = listview.SelectedIndex;
double bps = Interface.BitsPerSecond;
mbitPerSecondValue.Add(bps);
}
So my question is why ?
Edit
Currently i changed my function:
private void StartStatistics()
{
int index = lvAdapters.SelectedIndex;
Interface inf = new Interface();
ThreadStart tStarter = delegate
{
foreach (Interface item in Interface.MachineInterfaces)
item.Start();
};
Thread thread = new Thread(tStarter);
thread.IsBackground = true;
thread.Start();
statisticsTimer.Start();
}
What i want to achieve is to open statistics on each interface on my machine , but again in the first interface (i have 2) i can see the traffic changing (BitsPerSecond) but in the second interface it's always zero (i make sure to generate some traffic through this interface so it not supposed to be zero)
For your second question, try to call each Interface's Start from different threads. The only suspicious thing I see is that maybe statCommunicator.ReceiveStatistics is blocking the thread and preventing the other Interfaces from being Started.
This should avoid that problem:
private void StartStatistics()
{
foreach (Interface item in Interface.MachineInterfaces)
{
ThreadStart tStarter = delegate
{
item.Start();
};
Thread thread = new Thread(tStarter);
thread.IsBackground = true;
thread.Start();
}
statisticsTimer.Start();
}
Well, it's obvious why it's working when defined as static: all instances of Interface are sharing the same property, so when you increment its value from one place, the new value is automatically available everywhere.
But as a regular non-static property, you have to make sure you're reading from the same instance you've previously modified. And you're not.
First of all, you're creating a new Interface (let's call it Interface A) and then calling its Start, passing another Interface (that we'll call Interface B) that you get from Interface.MachineInterfaces, as parameter:
private void StartStatistics()
{
...
Interface inf = new Interface();
ThreadStart tStarter = delegate
{
inf.Start(Interface.MachineInterfaces[index]); // send the selected interface
};
...
}
Inside the Start method of Interface A, you're subscribing to the statistics of Interface B, but the handler is still in Interface A:
public void Start(Interface inf)
{
OpenAdapterForStatistics(inf.PacketDevice);
}
public void OpenAdapterForStatistics(PacketDevice selectedOutputDevice)
{
...
statCommunicator.ReceiveStatistics(0, StatisticsHandler); //start the main loop
...
}
And when the handler in Interface A is called, it increments its own _bitsPerSecond value. Not Interface B's, but Interface A's.
private void StatisticsHandler(PacketSampleStatistics statistics)
{
...
_bitsPerSecond = statistics.AcceptedBytes * 8 / delayInSeconds; //calculate bits per second
...
}
But in the end, you're checking the value of BitsPerSecond in Interface B, taken again from Interface.MachineInterfaces!
private void statisticsTimer_Tick(object sender, EventArgs e)
{
...
double bps = Interface.MachineInterfaces[index].BitsPerSecond; // always zero !
...
}
-- SUGGESTED SOLUTION 1 --
Why don't you just make it so Start uses its own instance, so you don't have to create a new Interface just to use it?
public void Start()
{
OpenAdapterForStatistics(this.PacketDevice);
}
That way you can just do:
private void StartStatistics()
{
int index = listview.SelectedIndex; // select the selected interface from my `ListView` list.
ThreadStart tStarter = delegate
{
Interface.MachineInterfaces[index].Start(); // send the selected interface
};
Thread thread = new Thread(tStarter);
thread.IsBackground = true;
thread.Start();
statisticsTimer.Start(); // start my timer
}
...And you should get the desired output in your Timer Tick callback.
-- SUGGESTED SOLUTION 2 --
If you don't want to call Start from the original Interfaces inside Interface.MachineInterfaces, then you'll have to store the new Interface in some kind of Dictionary, so you can access it later to get BitsPerSecond from it:
private Dictionary<Interface, Interface> InterfaceDictionary = new Dictionary<Interface, Interface>();
private void StartStatistics()
{
int index = listview.SelectedIndex; // select the selected interface from my `ListView` list.
Interface inf = new Interface();
ThreadStart tStarter = delegate
{
inf.Start(Interface.MachineInterfaces[index]); // send the selected interface
};
Thread thread = new Thread(tStarter);
thread.IsBackground = true;
thread.Start();
statisticsTimer.Start(); // start my timer
if (InterfaceDictionary.ContainsKey(Interface.MachineInterfaces[index]))
InterfaceDictionary[Interface.MachineInterfaces[index]] = inf;
else
InterfaceDictionary.Add(Interface.MachineInterfaces[index], inf);
}
And in your Timer Tick callback, grab the data from the associated Interface, not from the one in Interface.MachineInterfaces:
private void statisticsTimer_Tick(object sender, EventArgs e)
{
int index = listview.SelectedIndex;
var interface = InterfaceDictionary[Interface.MachineInterfaces[index]];
double bps = interface.BitsPerSecond;
mbitPerSecondValue.Add(bps);
}
I am trying to input a file that contains the first and last names of several individuals from a file into a java program. I have a People class that has two Strings for the first and last names, as well as, accessors and mutators to access the information. Inside my main method, I have a while loop that brings in each person line by line until the end of the file. It is suppose to create a new instance of Person through the constructor for each line and make a copy to the array. When I print out the contents of the array once the while loop is over with, it seems that the array is filled with the information of the last person in the file. However, if I comment out the String[] values = line.split("\t"); and Person child = new Person(values[0], values[1]); lines and use a double dimensional array to hold a copy of all the information in the file, then it works fine. Is there something that I am doing wrong that is preventing me from retaining a copy of all the individual’s names contained in the file in the People array?
public class Person
{
protected static String first;
protected static String last;
private static int id;
public Person(String l, String f)
{
last = l;
first = f;
} // end of constructor
public String getFirst()
{
return first;
} // end of getFirst method
public static String getLast()
{
return last;
} // end of getLast method
public static int getID()
{
return id;
} // end of getLast method
public static void setFirst(String name)
{
first = name;
} // end of setFirst method
public static void setLast(String name)
{
last = name;
} // end of setLast method
public static void setID(int num)
{
id = num;
} // end of setLast method
} // end of Person class
public class Driver
{
public static void main(String arg[])
{
Person[] temp = new Person[10];
try
{
BufferedReader br = new BufferedReader(new FileReader(arg[1]));
String line = null;
int counter = 0;
while ((line = br.readLine()) != null)
{
String[] values = line.split("\t");
Person child = new Person(values[0], values[1]);
temp[counter] = child;
System.out.println("Index " + counter + ": Last: " + child.getLast() + " First: " + child.getFirst());
System.out.println("Index " + counter + ": Last: " + temp[counter].getLast() + " First: " + temp[counter].getFirst() + "\n");
counter++;
}
br.close();
}
catch(Exception e)
{
System.out.println("Could not find file");
}
for(int row = 0; row < 7; row++)
{
System.out.print("Row: " + row + " Last: " + temp[row].getLast() + " First: " + temp[row].getFirst() + "\n");
}
}
} // end of Driver class
The fields in class Person should not be static, a static field means shares the value for all instances of the class, that means all the 10 instances Person have the same "first", "last" and "id" values. And you need to change the methods of Person to non-static too, since static method cannot access static fields.