In Apache Flink, I am not able to see the output in std out, but my job is running successfully and data is coming
As you are running your job on a cluster, DataStreams are printed to the stdout of the TaskManager process. This TaskManager stdout is directed to an .out file in the ./log/ directory of the Flink root directory. I believe this is here you have seen your output.
I don't know if it is possible to change the stdout of TaskManagers, however, a quick and dirty solution could be to write the output to a socket :
output.writeToSocket(outputHost, outputPort, new SimpleStringSchema())
public static void main(String[] args) throws Exception {
// the host and the port to connect to
final String hostname = "192.168.1.73";
final int port = 9000;
final StreamExecutionEnvironment env = StreamExecutionEnvironment.createRemoteEnvironment("192.168.1.68", 6123);
// get input data by connecting to the socket
DataStream<String> text = env.socketTextStream(hostname, port, "\n");
// parse the data, group it, window it, and aggregate the counts
DataStream<WordWithCount> windowCounts = text
.flatMap(new FlatMapFunction<String, WordWithCount>() {
public void flatMap(String value, Collector<WordWithCount> out) {
for (String word : value.split("\\s")) {
out.collect(new WordWithCount(word, 1L));
}
}
})
.keyBy("word").timeWindow(Time.seconds(5))
.reduce(new ReduceFunction<WordWithCount>() {
public WordWithCount reduce(WordWithCount a, WordWithCount b) {
return new WordWithCount(a.word, a.count + b.count);
}
});
// print the results with a single thread, rather than in parallel
windowCounts.print().setParallelism(1);
env.execute("Socket Window WordCount");
}
public static class WordWithCount {
public String word;
public long count;
public WordWithCount() {
}
public WordWithCount(String word, long count) {
this.word = word;
this.count = count;
}
#Override
public String toString() {
return word + " : " + count;
}
}
Related
This is my first app on Android with Java and SQLite.
ISSUE:
I have a local SQLIte db on my app. I was very surprised to see how easy it is to get access to the db once you have installed the app (no need to be a programmer nor a hacker).
I tried adding SQLCipher to my app but it only worked for newer Android versions 11 & 12 and didn't work for Android 9 for example and it did make my app's size much bigger.
After researching more I found a better solution for my case which doesn"t involve crypting the db with SQLCipher but rather it consists of corrupting the first bytes of the db file then after each launch of the app the code will decorrupt the file and use the fixed file instead. This insures that anyone who decompiles the apk will only get access to a corrupt db file and will have to put more effort to fix it which is my goal.
I came across this solution in a reply [here][1] but I don't know how to implement it as I am new to Android and SQLite programming. Any help is much appreciated on how to actually do it.
These are the steps as mentioned by the user: farhad.kargaran which need more explanation as I don't get how to do it:
1- corrupt the db file (convert it to byte array and change some values)
2- copy it in asset folder
3- in first run fix corrupted file from asset and copy it in database
folder.
Change first 200 byte values like this:
int index = 0;
for(int i=0;i<100;i++)
{
byte tmp = b[index];
b[index] = b[index + 1];
b[index + 1] = tmp;
index += 2;
}
As only the first 200 bytes were replaced, the same code is used for fixing first 200 byte values.
Here is my code for the SQLiteOpenHelper if needed:
public class DatabaseHelper extends SQLiteOpenHelper {
private static final String TAG = DatabaseHelper.class.getSimpleName();
public static String DB_PATH;
public static String DB_NAME;
public SQLiteDatabase database;
public final Context context;
public SQLiteDatabase getDb() {
return database;
}
public DatabaseHelper(Context context, String databaseName, int db_version) {
super(context, databaseName, null, db_version);
this.context = context;
DB_PATH = getReadableDatabase().getPath();
DB_NAME = databaseName;
openDataBase();
// prepare if need to upgrade
int cur_version = database.getVersion();
if (cur_version == 0) database.setVersion(1);
Log.d(TAG, "DB version : " + db_version);
if (cur_version < db_version) {
try {
copyDataBase();
Log.d(TAG, "Upgrade DB from v." + cur_version + " to v." + db_version);
database.setVersion(db_version);
} catch (IOException e) {
Log.d(TAG, "Upgrade error");
throw new Error("Error upgrade database!");
}
}
}
public void createDataBase() {
boolean dbExist = checkDataBase();
if (!dbExist) {
this.getReadableDatabase();
this.close();
try {
copyDataBase();
} catch (IOException e) {
Log.e(TAG, "Copying error");
throw new Error("Error copying database!");
}
} else {
Log.i(this.getClass().toString(), "Database already exists");
}
}
private boolean checkDataBase() {
SQLiteDatabase checkDb = null;
try {
String path = DB_PATH + DB_NAME;
checkDb = SQLiteDatabase.openDatabase(path, null, SQLiteDatabase.OPEN_READONLY);
} catch (SQLException e) {
Log.e(TAG, "Error while checking db");
}
if (checkDb != null) {
checkDb.close();
}
return checkDb != null;
}
private void copyDataBase() throws IOException {
InputStream externalDbStream = context.getAssets().open(DB_NAME);
String outFileName = DB_PATH + DB_NAME;
OutputStream localDbStream = new FileOutputStream(outFileName);
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = externalDbStream.read(buffer)) > 0) {
localDbStream.write(buffer, 0, bytesRead);
}
localDbStream.close();
externalDbStream.close();
}
public SQLiteDatabase openDataBase() throws SQLException {
String path = DB_PATH + DB_NAME;
if (database == null) {
createDataBase();
database = SQLiteDatabase.openDatabase(path, null, SQLiteDatabase.OPEN_READWRITE);
}
return database;
}
#Override
public synchronized void close() {
if (database != null) {
database.close();
}
super.close();
}
Much appreciated.
[1]: https://stackoverflow.com/a/63637685/18684673
As part of the copyDatabase, correct and then write the corrupted data, then copy the rest.
Could be done various ways
e.g.
long buffersRead = 0; //<<<<< ADDED for detecting first buffer
byte[] buffer = new byte[1024];
int bytesRead;
while ((bytesRead = externalDbStream.read(buffer)) > 0) {
if (bufferesRead++ < 1) {
//correct the first 200 bytes here before writing ....
}
localDbStream.write(buffer, 0, bytesRead);
}
I would like to understand how implement some form of concurrency that would limit the number of readers.
void *reader(void *rno)
{
// Reader acquire the lock before modifying numreader
pthread_mutex_lock(&mutex);
numreader++;
if(numreader == 1) {
sem_wait(&wrt);
}
pthread_mutex_unlock(&mutex);
// Reading Section
printf("Reader %d: read cnt as %d\n",*((int *)rno),cnt);
// Reader acquire the lock before modifying numreader
pthread_mutex_lock(&mutex);
numreader--;
if(numreader == 0) {
sem_post(&wrt);
}
pthread_mutex_unlock(&mutex);
}
I've tried this, While this just enables all readers to run and calls the writers afterwards, I want to understand how to call a limited number of readers at a time, then, allow writers to run.
wrt is a binary semaphore but if it is a counting semaphore initialized to the limit, does this help in achieving the goal?
Yes, a counting semaphore with the count set to the max reader count can solve it. Below is a little Java code you can run to understand.
import java.util.ArrayList;
import java.util.List;
import static java.lang.System.currentTimeMillis;
class CountingSemaphore {
private final int maxCount;
private int signals = 0;
public CountingSemaphore(int maxCount) {
this.maxCount = maxCount;
}
public synchronized void take() throws InterruptedException {
while (signals==this.maxCount) wait();
this.signals++;
this.notify();
}
public synchronized void release() throws InterruptedException{
while(this.signals == 0) wait();
this.signals--;
this.notify();
}
public synchronized int getThreadCount(){
return signals;
}
}
class Reader{
private final int id;
public Reader(int id) {
this.id = id;
}
public void read(CountingSemaphore semaphore) throws InterruptedException {
semaphore.take();
System.out.printf("Reader %d is reading. Reader count is %d%n", id, semaphore.getThreadCount());
semaphore.release();
}
}
public class ConcurrentReaderTest {
public static void main(String[] args){
CountingSemaphore countingSemaphore = new CountingSemaphore(5);
List<Thread> readerThreads = new ArrayList<>();
for(int i=0;i<10;i++){
Reader reader = new Reader(i);
readerThreads.add(new Thread(()-> {
try {
reader.read(countingSemaphore);
} catch (InterruptedException e) {
e.printStackTrace();
}
}));
}
readerThreads.stream().forEach(r->r.start());
long startTime = currentTimeMillis();
readerThreads.stream().forEach(r->r.run());
System.out.println("time taken: "+ String.valueOf(currentTimeMillis()-startTime));
}
}
Write a program that shall calculate the vocabulary richness of a text in a file and the frequency of the most common word. The vocabulary richness is the number of words in the text divided by the number of distinct words. The frequency of a word is the number of times the word is mentioned in the text divided by the total number of words in the text.
Define and implement class WordCounter with two private fields String word and int count, constructor WordCounter(String word), and public methods String getName(), int getCount(), and void addToCounter().
Define and implement class Corpus (as in text corpus) with one private field ArrayList<WordCounter> words, constructor Corpus(BufferedReader infile), and public methods double getVocabularyRichness() and String getMostFrequentWord().
Implement a test program (as the public static void main method in Corpus) that reads all files in a specific folder, creates a Corpus object from each (previously opened) file, and saves the requested statistics into another file stats.csv. You can either create a new Corpus object for each file or define an ArrayList<Corpus> of the corpora.
Each line of the CSV file must consist of three fields separated by commas (but no spaces!): the file name, the vocabulary richness, and the most frequently used word. Run your program on all Shakespeare's plays. Submit the CSV file together with the Java file.
I wrote what I think is the correct implementation of the HW problem because it works properly for some of the text files, however only the words.get(i).getName() (I tested with words.get(i).getCount()) method will print a blank space for some of the files. I have tried everything, and can't seem to figure it out. Can you please give me a hint or some guidance as to how to fix this issue?
public class Corpus {
private ArrayList<WordCounter> words = new ArrayList <WordCounter>() ;
Corpus(BufferedReader infile){
String ln;
try {
while((ln = infile.readLine()) != null) {
for (String word : ln.toLowerCase().split("([,.\\s]+)")) {
int reference = 0;
for(int i = 0; i < words.size(); i++) {
if (word.equals(words.get(i).getName())) {
reference++;
words.get(i).addToCounter();
} }
if (reference==0) { words.add(new WordCounter(word)); }
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
public double getVocabularyRichness() {
int word_count=0;
for(int i = 0; i < words.size(); i++) {
word_count=word_count+words.get(i).getCount();
}
return (double)word_count/(double)words.size();
}
public String getMostFrequentWord() {
String winner = "*AN ERROR OCCURRED*";
int max_count = 0;
for(int i = 0; i < words.size(); i++) {
if(words.get(i).getCount() > max_count){
max_count = words.get(i).getCount();
}
}
for(int i = 0; i < words.size(); i++) {
if(words.get(i).getCount() == max_count){
winner = words.get(i).getName();
}
}
//winner="Test " + String.valueOf(words.get(i).getName());;
//return String.valueOf(max_count);
return winner;
}
public static void main(String [] args) throws Exception{
BufferedWriter writer = null;
File folder_location = new File("/Users/joaquindelaguardia/Desktop/Shakespeare");
File[] file_array = folder_location.listFiles();
for(File iteration_file: file_array) {
FileReader current_file = new FileReader(iteration_file);
BufferedReader infile = new BufferedReader(current_file);
Corpus obj1 = new Corpus(infile);
String file_name = iteration_file.getName();
String frequent_word = obj1.getMostFrequentWord();
String vocabulary_richness = String.valueOf(obj1.getVocabularyRichness());
System.out.println(file_name);
System.out.println(frequent_word);
System.out.println(vocabulary_richness);
System.out.println("-----------------------------");
//FileWriter file_writer = new FileWriter("/Users/joaquindelaguardia/Desktop/stats.csv");
//writer = new BufferedWriter(file_writer);
//String output = file_name+", "+frequent_word+", "+vocabulary_richness + "\n";
//writer.append(output);
}
//writer.close();
}
}
public class WordCounter {
private String word;
private int count=1;
WordCounter(String word){
this.word=word;
}
public String getName() {
return word;
}
public int getCount() {
return count;
}
public void addToCounter() {
count++;
}
}
Im testing the information by printing before appending to file, and as you can see with the small fragment of the output included below, for some cases it prints the most common word (and) while in the second case it doesn't print anything.
shakespeare-lovers-62.txt
and
2.2409948542024014
shakespeare-julius-26.txt
6.413205537806177
I am new in using TPL in .Net applications. While creating a simple console application to achieve some parallel tasks those are dynamically created, I am stuck with some issues.
Problem here is that when 10 tasks are created and run, although the console is showing all the 10 tasks, when writing those into a log file after putting a delay between consoling and logging, the log file misses some of the items randomly.
Below is my sample code (This is just a skeleton of my actual code)
class Program
{
public static int datacount = 10;
static void Main(string[] args)
{
List<Task> tasks = new List<Task>();
var s1 = DateTime.Now;
var transList = GenerateTransactionList();
foreach (var transaction in transList)
{
Transactions transactionNew = new Transactions();
transactionNew = transaction;
tasks.Add(Task.Factory.StartNew(() => serialMethod(transactionNew)));
}
Task.WhenAll(tasks).Wait();
Console.WriteLine("Completed!!!");
}
private static List<Transactions> GenerateTransactionList()
{
Random r = new Random();
List<Transactions> transactionList = new List<Transactions>();
for (int i = 1; i <= datacount; ++i)
{
Transactions tr = new Transactions();
tr.ID = 0;
tr.Amount = r.Next(1, 10);
tr.Created_By = "Iteration" + i;
tr.Notes = "Iteration" + i;
tr.Created_On = DateTime.Now;
transactionList.Add(tr);
}
return transactionList;
}
private static async Task<string> serialMethod(Transactions tlist)
{
Console.WriteLine("Started Serial Iteration" + tlist.Notes);
try
{
Console.WriteLine("Finished Serial Iteration" + tlist.Notes);
Thread.Sleep(10000);//doing some time consuming process
WriteLog("Parallel2", DateTime.Now, DateTime.Now, tlist.Notes);
return "Success";
}
catch (Exception ex)
{
Console.WriteLine("serialmethod" + ex.Message);
return "Failure";
}
}
public static void WriteLog(string type,
DateTime startTime, DateTime endTime,
string dataSet)
{
try
{
string logFolderPath = AppDomain.CurrentDomain.BaseDirectory + #"\Logs";
if (!Directory.Exists(logFolderPath))
Directory.CreateDirectory(logFolderPath);
string logFilePath = logFolderPath + #"\Log_" + DateTime.Today.ToString("yyyy.MM.dd") + ".csv";
string line = string.Empty;
if (!File.Exists(logFilePath))
{
line = #"""Type"",""Start Time"",""End Time"",""Duration"",""Iteration""";
writeLineToFile(logFilePath, line);
}
string duration = (endTime - startTime).ToString();
line = "\"" + type + "\"," +
"\"" + startTime.ToString("MM/dd/yyyy hh:mm:ss tt") + "\"," +
"\"" + endTime.ToString("MM/dd/yyyy hh:mm:ss tt") + "\"," +
"\"" + duration + "\"," +
"\"" + dataSet + "\"";
writeLineToFile(logFilePath, line);
}
catch (Exception)
{
//do nothing
}
}
private static void writeLineToFile(string fileName, string line)
{
using (var writer = new StreamWriter(fileName, true))
{
writer.WriteLine(line);
}
}
}
class Transactions
{
public int ID { get; set; }
public decimal Amount { get; set; }
public int Points { get; set; }
public string Notes { get; set; }
public string Created_By { get; set; }
public DateTime Created_On { get; set; }
}
Do you have any idea why this is happening. I have tried using ConcurrentBag instead of list. But that too is not helping. Please guide and let me know if I am missing anything or my implementation is completely wrong.
There a re a bunch of error-prone lines in your code:
You're overriding the reference for transaction in your foreach loop
You're using StartNew method instead of Tas.Run
You're using blocking WaitAll instead of await WhenAll, so you do block one thread in your application for no reason
You can simply switch to Parallel.Foreach instead of foreach
And most important: you're writing to the same file from different threads simultaneously, so they are basically interrupting each other. Either use some blocking to write the file (which cannot be done in parallel) or use some library for logging, like NLog or whatever, so it will handle logging for you
Your threads can run into situation when some of them trying to create file when other already done that, so move out the creation logic for file into one place (which the libraries like NLog will do for you properly)
Try to use object initializers instead of setting one property after another:
var tr = new Transactions
{
ID = 0,
Amount = r.Next(1, 10),
Created_By = "Iteration" + i,
Notes = "Iteration" + i,
Created_On = DateTime.Now
}
I want to use the class IpAddress instead of String in my List, like this:
ArrayList<IpAddress> IpAddresses = new ArrayList<>();
But it doesn't work. I cannot add the string line to my list, so the program does not work as I want. What should I do?
public class IpAddress implements Comparable<IpAddress> {
private String ip;
public IpAddress(String ip) {
this.ip = ip;
}
#Override
public int compareTo(IpAddress o) {
return this.ip.compareTo(o.ip);
}
}
public class IPvLIST {
public static void main(String[] args) throws IOException {
FileReader fr = new FileReader(new File("ip.txt"));
BufferedReader br = new BufferedReader(fr);
ArrayList<String> IpAddresses = new ArrayList<>();
String line;
int n = 0;
while ((line = br.readLine()) != null) {
IpAddresses.add(line);
}
}
}
You need to convert the string line to an IpAddress. Based on the code you have, change the while statement to
while ((line = br.readLine()) != null) {
IpAddresses.add(new IpAddress(line));
}
Depending if this is a simple test program or something more, you might also look into the .NET class IPAddress. This has a parse function that will take the string and convert it to an IPAddress which makes it easier to perform actual network functions.