Kotlin room database boolean - database

Im trying to create an app where a user can has a list of goals of steps to reach they create and then choose one of them to be active and to follow. The database works when I was just using the goal id, name, and steps but now I realised I need to insert another column defining when a goal is active so Im trying to insert that, however I don't know how I should handle the boolean especially in the repository and viewModel. I'd appreciate any help. Thanks in advance
here's my code
interface Dao {
#Insert(onConflict = OnConflictStrategy.IGNORE)
suspend fun insert(goal: Goal)
#Update
suspend fun updateGoal(goal: Goal)
#Query("SELECT * FROM user_goal_table order by goalId")
fun getAll(): LiveData<List<Goal>>
#Query("SELECT * FROM user_goal_table WHERE goalId = :key")
suspend fun getGoal(key: Int): Goal
#Delete
suspend fun delete(goal: Goal)
#Query("SELECT * FROM user_goal_table WHERE goal_is_active = 1 order by goalId")
suspend fun makeGoalActive(key: Int): Goal
class Repository (private val dao : Dao){
val allGoals: LiveData<List<Goal>> = Dao.getAll()
suspend fun insert(goal: Goal){
dao.insert(goal)
}
suspend fun update(goal: Goal){
dao.update(goal)
}
suspend fun delete(goal: Goal){
dao.delete(goal)
}
suspend fun active(goal: Goal, int: Int){
dao.makeGoalActive(int)
}
class ViewModel (application: Application) : AndroidViewModel(application) {
val allGoals: LiveData<List<Goal>>
private val repository: Repository
init{
val dao = GoalDatabase.getInstance(application).getGoalDao()
repo = Repository(dao)
allGoals = repository.allGoals
}
fun insert(goal: Goal) = viewModelScope.launch(Dispatchers.IO){
repository.insert(goal)
}
fun update(goal: Goal) = viewModelScope.launch(Dispatchers.IO){
repository.update(goal)
}
fun delete(goal: Goal) = viewModelScope.launch(Dispatchers.IO){
repository.delete(goal)
}

Just add a new property like isActive: Boolean to your Goal class and then use #Update annotation in Room (that you've already implemented in updateGoal(goal: Goal) method of your Dao) or UPDATE command itself in SQLite to update the row you want to change its isActive state. For using SQLite do something like below:
#Query("UPDATE user_goal_table SET isActive = 1 WHERE goalId = :goalId")
suspend fun makeGoalActive(goalId: Int)
For boolean properties, use 1 for true and 0 for false in SQLite commands.
In Repository, this method is enough:
suspend fun active(goalId: Int) {
dao.makeGoalActive(int)
}
And in the ViewModel:
fun insert(goal: Goal) = viewModelScope.launch {
repository.insert(goal)
}
Btw, you don't need to determine IO dispatcher for Room methods, Room uses its own dispatcher to run queries.

Related

Room Data Base Create Instance

I want to Create An Instance Of Room Data base in Composable
But
val db = Room.databaseBuilder(applicationContext, UserDatabase::class.java,"users.db").build()
is not working here not getting applicationContext
How to create an instance of context in composable
Have you tried getting the context with : val context = LocalContext.current and then adding this to get your applicationContext?
Like this: context.applicationContext or using simply val db = Room.databaseBuilder(context, UserDatabase::class.java,"users.db").build()
Room (and the underlying SQliteOpenHelper) only need the context to open the database (or more correctly to instantiate the underlying SQLiteOpenHelper).
Room/Android SQLiteOpenHelper uses the context to ascertain the Application's standard (recommended) location (data/data/<the_package_name>/databases). e.g. in the following demo (via Device Explorer):-
The database, as it is still open includes 3 files (the -wal and -shm are the Write Ahead Logging files that will at sometime be committed/written to the actual database (SQLite handles that)).
so roughly speaking Room only needs to have the context so that it can ascertain /data/data/a.a.so75008030kotlinroomgetinstancewithoutcontext/databases/testit.db (in the case of the demo).
So if you cannot use the applicationContext method then you can circumvent the need to provide the context, if using a singleton approach AND if after instantiating the singleton.
Perhaps consider this demo:-
First some pretty basic DB Stuff (table (#Entity annotated class), DAO functions and #Database annotated abstract class WITH singleton approach). BUT with some additional functions for accessing the instance without the context.
#Entity
data class TestIt(
#PrimaryKey
val testIt_id: Long?=null,
val testIt_name: String
)
#Dao
interface DAOs {
#Insert(onConflict = OnConflictStrategy.IGNORE)
fun insert(testIt: TestIt): Long
#Query("SELECT * FROM testit")
fun getAllTestItRows(): List<TestIt>
}
#Database(entities = [TestIt::class], exportSchema = false, version = 1)
abstract class TestItDatabase: RoomDatabase() {
abstract fun getDAOs(): DAOs
companion object {
private var instance: TestItDatabase?=null
/* Extra/not typical for without a context (if wanted)*/
fun isInstanceWithoutContextAvailable() : Boolean {
return instance != null
}
/******************************************************/
/* Extra/not typical for without a context */
/******************************************************/
fun getInstanceWithoutContext(): TestItDatabase? {
if (instance != null) {
return instance as TestItDatabase
}
return null
}
/* Typically the only function*/
fun getInstance(context: Context): TestItDatabase {
if (instance==null) {
instance = Room.databaseBuilder(context,TestItDatabase::class.java,"testit.db")
.allowMainThreadQueries() /* for convenience/brevity of demo */
.build()
}
return instance as TestItDatabase
}
}
}
And to demonstrate (within an activity for brevity) :-
class MainActivity : AppCompatActivity() {
lateinit var roomInstance: TestItDatabase
lateinit var dao: DAOs
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
roomInstance = TestItDatabase.getInstance(this) /* MUST be used before withoutContext functions but could be elsewhere shown here for brevity */
dao = roomInstance.getDAOs()
//dao.insert(TestIt(testIt_name = "New001")) /* Removed to test actually doing the database open with the without context */
logDataWithoutContext()
addRowWithoutContext()
addRowWithApplicationContext()
logDataWithoutContext()
}
private fun logDataWithoutContext() {
Log.d("${TAG}_LDWC","Room DB Instantiated = ${TestItDatabase.isInstanceWithoutContextAvailable()}")
for (t in TestItDatabase.getInstanceWithoutContext()!!.getDAOs().getAllTestItRows()) {
Log.d("${TAG}_LDWC_DATA","TestIt Name is ${t.testIt_name} ID is ${t.testIt_id}")
}
}
private fun addRowWithoutContext() {
Log.d("${TAG}_LDWC","Room DB Instantiated = ${TestItDatabase.isInstanceWithoutContextAvailable()}")
if (TestItDatabase.getInstanceWithoutContext()!!.getDAOs()
.insert(TestIt(System.currentTimeMillis(),"NEW AS PER ID (the time to millis) WITHOUT CONTEXT")) > 0) {
Log.d("${TAG}_ARWC_OK","Row successfully inserted.")
} else {
Log.d("${TAG}_ARWC_OUCH","Row was not successfully inserted (duplicate ID)")
}
}
private fun addRowWithApplicationContext() {
TestItDatabase.getInstance(applicationContext).getDAOs().insert(TestIt(System.currentTimeMillis() / 1000,"NEW AS PER ID (the time to seconds) WITH CONTEXT"))
}
}
The result output to the log showing that the database access, either way, worked:-
2023-01-05 12:45:39.020 D/DBINFO_LDWC: Room DB Instantiated = true
2023-01-05 12:45:39.074 D/DBINFO_LDWC: Room DB Instantiated = true
2023-01-05 12:45:39.077 D/DBINFO_ARWC_OK: Row successfully inserted.
2023-01-05 12:45:39.096 D/DBINFO_LDWC: Room DB Instantiated = true
2023-01-05 12:45:39.098 D/DBINFO_LDWC_DATA: TestIt Name is NEW AS PER ID (the time to seconds) WITH CONTEXT ID is 1672883139
2023-01-05 12:45:39.098 D/DBINFO_LDWC_DATA: TestIt Name is NEW AS PER ID (the time to millis) WITHOUT CONTEXT ID is 1672883139075
note that the shorter id was the last added but appears first due to it being selected first as it appears earlier in the index that the SQlite Query Optimiser would have used (aka the Primary Key).
basically the same date time second wise but the first insert included milliseconds whilst the insert via AddRowWithApplicationContext drops the milliseconds.

Kotlin Room Database: How to save result from sum query? [duplicate]

This question already has answers here:
Android Room - simple select query - Cannot access database on the main thread
(23 answers)
Closed 8 months ago.
I´m new to Kotlin and trying to learn it by programming an app for work time recording.
I´ve created a room database which works fine when inserting data. In the next step I would like to retrieve the sum of a column and store this value in a variable. And this is the point where I get stuck. Here are the affected snippets of my code.
data class sumPojo(var sumOvertime: Double)
#Dao
interface OvertimeDao {
#Query(value = "SELECT SUM(overtime) as sumOvertime FROM TableOvertime")
fun getSumOvertime(): sumPojo
}
class OvertimeRepository(private val overtimeDao: OvertimeDao) {
val getSumOvertime: sumPojo = overtimeDao.getSumOvertime()
}
class OvertimeViewModel(application: Application): AndroidViewModel(application) {
private val repository : OvertimeRepository
val getSumOvertime : sumPojo
init {
val overtimeDao = OvertimeDatabase.getDatabase(application).overtimeDao()
repository = OvertimeRepository(overtimeDao)
getSumOvertime = repository.getSumOvertime
}
}
class inputWorktimeFragment : Fragment() {
private lateinit var mOvertimeViewModel : OvertimeViewModel
override fun onCreateView(
inflater: LayoutInflater, container: ViewGroup?,
savedInstanceState: Bundle?
): View? {
val view = inflater.inflate(R.layout.fragment_input_worktime, container, false)
mOvertimeViewModel = ViewModelProvider(this)[OvertimeViewModel::class.java]
val sumOvertime: sumPojo = mOvertimeViewModel.getSumOvertime
return view
}
}
Following the error message which I receive.
2022-07-06 16:24:03.067 7338-7338/com.example.workingtimerecorder E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.example.workingtimerecorder, PID: 7338
java.lang.RuntimeException: Cannot create an instance of class com.example.workingtimerecorder.data.OvertimeViewModel
at androidx.lifecycle.ViewModelProvider$AndroidViewModelFactory.create(ViewModelProvider.kt:320)
at androidx.lifecycle.ViewModelProvider$AndroidViewModelFactory.create(ViewModelProvider.kt:278)
at androidx.lifecycle.SavedStateViewModelFactory.create(SavedStateViewModelFactory.kt:128)
at androidx.lifecycle.ViewModelProvider.get(ViewModelProvider.kt:187)
at androidx.lifecycle.ViewModelProvider.get(ViewModelProvider.kt:153)
at com.example.workingtimerecorder.fragments.inputWorktimeFragment.onCreateView(InputWorktimeFragment.kt:55)
at androidx.fragment.app.Fragment.performCreateView(Fragment.java:3104)
at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:524)
at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:261)
at androidx.fragment.app.FragmentStore.moveToExpectedState(FragmentStore.java:113)
at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1424)
at androidx.fragment.app.FragmentManager.dispatchStateChange(FragmentManager.java:2968)
at androidx.fragment.app.FragmentManager.dispatchViewCreated(FragmentManager.java:2879)
at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:3129)
at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:552)
at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:261)
at androidx.fragment.app.FragmentStore.moveToExpectedState(FragmentStore.java:113)
at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1424)
at androidx.fragment.app.FragmentManager.dispatchStateChange(FragmentManager.java:2968)
at androidx.fragment.app.FragmentManager.dispatchActivityCreated(FragmentManager.java:2886)
at androidx.fragment.app.FragmentController.dispatchActivityCreated(FragmentController.java:263)
at androidx.fragment.app.FragmentActivity.onStart(FragmentActivity.java:351)
at androidx.appcompat.app.AppCompatActivity.onStart(AppCompatActivity.java:246)
at android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1455)
at android.app.Activity.performStart(Activity.java:8076)
at android.app.ActivityThread.handleStartActivity(ActivityThread.java:3660)
at android.app.servertransaction.TransactionExecutor.performLifecycleSequence(TransactionExecutor.java:221)
at android.app.servertransaction.TransactionExecutor.cycleToPath(TransactionExecutor.java:201)
at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:173)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:97)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2210)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loopOnce(Looper.java:201)
at android.os.Looper.loop(Looper.java:288)
at android.app.ActivityThread.main(ActivityThread.java:7839)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Constructor.newInstance0(Native Method)
at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
at androidx.lifecycle.ViewModelProvider$AndroidViewModelFactory.create(ViewModelProvider.kt:312)
at androidx.lifecycle.ViewModelProvider$AndroidViewModelFactory.create(ViewModelProvider.kt:278) 
at androidx.lifecycle.SavedStateViewModelFactory.create(SavedStateViewModelFactory.kt:128) 
at androidx.lifecycle.ViewModelProvider.get(ViewModelProvider.kt:187) 
at androidx.lifecycle.ViewModelProvider.get(ViewModelProvider.kt:153) 
at com.example.workingtimerecorder.fragments.inputWorktimeFragment.onCreateView(InputWorktimeFragment.kt:55) 
at androidx.fragment.app.Fragment.performCreateView(Fragment.java:3104) 
at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:524) 
at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:261) 
at androidx.fragment.app.FragmentStore.moveToExpectedState(FragmentStore.java:113) 
at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1424) 
at androidx.fragment.app.FragmentManager.dispatchStateChange(FragmentManager.java:2968) 
at androidx.fragment.app.FragmentManager.dispatchViewCreated(FragmentManager.java:2879) 
at androidx.fragment.app.Fragment.performViewCreated(Fragment.java:3129) 
at androidx.fragment.app.FragmentStateManager.createView(FragmentStateManager.java:552) 
at androidx.fragment.app.FragmentStateManager.moveToExpectedState(FragmentStateManager.java:261) 
at androidx.fragment.app.FragmentStore.moveToExpectedState(FragmentStore.java:113) 
at androidx.fragment.app.FragmentManager.moveToState(FragmentManager.java:1424) 
at androidx.fragment.app.FragmentManager.dispatchStateChange(FragmentManager.java:2968) 
at androidx.fragment.app.FragmentManager.dispatchActivityCreated(FragmentManager.java:2886) 
at androidx.fragment.app.FragmentController.dispatchActivityCreated(FragmentController.java:263) 
at androidx.fragment.app.FragmentActivity.onStart(FragmentActivity.java:351) 
at androidx.appcompat.app.AppCompatActivity.onStart(AppCompatActivity.java:246) 
at android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1455) 
at android.app.Activity.performStart(Activity.java:8076) 
at android.app.ActivityThread.handleStartActivity(ActivityThread.java:3660) 
at android.app.servertransaction.TransactionExecutor.performLifecycleSequence(TransactionExecutor.java:221) 
at android.app.servertransaction.TransactionExecutor.cycleToPath(TransactionExecutor.java:201) 
at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:173) 
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:97) 
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2210) 
at android.os.Handler.dispatchMessage(Handler.java:106) 
at android.os.Looper.loopOnce(Looper.java:201) 
at android.os.Looper.loop(Looper.java:288) 
at android.app.ActivityThread.main(ActivityThread.java:7839) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003) 
Caused by: java.lang.IllegalStateException: Cannot access database on the main thread since it may potentially lock the UI for a long period of time.
at androidx.room.RoomDatabase.assertNotMainThread(RoomDatabase.java:469)
at androidx.room.RoomDatabase.query(RoomDatabase.java:525)
at androidx.room.util.DBUtil.query(DBUtil.java:86)
2022-07-06 16:24:03.068 7338-7338/com.example.workingtimerecorder E/AndroidRuntime: at com.example.workingtimerecorder.data.OvertimeDao_Impl.getSumOvertime(OvertimeDao_Impl.java:124)
at com.example.workingtimerecorder.data.OvertimeRepository.<init>(OvertimeRepository.kt:16)
at com.example.workingtimerecorder.data.OvertimeViewModel.<init>(OvertimeViewModel.kt:24)
... 40 more
I would appreciate your help or any hints which could lead me into the rigth direction. I`ve read about several similar issues here on Stackoverflow but none of them could help me.
Caused by: java.lang.IllegalStateException: Cannot access database on the main thread since it may potentially lock the UI for a long period of time.
I'd try returning either flow or live data and using view model scope to start a co routine.
in the dao
//change
fun getSumOvertime(): sumPojo
to
fun getSumOvertime(): LiveData<sumPojo>
//or
suspend fun getSumOvertime(): Flow<sumPojo>
in the repo
//livedata
fun getSumPojo():LiveData<sumPojo>{return yourdb.yourdao.getSumovertime}
//flow
suspend fun getSumPojo():Flow<sumPojo>{return yourdb.yourdao.getSumovertime}
in the view model id gain access to the values like such
// create private and public accessors
private var _sumPojo = MutableLiveData<SumPojo> //use Int as mutable type if you do not have an object to map to
val sumPojo: LiveData<SumPojo> // again if you do not have an object for sumPojo use Int as live data type
get() = _sumPojo
private fun refreshSumPojo()=viewModelScope.launch{return yourRepo.getSumPojo()}
init {_sumPojo.value = refreshSumPojo}
in the view I would use databinding and bind the text view to the public accessor
in your view.xml file
<TextView
android:text="#{viewModel.sumPojo.toString()}"
" />

is it possible to make different keys have independent watermark

I am using Flink 1.12 and I have a keyed stream, in my code it looks that both A and B share the same watermark? and therefore B is determined as late because A's coming has upgraded the watermark to be 2020-08-30 10:50:11?
The output is A(2020-08-30 10:50:08, 2020-08-30 10:50:16):2020-08-30 10:50:15,there is no output for B
I would ask whether it is possible to make different keys have independent watermark? A's watermark and B'watermark change independently
The application code is:
import java.text.SimpleDateFormat
import java.util.Date
import java.util.concurrent.TimeUnit
import org.apache.flink.streaming.api.TimeCharacteristic
import org.apache.flink.streaming.api.functions.AssignerWithPunctuatedWatermarks
import org.apache.flink.streaming.api.scala.function.WindowFunction
import org.apache.flink.streaming.api.scala.{StreamExecutionEnvironment, _}
import org.apache.flink.streaming.api.watermark.Watermark
import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.streaming.api.windowing.windows.TimeWindow
import org.apache.flink.util.Collector
object DemoDiscardLateEvent4_KeyStream {
def to_milli(str: String) =
new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse(str).getTime
def to_char(milli: Long) = {
val date = if (milli <= 0) new Date(0) else new Date(milli)
new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(date)
}
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val data = Seq(
("A", "2020-08-30 10:50:15"),
("B", "2020-08-30 10:50:07")
)
env.fromCollection(data).setParallelism(1).assignTimestampsAndWatermarks(new AssignerWithPunctuatedWatermarks[(String, String)]() {
var maxSeen = Long.MinValue
override def checkAndGetNextWatermark(lastElement: (String, String), extractedTimestamp: Long): Watermark = {
val eventTime = to_milli(lastElement._2)
if (eventTime > maxSeen) {
maxSeen = eventTime
}
//Allow 4 seconds late
new Watermark(maxSeen - 4000)
}
override def extractTimestamp(element: (String, String), previousElementTimestamp: Long): Long = to_milli(element._2)
}).keyBy(_._1).window(TumblingEventTimeWindows.of(Time.of(8, TimeUnit.SECONDS))).apply(new WindowFunction[(String, String), String, String, TimeWindow] {
override def apply(key: String, window: TimeWindow, input: Iterable[(String, String)], out: Collector[String]): Unit = {
val start = to_char(window.getStart)
val end = to_char(window.getEnd)
val sb = new StringBuilder
//the start and end of the window
sb.append(s"$key($start, $end):")
//The content of the window
input.foreach {
e => sb.append(e._2 + ",")
}
out.collect(sb.toString().substring(0, sb.length - 1))
}
}).print()
env.execute()
}
}
While it would sometimes be helpful if Flink offered per-key watermarking, it does not.
Each parallel instance of your WatermarkStrategy (or in this case, of your AssignerWithPunctuatedWatermarks) is generating watermarks independently, based on the timestamps of the events it observes (regardless of their keys).
One way to work around the lack of this feature is to not use watermarks at all. For example, if you would be using per-key watermarks to trigger keyed event-time windows, you can instead implement your own windows using a KeyedProcessFunction, and instead of using watermarks to trigger event time timers, keep track of the largest timestamp seen so far for each key, and whenever updating that value, determine if you now want to close one or more windows for that key.
See one of the Flink training lessons for an example of how to implement keyed tumbling windows with a KeyedProcessFunction. This example depends on watermarks but should help you get started.

What's the relationship between key and Window instance in KeyedStream#timeWindow#process

For KeyedStream#timeWindow#process, I am wonderring whether one window instance will only contain the same key, and different keys will use different window instances.
From the output of the following application, i see that one window instance will only contain the same key, and different keys will use different windows.
But I want to ask and confirm, thanks!
import org.apache.flink.streaming.api.functions.source.{RichParallelSourceFunction, SourceFunction}
import scala.util.Random
class KeyByAndWindowAndProcessTestSource extends RichParallelSourceFunction[Int] {
override def run(ctx: SourceFunction.SourceContext[Int]): Unit = {
while (true) {
val i = new Random().nextInt(30)
ctx.collect(i)
ctx.collect(i)
ctx.collect(i)
Thread.sleep(1000)
}
}
override def cancel(): Unit = {
}
}
The applications is:
import org.apache.flink.streaming.api.TimeCharacteristic
import org.apache.flink.streaming.api.scala.function.ProcessWindowFunction
import org.apache.flink.streaming.api.scala.{DataStream, StreamExecutionEnvironment}
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.streaming.api.windowing.windows.TimeWindow
import org.apache.flink.util.Collector
import org.apache.flink.api.scala._
object KeyByAndWindowTest {
def main(args: Array[String]): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setParallelism(1)
env.getCheckpointConfig.setCheckpointInterval(10 * 1000)
env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime)
val ds: DataStream[Int] = env.addSource(new KeyByAndWindowAndProcessTestSource)
val ds2 = ds.keyBy(i => i).timeWindow(Time.seconds(4)).process(new MyProcessFunction())
ds2.print()
env.execute()
}
}
class MyProcessFunction extends ProcessWindowFunction[Int, String, Int, TimeWindow] {
override def process(
key: Int,
ctx: Context,
vals: Iterable[Int],
out: Collector[String]): Unit = {
println(new java.util.Date())
println(s"key=${key}, vals = ${vals.mkString(",")}, hashCode=${System.identityHashCode(ctx.window)}")
}
}
The output is:
Sat Sep 14 13:08:24 CST 2019
key=26, vals = 26,26,26, hashCode=838523304
Sat Sep 14 13:08:24 CST 2019
key=28, vals = 28,28,28, hashCode=472721641
Sat Sep 14 13:08:24 CST 2019
key=18, vals = 18,18,18,18,18,18, hashCode=1668151956
Actually, with respect to ProcessingTimeWindow, a new window object is created for each element.
Here is the source code of TumblingProcessingTimeWindows#assignWindows:
public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerContext context) {
final long now = context.getCurrentProcessingTime();
long start = TimeWindow.getWindowStartWithOffset(now, offset, size);
return Collections.singletonList(new TimeWindow(start, start + size));
}
So System.identityHashCode will always return a unique hash code for different keys, and your test code does not prove anything.
Under the hood, elements are grouped by the key of elementKey + assignedWindow, so I think it's right to say "one window instance will only contain the same key, and different keys will use different window instances".
Original Answer:
I hope I get your question right...
ProcessWindowFunction#process will be invoked for each window and key once (or multiple times depending on the windows's trigger). Internally, window and key make up a composite partition key.
In terms of Java object instances, one instance of ProcessWindowFunction will deal with many keys. Specifically, there will be degree of parallelism many ProcessWindowFunctions.
Follow Up:
So I did not get it right :)
For every record, which is processed by the WindowOperator a new Window object is created, with the correct start/end time for the record.
This means that each invocation of ProcessWindowFunction#process will be passed a new Window object.
It is important to understand, that a Window in Flink is a very light object, which is just used as an additional part (the namespace) of the overall key. It does not hold any data and/or logic.
May I ask for the background of the question?

Watermarks in a RichParallelSourceFunction

I am implementing a SourceFunction, which reads Data from a Database.
The job should be able to be resumed if stopped or crushed (i.e savepoints and checkpoints) with the data being processed exactly once.
What I have so far:
#SerialVersionUID(1L)
class JDBCSource(private val waitTimeMs: Long) extends
RichParallelSourceFunction[Event] with StoppableFunction with LazyLogging{
#transient var client: PostGreClient = _
#volatile var isRunning: Boolean = true
val DEFAULT_WAIT_TIME_MS = 1000
def this(clientConfig: Serializable) =
this(clientConfig, DEFAULT_WAIT_TIME_MS)
override def stop(): Unit = {
this.isRunning = false
}
override def open(parameters: Configuration): Unit = {
super.open(parameters)
client = new JDBCClient
}
override def run(ctx: SourceFunction.SourceContext[Event]): Unit = {
while (isRunning){
val statement = client.getConnection.createStatement()
val resultSet = statement.executeQuery("SELECT name, timestamp FROM MYTABLE")
while (resultSet.next()) {
val event: String = resultSet.getString("name")
val timestamp: Long = resultSet.getLong("timestamp")
ctx.collectWithTimestamp(new Event(name, timestamp), timestamp)
}
}
}
override def cancel(): Unit = {
isRunning = false
}
}
How can I make sure to only get the rows of the database which aren't processed yet?
I assumed the ctx variable would have some information about the current watermark so that I could change my query to something like:
select name, timestamp from myTable where timestamp > ctx.getCurrentWaterMark
But it doesn't have any relevant methods for me. Any Ideas how to solve this problem would be appreciated
You have to implement CheckpointedFunction so that you can manage checkpointing by yourself. The documentation of the interface is pretty comprehensive but if you need an example I advise you to take a look at an example.
In essence, your function must implement CheckpointedFunction#snapshotState to store the state you need using Flink's managed state and then, when performing a restore, it will read that same state in CheckpointedFunction#initializeState.

Resources