I am using NXP LPC17xx family microcontrollers (LPC1759 and LPC1768).
How can I determine if RTC is running for sure?
I am doing a test on
LPC_RTC->CCR & RTC_CCR_CLKEN
but it seems no much reliable.
I have seen values in year 3197 or so when turning on my device.
How can I tell if RTC is running and its value is not corrupt?
EDIT:
I ended up adding a simple sanity check in RTC values:
bool DateTime::validate( const RTC_TIME_Type &time_info )
{
if ( time_info.YEAR > 2100
|| time_info.DOY > 366
|| time_info.MONTH > 12
|| time_info.DOM > 31
|| time_info.HOUR > 23
|| time_info.MIN > 59
|| time_info.SEC > 59 )
return false;
return true;
}
It is run during my POST, as suggested bellow.
I battled with the RTC on that chip's grandfather (LPC2148) about 5 years ago. If you look on the Yahoo LPC2000 group (it also covers the LPC1000 chips) you'll see the RTC & its issues come up a lot.
Anyway, I'm going from memory here, but I think I concluded that reading the status register wasn't reliable enough. Maybe the issue was that when power was removed, if the battery backup was absent, things would get scrambled...
So what I recall I did was the following, during the boot phase:
(1) Enable RTC peripheral
(2) Read ALL RTC registers. In firmware, have "out of bounds" min & max values for each field (e.g. year must be at least 2005, and no larger than 2030)
(3) If any value is out of range, reset date & time to some hard-coded value (e.g. Jan. 1, 2005) (product would let user adjust time/date once booted)
(4) Take snapshot of registers; wait AT LEAST one second (use timer peripheral to measure time), then make sure the value changed. I might have gone so far during boot-up as to set the values so that a 1-second tick would/should cause everything to roll over (probably 1 second before midnight, Dec. 31), ensure everything changes, then write back the original value + 1 second. (You would want to do this right as the value changes to avoid slipping seconds)
I will try to dig up the code and see if there was something more. I just recall finally concluding I had to run the damn thing & watch it work before I passed the P.O.S.T. for that peripheral.
(I kind of mentioned it in passing, but just to re-iterate... if your values seem corrupted on power-on, make sure your battery back-up circuitry is rock solid - even a basic circuit with a pair of diodes normally suffices. Could be that the clock is running when the product is running, but that when power is removed, its brains get scrambled.)
Also confronted with temperamental rtc's here....
I don't think a really reliable test is possible, you could store the last recorded time somewhere in non volatile memory, and check that the clock hasn't moved to a past date, and you can also test that the delta between two checks is not too big. That would catch something like the year 3000, but you cannot reduce the tested time lapse to say 1 month - you want the thing to wake up even if it was shelved for say a year.
Do you have the possibility to consult a time source at startup? Eg an ntp server or some other device that your controller speaks with that can be considered synchronized with a reliable time source?
You can route the RTC clock to an external pin and watch it on an oscilloscope or a logic analyzer.
IIRC I did just that for LPC1766/1768 (I have two identical boards populated with different processors).
Related
I am working on the ESP32 microcontroller and I would like to implement iBeacon advertising feature. I have been reading about the iBeacon. I have learnt about the specific format that iBeacon packet uses:
https://os.mbed.com/blog/entry/BLE-Beacons-URIBeacon-AltBeacons-iBeacon/
From what I understand, iBeacon preset is set and not meant to be modified. I must set a custom UUID, major and minor numbers such as:
uint8_t Beacon_UUID[16] = {0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77,0x88,0x99,0xAA,0xBB,0xCC,0xDD,0xEE,0xFF};
uint8_t Beacon_MAJOR[2] = {0x12,0x34};
uint8_t Beacon_MINOR[2] = {0x56,0x78};
The only thing that I am confused about is the TX Power byte. What should I set it to?
According to the website that I have referred above:
Blockquote
A scanning application reads the UUID, major number and minor number and references them against a database to get information about the beacon; the beacon itself carries no descriptive information - it requires this external database to be useful. The TX power field is used with the measured signal strength to determine how far away the beacon is from the smart phone. Please note that TxPower must be calibrated on a beacon-by-beacon basis by the user to be accurate.
Blockquote
It mentions what is TxPower and how it should be determined but I still cannot make any sense out of it. Why would I need to measure how far away the beacon is from the smart phone if? That should be done by the iBeacon scanner not the advertiser(me).
When you are making a hardware device transmit iBeacon, it is your responsibility to measure the output power of the transmitter and put the corresponding value into the TxPower byte of the iBeacon transmission.
Why? Because receiving applications that detect your beacon need to know how strong your transmitter is to estimate distance. Otherwise there would be no way for the receiving application to tell if a medium signal level like -75 dB is from a nearby weak transmitter or a far away strong transmitter.
The basic procedure is to put a receiver exactly one meter away from your transmitter and measure the RSSI at that distance. The one meter RSSI is what you put into TxPower byte of the iBeacon advertisement.
The specifics of how to measure this properly can be a bit tricky, because every receiver has a different "specificity" meaning they will read a higher or lower RSSI depending on their antenna gain. When Apple came out with iBeacon several years ago, they declared the reference receiver an iPhone 4S -- this was the newest phone available at that time. You would run beacon detection app like AirLocate (not available in the App Store) or my Beacon Locate (available in the App Store). The basic procedure is to aim the back of the phone at the beacon when it is exactly one meter away and use the app to measure the RSSI. Many detector apps have a "calibrate" feature which averages RSSI measurements over 30 seconds or so. For best results when calibrating, do this with both transmitter and receiver at least 3 feet above the ground and minimize metal or dense walls nearby. Ideally, you would do this outdoors using two plastic tripods (or do the same inside an antenna chamber.)
It is hard to find a reference iPhone 4S these days, and other iPhone models can measure surprisingly different RSSI values. My tests show that an iPhone SE 2nd edition measures signals very similarly to an iPhone 4S. But even these models are not made anymore. If you cannot get one of these, use the oldest iPhone you can get without a case and take the best measurement you can as described above. Obviously a ideal measurement requires more effort -- you have to decide how much effort you are willing to put into this. An ideal measurement is only important if you expect receiving applications to want to get the best distance measurements possible.
Problem Definition & Establishing Concepts
Let’s say we have a TumblingEventTimeWindow with size 5 minutes. And we have events containing 2 basic pieces of information:
number
event timestamp
In this example, we kick off our Flink topology at 12:00 PM worker machines’ wall clock time (of course workers can have out of sync clocks but that’s out of the scope of this question). This topology contains one processing operator whose responsibility is to sum up the values of events belonging to each window and a KAFKA Sink which is irrelevant with regard to this question.
This window has a BoundedOutOfOrdernessTimestampExtractor with allowed latency of one minute.
Watermark: To my understanding, watermark in Flink and Spark Structured Stream is defined as (max-event-timestamp-seen-so-far - allowed-lateness). Any event whose event timestamp is less than or equal to this watermark will be discarded and ignored in result computations.
Part 1 (Determining Boundaries Of The Window)
Happy (Real-Time) Path
In this scenario several events arrive at the Flink Operator with different event timestamps spanning 12:01 - 12:09. Also, the event timestamps are relatively aligned with our processing time (shown in the X axis below). Since we're dealing with EVENT_TIME characteristic, whether or not an even belongs to a particular event should be determined via its event timestamp.
Old Data Rushing In
In that flow I have assumed the boundaries of the two tumbling windows are 12:00 -- 12:05 and 12:05 -- 12:10 just because we have kicked off the execution of the topology at 12:00. If that assumption is correct (I hope not), then what happens in case of a back-filling situation in which several old events coming in with much older event timestamps and we have kicked off the topology at 12:00 again? (old enough that our lateness allowance does not cover them). Something like the following:
If it goes like that, then our events won't be captured in any window of course, so again, I'm hoping that's not the behavior :)
The other option would be to determine windows' boundaries via the event timestamp of the arriving events. If that's the case, how would that work? The smallest event timestamp noticed becomes the beginning of the first window and from there based on the size (in this case 5 minutes), the consequent boundaries are determined? Cause that approach will have flaws and loopholes too. Can you please explain how does this work and how the boundaries of windows are determined?
Backfilling Events Rushing In
The answer to the previous question will address this as well, but I think it would be helpful to explicitly mention it here. Let's say I have this TumblingEventTimeWindow of size 5 minutes. Then at 12:00 I kick off a backfilling job which rushes in many events to the Flink operator whose timestamps cover the range 10:02 - 10:59; but since this is a backfilling job, the whole execution takes about 3 minutes to finish.
Will the job allocate 12 separate windows and populate them correctly based on the events' event timestamps? What would be the boundaries of those 12 windows? And will I end up with 12 output events each of which having the summed up value of each allocated window?
Part 2 (Unit/Integration Testing Of Such Stateful Operators)
I also have some concerns regarding automated testing of such logic and operators. Best way to manipulate processing time, trigger certain behaviors in such a way that shape desired windows' boundaries for testing purposes. Specially since the stuff that I've read so far on leveraging Test Harnesses seem a bit confusing and can cause some cluttered code potentially which is not that easy to read:
Unit Test Stateful Operators
Lateness Testing of Window in Flink
References
Most of what I've learned in this area and the source of some of my confusion can be found in the following places:
Timestmap Extractors & Watermark Emitters
Event Time Processing & Watermarking
Handling Late Data & Watermarking in Spark
The images in that section of Spark doc were super helpful and educative. But at the same time the way windows' boundaries are aligned with those processing times and not event timestamps, caused some confusion for me.
Also, in that visualization, it seems like the watermark is computed once every 5 minutes since that's the sliding specification of the window. Is that the determining factor for how often the watermark should be computed? How does this work in Flink with regard to different windows (e.g. Tumbling, Sliding, Session and more)?!
HUGE thanks in advance for your help and if you know about any better references with regard to these concepts and their internals working, please let me know.
UPDATES AFTER #snntrable Answer Below
If you run a Job with event time semantics, the processing time at the window operators is completely irrelevant
That is correct and I understand that part. Once you're dealing with EVENT_TIME characteristics, you're pretty much divorced from processing time in your semantics/logic. The reason I brought up the processing time was my confusion with regard to the following key question which still is a mystery to me:
How does the windows' boundaries are computed?!
Also, thanks a lot for clarifying the distinction between out-of-orderness and lateness. The code I was dealing with totally threw me off by having a misnomer (the constructor argument to a class inheriting from BoundedOutOfOrdernessTimestampExtractor was named maxLatency) :/
With that in mind, let me see if I can get this correct with regard to how watermark is computed and when an event will be discarded (or side-outputted):
Out of Orderness Assigner
current-watermark = max-event-time-seen-so-far - max-out-of-orderness-allowed
Allowed Lateness
current-watermark = max-event-time-seen-so-far - allowed-lateness
Regular Flow
current-watermark = max-event-time-seen-so-far
And in any of these cases, whatever event whose event timestamp is less than or equal to the current-watermark, will be discarded (side-outputted), correct?!
And this brings up a new question. When would you wanna use out of orderness as opposed to lateness? Since the current watermark computation (mathematically) can be identical in these cases. And what happens when you use both (does that even make sense)?!
Back To Windows' Boundaries
This is still the main mystery to me. Given all the discussion above, let'e revisit the concrete example I provided and see how the windows' boundaries are determined here. Let's say we have the following scenario (events are in the shape of (value, timestamp)):
Operator kicked off at 12:00 PM (that's the processing time)
Events arriving at the operator in the following order
(1, 8:29)
(5, 8:26)
(3, 9:48)
(7, 9:46)
We have a TumblingEventTimeWindow with size 5 minutes
The window is applied to a DataStream with BoundedOutOfOrdernessTimestampExtractor which has 2 minute maxOutOfOrderness
Also, the window is configured with allowedLateness of 1 minute
NOTE: If you cannot have both out of orderness and lateness or does not make sense, please only consider the out of orderness in the example above.
Finally, can you please layout the windows which will have some events allocated to them and, please specify the boundaries of those windows (beginning and end timestamps of the window). I'm assuming the boundaries are determined by events' timestamps as well but it's a bit tricky to figure them out in concrete examples like this one.
Again, HUGE thanks in advance and truly appreciate your help :)
Original Answer
Watermark: To my understanding, watermark in Flink and Spark Structured Stream is defined as (max-event-timestamp-seen-so-far - allowed-lateness). Any event whose event timestamp is less than or equal to this watermark will be discarded and ignored in result computations.
This is not correct and might be the source of the confusion. Out-of-Orderness and Lateness are different concepts in Flink. With the BoundedOutOfOrdernessTimestampExtractor the watermark is max-event-timestamp-seen-so-far - max-out-of-orderness. More about Allowed Lateness in the Flink Documentation [1].
If you run a Job with event time semantics, the processing time at the window operators is completely irrelevant:
events will be assigned to their windows based on their event time timestamp
time windows will be triggered once the watermarks reaches their maximum timestamp (window end time -1).
events with a timestamp older than current watermark - allowed lateness are discarded or sent to the late data side output [1]
This means, if you start a job at 12:00pm (processing time) and start ingesting data from the past, the watermark will also be (even further) in the past. So, the configured allowedLateness is irrelevant, because the data is not late with respect to even time.
On the other hand, if you first ingest some data from 12:00pm and afterwards data from 10:00pm, the watermark will have already advanced to ~12:00pm before you ingest the old data. In this case the data from 10:00pm will be "late". If it is later than the configured allowedLateness (default=0) it is discarded (default) or sent to a side output (if configured) [1].
Follow Up Answers
The timeline for an event time window is the following:
first element with timestamp within the a window arrives -> state for this window (& key) is created
watermark >= window_endtime - 1 arrives -> window is fired (results are emitted), but state is not discarded
watermark >= window_endtime + allowed_latenes arrives -> state is discarded
Between 2. and 3. events for this window are late, but within the allowed lateness. The events are added to the existing state and - per default - the window is fired on each record emitting a refined result.
After 3. events for this window will be discarded (or sent to the late output sink).
So, yes, it makes sense to configure both. The out of orderness determines, when the window is fired for the first time, while the allowed lateness determines how long the state is kept around to potentially update the results.
Regarding the boundaries: tumbling event time windows have a fixed length, are aligned across keys and start at the unix epoch. Empty windows, don't exist. For your example this means:
(1, 8:29) is added to window (8:25 - 8:29:59:999)
(5, 8:26) is added to window (8:25 - 8:29:59:999)
(3, 9:48) is added to window (9:45 - 9:49:59:999)
(8:25 - 8:29:59:999) is fired because the watermark has advanced to 9:48-0:02=9:46, which is larger than the last timestamp of the window. The window state is also discarded, because the watermark has advanced to 9:46, which is also larger than the end time of the window + the allowed lateness (1 minute)
(7, 9:46) is added to window is added to window (9:45 - 9:49:59:999)
Hope this helps.
Konstantin
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/windows.html#allowed-lateness
I am trying to shape a JMeter test involving a Concurrency Thread Group and a Throughput Shaping Timer as documented here and here. the timer is configured to run ten ramps and stages with RPS from 1 to 333.
I want to set up the Concurrency Thread Group to use the schedule feedback function and added the formula in the Target concurrency field (I have updated the example from tst-name to the actual timer name). ramp-up time and steps I have set to 1 as I assume the properties are not that important if the throughput is managed by the timer; the Hold Target Rate time is 8000, which is longer than the steps added in the timer (6200).
When I run the test, it ends without any exceptions within 3 seconds or so. The log file shows a few rows about starting and ending threads but nothing alarming.
The only thing I find suspicious is the Log entry "VirtualUserController: Test limit reached, thread is done plus thread name.
I am not getting enough clues from the documentation linked here to figure this out myself, do you have any hints?
According to the documentation rampup time and steps should be blank:
When using this approach, leave Concurrency Thread Group Ramp Up Time and Ramp-Up Steps Count fields blank"
So your assumption that setting them to 1 is OK, seems false...
I need to create a board that will contain about 50 LED’s. The LED’s need to be turned on/off individually, and together (timer is based on days suppose every Monday all the 50 LED’s turn on and every day of the week 10 LED’s are turned on).
In my research I have found the LM555 timer but that would lead to a huge circuit of 50 different timers
What is the most efficient way of controlling these LED’s
My first answer was really stupid (i'm so sorry about that) and i don't know why I didn't think before about the amazing "Shift Registers".
Your Arduino don't have so much pin to light every LEDs you had. But using shift registers this is possible. From an 8-bit shift register you can light 8 LEDs using just 2-3 PIN on your Arduino board (1 for clock, 1 for data and 1 for latch).
So to light 50 PINs you just need 7 of this components (for example).
How to use it? There are a lot of tutorials on internet, usually the sparkFun tutorial is my favorite.
Below: an image from Arduino site. It explained how to connect 16 LEDs.
Anyway from the software side, what you have to know ShiftOut function.
After that you have to use time function.
First of all initialize your time variables as you can see in link I posted above.
After, in your loop:
Put HIGH your Datapin connected to 50 leds when previousMonth != month.
Put HIGH dataPin connected to 10 leds when daySum == 10 (so, when previousDay != day you have to increase daySum using daySum++).
I have a setup with a Beaglebone Black which communicates over I²C with his slaves every second and reads data from them. Sometimes the I²C readout fails though, and I want to get statistics about these fails.
I would like to implement an algorithm which displays the percentage of successful communications of the last 5 minutes (up to 24 hours) and updates that value constantly. If I would implement that 'normally' with an array where I store success/no success of every second, that would mean a lot of wasted RAM/CPU load for a minor feature (especially if I would like to see the statistics of the last 24 hours).
Does someone know a good way to do that, or can anyone point me in the right direction?
Why don't you just implement a low-pass filter? For every successfull transfer, you push in a 1, for every failed one a 0; the result is a number between 0 and 1. Assuming that your transfers happen periodically, this works well -- and you just have to adjust the cutoff frequency of that filter to your desired "averaging duration".
However, I can't follow your RAM argument: assuming you store one byte representing success or failure per transfer, which you say happens every second, you end up with 86400B per day -- 85KB/day is really negligible.
EDIT Cutoff frequency is something from signal theory and describes the highest or lowest frequency that passes a low or high pass filter.
Implementing a low-pass filter is trivial; something like (pseudocode):
new_val = 1 //init with no failed transfers
alpha = 0.001
while(true):
old_val=new_val
success=do_transfer_and_return_1_on_success_or_0_on_failure()
new_val = alpha * success + (1-alpha) * old_val
That's a single-tap IIR (infinite impulse response) filter; single tap because there's only one alpha and thus, only one number that is stored as state.
EDIT2: the value of alpha defines the behaviour of this filter.
EDIT3: you can use a filter design tool to give you the right alpha; just set your low pass filter's cutoff frequency to something like 0.5/integrationLengthInSamples, select an order of 0 for the IIR and use an elliptic design method (most tools default to butterworth, but 0 order butterworths don't do a thing).
I'd use scipy and convert the resulting (b,a) tuple (a will be 1, here) to the correct form for this feedback form.
UPDATE In light of the comment by the OP 'determine a trend of which devices are failing' I would recommend the geometric average that Marcus Müller ꕺꕺ put forward.
ACCURATE METHOD
The method below is aimed at obtaining 'well defined' statistics for performance over time that are also useful for 'after the fact' analysis.
Notice that geometric average has a 'look back' over recent messages rather than fixed time period.
Maintain a rolling array of 24*60/5 = 288 'prior success rates' (SR[i] with i=-1, -2,...,-288) each representing a 5 minute interval in the preceding 24 hours.
That will consume about 2.5K if the elements are 64-bit doubles.
To 'effect' constant updating use an Estimated 'Current' Success Rate as follows:
ECSR = (t*S/M+(300-t)*SR[-1])/300
Where S and M are the count of errors and messages in the current (partially complete period. SR[-1] is the previous (now complete) bucket.
t is the number of seconds expired of the current bucket.
NB: When you start up you need to use 300*S/M/t.
In essence the approximation assumes the error rate was steady over the preceding 5 - 10 minutes.
To 'effect' a 24 hour look back you can either 'shuffle' the data down (by copy or memcpy()) at the end of each 5 minute interval or implement a 'circular array by keeping track of the current bucket index'.
NB: For many management/diagnostic purposes intervals of 15 minutes are often entirely adequate. You might want to make the 'grain' configurable.