LLDB Automatic Stepping Through Program With Python Returning Steps/Lines That Don't Appear Manually - lldb

Following LLDB - automatically step, and print list of lines, until a breakpoint, I was able to successfully write a csv file of the line numbers and file names of each step when stepping through my program. However, I find my output very strange. My CSV file contains lines that dun appear when I step through manually. For example:
431,/Users/user/Documents/zDEVELOP/bash-3.2.57/lib/readline/input.c
435,/Users/user/Documents/zDEVELOP/bash-3.2.57/lib/readline/input.c
132,/Users/user/Documents/zDEVELOP/bash-3.2.57/lib/readline/input.c
132,/Users/user/Documents/zDEVELOP/bash-3.2.57/lib/readline/input.c
452,/Users/user/Documents/zDEVELOP/bash-3.2.57/lib/readline/input.c
462,/Users/user/Documents/zDEVELOP/bash-3.2.57/lib/readline/input.c
For eg., if you look at input.c:
int
rl_getc (stream)
FILE *stream;
{
int result;
unsigned char c;
while (1)
{
#if defined (__MINGW32__)
...
Line 462 is actually this line:
{
Which is not a line that is usually stopped at.
This is my Python code:
import lldb
import csv
lldb.debugger.SetAsync(False)
i = 1
while i:
with open('Z.csv', mode='a') as employee_file:
employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
l = lldb.thread.frames[0].addr.GetLineEntry().line
fn = str(lldb.thread.frames[0].addr.GetLineEntry().GetFileSpec())
employee_writer.writerow([l, fn])
lldb.thread.StepInto()
i=i+1
In lldb, I ran the script command, and then pasted the above code in the embedded Python interpreter.
This is how it would be like if I had manually step through the code:
Process 14194 launched: '/Users/user/Documents/zDEVELOP/bash-3.2.57/bash' (x86_64)
bash-3.2$ bash was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x000000010007c9a4 bash`rl_read_key at input.c:421 [opt]
418 {
419 int c;
420
-> 421 rl_key_sequence_length++;
422
423 if (rl_pending_input)
424 {
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007c9bd bash`rl_read_key at input.c:423 [opt]
420
421 rl_key_sequence_length++;
422
-> 423 if (rl_pending_input)
424 {
425 c = rl_pending_input;
426 rl_clear_pending_input ();
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007c9df bash`rl_read_key at input.c:431 [opt]
428 else
429 {
430 /* If input is coming from a macro, then use that. */
-> 431 if (c = _rl_next_macro_key ())
432 return (c);
433
434 /* If the user has an event function, then call it periodically. */
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007bf94 bash`_rl_next_macro_key at macro.c:105 [opt]
102 {
103 int c;
104
-> 105 if (rl_executing_macro == 0)
106 return (0);
107
108 if (rl_executing_macro[executing_macro_index] == 0)
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007c0a0 bash`_rl_next_macro_key at macro.c:122 [opt]
119 #else
120 return (rl_executing_macro[executing_macro_index++]);
121 #endif
-> 122 }
123
124 /* Save the currently executing macro on a stack of saved macros. */
125 void
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007c9e4 bash`rl_read_key at input.c:431 [opt]
428 else
429 {
430 /* If input is coming from a macro, then use that. */
-> 431 if (c = _rl_next_macro_key ())
432 return (c);
433
434 /* If the user has an event function, then call it periodically. */
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007ca01 bash`rl_read_key at input.c:435 [opt]
432 return (c);
433
434 /* If the user has an event function, then call it periodically. */
-> 435 if (rl_event_hook)
436 {
437 while (rl_event_hook && rl_get_char (&c) == 0)
438 {
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007cdd8 bash`rl_read_key at input.c:451 [opt]
448 }
449 else
450 {
-> 451 if (rl_get_char (&c) == 0)
452 c = (*rl_getc_function) (rl_instream);
453 }
454 }
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007cdd8 bash`rl_read_key [inlined] rl_get_char at input.c:132 [opt]
129 rl_get_char (key)
130 int *key;
131 {
-> 132 if (push_index == pop_index)
133 return (0);
134
135 *key = ibuffer[pop_index++];
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007cde7 bash`rl_read_key at input.c:452 [opt]
449 else
450 {
451 if (rl_get_char (&c) == 0)
-> 452 c = (*rl_getc_function) (rl_instream);
453 }
454 }
455
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007c4a0 bash`rl_getc(stream=0x00007fff8abf8110) at input.c:462 [opt]
459 int
460 rl_getc (stream)
461 FILE *stream;
-> 462 {
463 int result;
464 unsigned char c;
465
Target 0: (bash) stopped.
(lldb) s
Process 14194 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = step in
frame #0: 0x000000010007c4ad bash`rl_getc(stream=0x00007fff8abf8110) at input.c:472 [opt]
469 if (isatty (fileno (stream)))
470 return (getch ());
471 #endif
-> 472 result = read (fileno (stream), &c, sizeof (unsigned char));
473
474 if (result == sizeof (unsigned char))
475 return (c);
Target 0: (bash) stopped.
What could be the reason why this happens? And is there a way to solve this problem?
EDIT:
After trying the following new code, I still have the same problematic output:
import lldb
import csv
lldb.debugger.SetAsync(False)
i = 1
while i:
with open('Z.csv', mode='a') as employee_file:
employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
l = lldb.thread.frames[0].line_entry.line
fn = str(lldb.thread.frames[0].line_entry.GetFileSpec())
employee_writer.writerow([l, fn])
lldb.thread.StepInto()
i=I+1
EDIT 2:
This is how I obtained the executable that I ran lldb on:
First, downloaded this:
http://ftp.gnu.org/gnu/bash/bash-3.2.57.tar.gz
Then followed the instructions on:
https://www.gnu.org/software/bash/manual/bash.html#Basic-Installation
That's how I got the file:
/Users/user/Documents/zDEVELOP/bash-3.2.57/bas
Which I executed lldb on.
These is the sequence of lldb commands I used from start to finish:
user$ lldb
(lldb) file /Users/user/Documents/zDEVELOP/bash-3.2.57/bash
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Users/user/miniconda3/lib/python2.7/copy.py", line 52, in <module>
import weakref
File "/Users/user/miniconda3/lib/python2.7/weakref.py", line 14, in <module>
from _weakref import (
ImportError: cannot import name _remove_dead_weakref
Current executable set to '/Users/user/Documents/zDEVELOP/bash-3.2.57/bash' (x86_64).
(lldb) breakpoint set -n main
Breakpoint 1: where = bash`main + 30 at shell.c:354, address = 0x000000010000152e
(lldb) run
Process 3118 launched: '/Users/user/Documents/zDEVELOP/bash-3.2.57/bash' (x86_64)
bash was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 3118 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x000000010000152e bash`main(argc=1, argv=0x00007ffeefbffa70, env=0x00007ffeefbffa80) at shell.c:354 [opt]
351 #endif
352
353 /* Catch early SIGINTs. */
-> 354 code = setjmp (top_level);
355 if (code)
356 exit (2);
357
Target 0: (bash) stopped.
(lldb) script
Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.
And then I executed the following code in the Python interpreter:
import lldb
import csv
lldb.debugger.SetAsync(False)
i = 1
while i:
with open('Z.csv', mode='a') as employee_file:
employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
l = lldb.thread.frames[0].line_entry.line
fn = str(lldb.thread.frames[0].line_entry.GetFileSpec())
employee_writer.writerow([l, fn])
lldb.thread.StepInto()
i=I+1

Related

lldb memory read error on Mac

I'm experimenting with lldb and I wrote a simple C application. I want to debug it in terminal using lldb. When I want to see the stack frame, i get a memory read error:
(lldb) target create "./auth_overflow"
Current executable set to './auth_overflow' (x86_64).
(lldb) br s -l 25
Breakpoint 1: where = auth_overflow`main + 69 at auth_overflow.c:25, address = 0x0000000100000e25
(lldb) br s -l 9
Breakpoint 2: where = auth_overflow`check_authentication + 47 at auth_overflow.c:9, address = 0x0000000100000d5f
(lldb) br s -l 16
Breakpoint 3: where = auth_overflow`check_authentication + 138 at auth_overflow.c:16, address = 0x0000000100000dba
(lldb) run AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Process 413 launched: './auth_overflow' (x86_64)
Process 413 stopped
* thread #1: tid = 0x33d2, 0x0000000100000e25 auth_overflow`main(argc=2, argv=0x00007fff5fbffcc0) + 69 at auth_overflow.c:25, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x0000000100000e25 auth_overflow`main(argc=2, argv=0x00007fff5fbffcc0) + 69 at auth_overflow.c:25
22 exit(0);
23 }
24
-> 25 if(check_authentication(argv[1])) {
26 printf("\n-=-=-=-=-=-=-=-=-=-=-=-=-=-\n");
27 printf(" Access Granted.\n");
28 printf("-=-=-=-=-=-=-=-=-=-=-=-=-=-\n");
(lldb) re r esp
esp = 0x5fbffc70
(lldb) x/16xw $esp
error: memory read failed for 0x5fbffc00
(lldb)
What do you suggest , I should do?
EDIT : Actually I don't want to debug the application, just to see how it works on lower level. Because of this I'd like to see the content of the current stack frame, something like this:
(lldb) x/16xw $esp
0xbffff7e0: 0xb8000ce0 0x00000002 0x00000000 0xb7fd6ff4
0xbffff7f0: 0x40f5f7f0 0x00000000 0x00000002 0x08048474
0xbffff800: 0x08048510 0xbffff874 0x00000001 0x00000001
0xbffff810: 0xbffff848 0x00000000 0xb8000ff4 0x08048371
(lldb)
This:
Current executable set to './auth_overflow' (x86_64).
shows you're on a 64 bit machine. That being the case, you want the 64 bit rsp register, not the 32 bit esp register. esp will give you the least significant 32 bits of the contents of rsp, which in this case is obviously not yielding a valid address for you.
x/16xw $rsp
is what you're looking for.
Sample LLDB session:
paul#horus:~/Documents/src/sandbox$ lldb ./testldb
(lldb) target create "./testldb"
Current executable set to './testldb' (x86_64).
(lldb) list testldb.c
1 #include <stdio.h>
2
3 void func(int i) {
4 printf("In func() with value %d\n", i);
5 }
6
7 int main(void) {
8 func(3);
9 return 0;
10 }
11
(lldb) b testldb.c:4
Breakpoint 1: where = testldb`func + 18 at testldb.c:4, address = 0x0000000100000f22
(lldb) run
Process 48270 launched: './testldb' (x86_64)
Process 48270 stopped
* thread #1: tid = 0xb8dbca, 0x0000000100000f22 testldb`func(i=3) + 18 at testldb.c:4, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
frame #0: 0x0000000100000f22 testldb`func(i=3) + 18 at testldb.c:4
1 #include <stdio.h>
2
3 void func(int i) {
-> 4 printf("In func() with value %d\n", i);
5 }
6
7 int main(void) {
(lldb) frame variable
(int) i = 3
(lldb) print &i
(int *) $0 = 0x00007fff5fbff9dc
(lldb) register read $rsp
rsp = 0x00007fff5fbff9d0
(lldb) x/16xw $rsp
0x7fff5fbff9d0: 0x00000000 0x00000000 0x00000000 0x00000003
0x7fff5fbff9e0: 0x5fbffa00 0x00007fff 0x00000f59 0x00000001
0x7fff5fbff9f0: 0x5fbffa18 0x00007fff 0x5fc0105e 0x00000000
0x7fff5fbffa00: 0x5fbffa18 0x00007fff 0x8fdc25fd 0x00007fff
(lldb)

Debugging malloc error in C

I am currently working on a huge program written by another version. Over the last days I implemented some new features, but today I realized that this does not work correctly with some old features.
I used the lldb debugger on the program and get the following error message:
MT(3934,0x7fff797e3310) malloc: *** error for object 0x307400650: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Process 3934 stopped
* thread #1: tid = 0x52b27, 0x00007fff92e24866 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
frame #0: 0x00007fff92e24866 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill + 10:
-> 0x7fff92e24866: jae 0x7fff92e24870 ; __pthread_kill + 20
0x7fff92e24868: movq %rax, %rdi
0x7fff92e2486b: jmpq 0x7fff92e21175 ; cerror_nocancel
0x7fff92e24870: ret
After moving up some levels I can see:
frame #6: 0x000000010000e602 MT`free_matrix(m=0x0000000307400520, nrl=1, nrh=7, ncl=1, nch=7) + 66 at nrutil.c:282
(lldb) up
frame #7: 0x000000010000b925 MT`equations + 1221 at equations.c:619
616
617 //Free memory for next round or final
618 free_vector(B,1,n);
-> 619 free_matrix(A,1,n,1,n);
620
If you want I could post the function for free_matrix(..) and the function which allocates the matrix but its basically malloc and free.
Now I definitely don't want anybody to give a solution. I am just asking how you would proceed in this case.
So my main question is: Is the error definitely tied to the matrix A or may there be other details I should search for?

How does ptrace work in Linux?

The ptrace system call allows the parent process to inspect the attached child. For example, in Linux, strace (which is implemented with the ptrace system call) can inspect the system calls invoked by the child process.
When the attached child process invokes a system call, the ptracing parent process can be notified. But how exactly does that happen? I want to know the technical details behind this mechanism.
Thank you in advance.
When the attached child process invokes a system call, the ptracing parent process can be notified. But how exactly does that happen?
Parent process calls ptrace with PTRACE_ATTACH, and his child calls ptrace with PTRACE_TRACEME option. This pair will connect two processes by filling some fields inside their task_struct (kernel/ptrace.c: sys_ptrace, child will have PT_PTRACED flag in ptrace field of struct task_struct, and pid of ptracer process as parent and in ptrace_entry list - __ptrace_link; parent will record child's pid in ptraced list).
Then strace will call ptrace with PTRACE_SYSCALL flag to register itself as syscall debugger, setting thread_flag TIF_SYSCALL_TRACE in child process's struct thread_info (by something like set_tsk_thread_flag(child, TIF_SYSCALL_TRACE);). arch/x86/include/asm/thread_info.h:
67 /*
68 * thread information flags
69 * - these are process state flags that various assembly files
70 * may need to access ...*/
75 #define TIF_SYSCALL_TRACE 0 /* syscall trace active */
99 #define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)
On every syscall entry or exit, architecture-specific syscall entry code will check this _TIF_SYSCALL_TRACE flag (directly in assembler implementation of syscall, for example x86 arch/x86/kernel/entry_32.S: jnz syscall_trace_entry in ENTRY(system_call) and similar code in syscall_exit_work), and if it is set, ptracer will be notified with signal (SIGTRAP) and child will be temporary stopped. This is done usually in syscall_trace_enter and syscall_trace_leave :
1457 long syscall_trace_enter(struct pt_regs *regs)
1483 if ((ret || test_thread_flag(TIF_SYSCALL_TRACE)) &&
1484 tracehook_report_syscall_entry(regs))
1485 ret = -1L;
1507 void syscall_trace_leave(struct pt_regs *regs)
1531 if (step || test_thread_flag(TIF_SYSCALL_TRACE))
1532 tracehook_report_syscall_exit(regs, step);
The tracehook_report_syscall_* are actual workers here, they will call ptrace_report_syscall. include/linux/tracehook.h:
80 /**
81 * tracehook_report_syscall_entry - task is about to attempt a system call
82 * #regs: user register state of current task
83 *
84 * This will be called if %TIF_SYSCALL_TRACE has been set, when the
85 * current task has just entered the kernel for a system call.
86 * Full user register state is available here. Changing the values
87 * in #regs can affect the system call number and arguments to be tried.
88 * It is safe to block here, preventing the system call from beginning.
89 *
90 * Returns zero normally, or nonzero if the calling arch code should abort
91 * the system call. That must prevent normal entry so no system call is
92 * made. If #task ever returns to user mode after this, its register state
93 * is unspecified, but should be something harmless like an %ENOSYS error
94 * return. It should preserve enough information so that syscall_rollback()
95 * can work (see asm-generic/syscall.h).
96 *
97 * Called without locks, just after entering kernel mode.
98 */
99 static inline __must_check int tracehook_report_syscall_entry(
100 struct pt_regs *regs)
101 {
102 return ptrace_report_syscall(regs);
103 }
104
105 /**
106 * tracehook_report_syscall_exit - task has just finished a system call
107 * #regs: user register state of current task
108 * #step: nonzero if simulating single-step or block-step
109 *
110 * This will be called if %TIF_SYSCALL_TRACE has been set, when the
111 * current task has just finished an attempted system call. Full
112 * user register state is available here. It is safe to block here,
113 * preventing signals from being processed.
114 *
115 * If #step is nonzero, this report is also in lieu of the normal
116 * trap that would follow the system call instruction because
117 * user_enable_block_step() or user_enable_single_step() was used.
118 * In this case, %TIF_SYSCALL_TRACE might not be set.
119 *
120 * Called without locks, just before checking for pending signals.
121 */
122 static inline void tracehook_report_syscall_exit(struct pt_regs *regs, int step)
123 {
...
130
131 ptrace_report_syscall(regs);
132 }
And ptrace_report_syscall generates SIGTRAP for debugger or strace via ptrace_notify/ptrace_do_notify:
55 /*
56 * ptrace report for syscall entry and exit looks identical.
57 */
58 static inline int ptrace_report_syscall(struct pt_regs *regs)
59 {
60 int ptrace = current->ptrace;
61
62 if (!(ptrace & PT_PTRACED))
63 return 0;
64
65 ptrace_notify(SIGTRAP | ((ptrace & PT_TRACESYSGOOD) ? 0x80 : 0));
66
67 /*
68 * this isn't the same as continuing with a signal, but it will do
69 * for normal use. strace only continues with a signal if the
70 * stopping signal is not SIGTRAP. -brl
71 */
72 if (current->exit_code) {
73 send_sig(current->exit_code, current, 1);
74 current->exit_code = 0;
75 }
76
77 return fatal_signal_pending(current);
78 }
ptrace_notify is implemented in kernel/signal.c, it stops the child and pass sig_info to ptracer:
1961 static void ptrace_do_notify(int signr, int exit_code, int why)
1962 {
1963 siginfo_t info;
1964
1965 memset(&info, 0, sizeof info);
1966 info.si_signo = signr;
1967 info.si_code = exit_code;
1968 info.si_pid = task_pid_vnr(current);
1969 info.si_uid = from_kuid_munged(current_user_ns(), current_uid());
1970
1971 /* Let the debugger run. */
1972 ptrace_stop(exit_code, why, 1, &info);
1973 }
1974
1975 void ptrace_notify(int exit_code)
1976 {
1977 BUG_ON((exit_code & (0x7f | ~0xffff)) != SIGTRAP);
1978 if (unlikely(current->task_works))
1979 task_work_run();
1980
1981 spin_lock_irq(&current->sighand->siglock);
1982 ptrace_do_notify(SIGTRAP, exit_code, CLD_TRAPPED);
1983 spin_unlock_irq(&current->sighand->siglock);
1984 }
ptrace_stop is in the same signal.c file, line 1839 for 3.13.

In the Linux kernel, what does it mean for a process to be "whole"?

This code is from the file /fs/proc/array.c in the Linux headers. What does the int whole parameter mean? I want to know why sometimes you need to accumulate min_flt and maj_flts from the sig_struct and other times it's ok to just read their values straight out of the task_struct
346 static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,
347 struct pid *pid, struct task_struct *task, int whole)
...
406 /* add up live thread stats at the group level */
407 if (whole) {
408 struct task_struct *t = task;
409 do {
410 min_flt += t->min_flt;
411 maj_flt += t->maj_flt;
412 gtime = cputime_add(gtime, t->gtime);
413 t = next_thread(t);
414 } while (t != task);
415
416 min_flt += sig->min_flt;
417 maj_flt += sig->maj_flt;
418 thread_group_times(task, &utime, &stime);
419 gtime = cputime_add(gtime, sig->gtime);
420 }
...
431 if (!whole) {
432 min_flt = task->min_flt;
433 maj_flt = task->maj_flt;
434 task_times(task, &utime, &stime);
435 gtime = task->gtime;
436 }
"whole" is just an argument name and in this context it seems to indicate "do_task_stat for the whole thread group".
do_task_stat is a static function only used within /fs/proc/array.c It is used in two places: proc_tid_stat (TID is "thread ID") and proc_tgid_stat (TGID is "thread group ID").
See Linux - Threads and Process for good explaination of thread groups.

significance of read_un/lock(&tasklist_lock)

While following path for wait system call, noticed that before calling
do_wait_thread we get hold of tasklist_lock. I am trying to understand the significance
of tasklist_lock & where its appropriate to use.
716 read_lock(&tasklist_lock);
1717 tsk = current;
1718 do {
1719 retval = do_wait_thread(wo, tsk);
1720 if (retval)
1721 goto end;
1722
1723 retval = ptrace_do_wait(wo, tsk);
1724 if (retval)
1725 goto end;
1726
1727 if (wo->wo_flags & __WNOTHREAD)
1728 break;
1729 } while_each_thread(current, tsk);
1730 read_unlock(&tasklist_lock);
I looked at the declaration of tasklist_lock, It is as follows.
/*
251 * This serializes "schedule()" and also protects
252 * the run-queue from deletions/modifications (but
253 * _adding_ to the beginning of the run-queue has
254 * a separate lock).
255 */
256 extern rwlock_t tasklist_lock;
257 extern spinlock_t mmlist_lock;
I am not able to understand where we should use this. Can you please let me know about it.
Appreciate your help.
The loop iterates over each thread of the current task. Holding the tasklist lock ensures that none of those threads disappear while the loop is running.

Resources