I am trying to write a Qemu TCG plugin to recognize if an instruction is executed in user mode or in kernel mode for a basic program.
I understand that a full system emulation is needed for the same. I wrote a TCG plugin, registering two Qemu callbacks qemu_plugin_register_vcpu_syscall_cb and qemu_plugin_register_vcpu_syscall_ret_cb. I am using the target arm-softmmu for full system emulation (and then the corresponding qemu-system-arm). I can see that there are svc calls in the user mode but I do not get any kernel-mode instructions. Any idea on what might be happening?
I have a problem with stability running TI AM5728 based custom board, similar to the Beaglebone X15. RTOS SW is running on one Cortex-A15 core MPU0 and sporadically (most often after several hours) freezes. When freezes it is impossible to connect to the MPU0 target by debugger, at the same time I can without any problems connect to the MPU1.
Debugger error:
CortexA15_0: Trouble Halting Target CPU: (Error -1323 # 0x1386AC)
Device failed to enter debug/halt mode because pipeline is stalled.
Power-cycle the board. If error persists, confirm configuration and/or
try more reliable JTAG settings (e.g. lower TCLK). (Emulation package
6.0.504.1)
For test purposes, I have started the simple program on the MPU1, and when MPU0 freezes MPU1 continue normal operation. WFE and WFI flag for MPU0 is inactive, moreover, I have made the additional test with trying to put MPU1 to the WFI/FORCED_OFF state. However, I still can connect with debugger and wakeup it from the FORCED_OFF state, as described in the technical manual.
I have dumped the registers by connection to the CS_DAP_DebugSS and have not found anything special. Register dump attached:
MPU_PRCM_PRM_C0_PM_CPU0_PWRSTCTRL
MPU_PRCM_DEVICE_PRM_RSTST
MPU_WUGEN_WKG_CONTROL_0
MPU_PRCM_CM_C0_CM_CPU0_CLKSTCTRL
What can be the potential stuck problem of just one core with failed attempts to connect with the debugger and second core running without problems?
Which hardware/software problem can potentially cause such behavior?
Thank you for any suggestions.
I just encounter the exact same problem.
Did you check your code at the address provided with the JTAG error (Error -1323 # 0x1386AC)? In my case it is an GPMC access, to a FPGA, which I can still access through CS_DAP_DebugSS.
I'm currently looking at errata i878, from revision L of the document. As it can take more than 48h to hang under stress test, I won't blindly apply the workaround. I'll modify my test, based on i878, trying to increase the failure rate, then I'll apply the workaround.
I am working on a project where I have a router with ARMv7 processor (Cortex A15) and OpenWRT OS. I have a shell on the router and can load kernel modules with insmod.
My goal is to write a kernel module in C which changes the HVBAR register and then executes the hvc instruction to get the processor in the hyp mode.
This is a scientific project where I want to check if I can place my own hypervisor on a running system. But before I start to write my own hypervisor I want to check if and how I can bring the processor in the hyp mode.
According to this picture take from armv7-a manual B.9.3.4 the system must be in insecure mode, not in user mode and the SCR.HCE bit must be set to 1.
My question is how I can prepare the processor with a C kernel module and inline assembly and then execute the hvc instruction. I want to do this with a kernel module because then I start in PL1. This pseudocode describes what I want to achieve:
call smc // to get in monitor mode
set SRC.HCE to 1 // to enable hvc instruction
set SRC.NS to 1 // to set the system to not secure
call hvc #0 // call the hvc instruction to produce a hypervisor exception
The easiest way to elevate privilege is to start off in the needed privilege mode already: You've a root shell. Is the boot chain verified? Could you replace bootloader or kernel, so your code naturally runs in PL2 (HYP) mode? If so, that's probably the easiest way to do it.
If you can't replace the relevant part of the boot chain, the details of writing the rootkit depend a lot on information about your system left out: In which mode is Linux started? Is KVM support enabled and active? Was PL2 initialized? Was it locked? Is there "secure" firmware you can exploit?
The objective is always the same: have HVBAR point at some code you can control and do a hvc. Depending on your environment, solutions may range from spraying as much RAM as possible with your code and hope (perhaps after some reboots) an uninitialized HVBAR would point at an instruction you control to inhibiting KVM from running and accessing the early hypervisor stub to install yourself instead.
Enumerating such exploits is a bit out of scope for a StackOverflow answer; this is rather dissertation material. Indeed, there's a doctoral thesis exactly on this topic:
Strengthening system security on the ARMv7 processor architecture with hypervisor-based security mechanisms
To start with - I don't have JTAG hardware debugger.
What I have:
Pandaboard and serial-USB cable to connect to console and my computer with Freebsd and GNU/Linux distribution.
What I'm looking for
- convinient way to trace/debug bootprocess inside FreeBSD kernel ( I'm mostly interested in this fragment: https://github.com/freebsd/freebsd/blob/master/sys/arm/arm/locore-v6.S and https://github.com/freebsd/freebsd/blob/master/sys/arm/arm/mp_machdep.c as I'm, going to modyfy those files ).
Based on my experience, there are few ways:
KDB / DDB: add call kdb_enter("A", "XYZ") to stop processing and enter interactive debug mode of DDB via serial.
printf-s in machine dependent (mach_dep) code
bootverbose, BUSDEBUG, VERBOSE_SYSINIT in machine independent code
Also it's worth to mention that DDB code contains functions to print registers, stack trace and etc.
For example, virt-what shows if you are running inside hardware virtualization "sandbox".
How to detect if you are running in ARM "TrustZone" sandbox?
TrustZone maybe different than what you think. There is a continuum of modes. From 'a simple API of trusted functions' to 'dual OSs' running in each world.
If there was more context given to the question, it would be helpful. Is this for programatically determining or for reverse engineering considerations? For the current Linux user-space, the answer is no.
Summary
No current user space utility.
Time based analysis.
Code based analysis.
CPU exclusion and SCR.
ID_PRF1 bits [7:4].
virt-what is not a fool-proof way of discovering if you are running under a hyper-visor. It is a program written for linux user-space. Mostly, these are shell scripts which examine /proc/cpuinfo, etc. procfs is a pseudo-file system which runs code in the kernel context and reports to user space. There is no such detection of TrustZone in the main line ARM linux. By design, ARM has made it difficult to detect. An design intent is to have code in the normal world run unmodified.
Code analysis
In order to talk to the secure world, the normal world needs SMC instructions. If your user space has access to kernel code or the vmlinux image, you can try to analyze the code sections for an SMC instruction. However, this code maybe present in the image, but never active. At least this says whether the Linux kernel has some support for TrustZone. You could write a kernel module which would trap any execution of an SMC instruction, but there are probably better solutions.
Timing analysis
If an OS is running in the secure world, some time analysis would show that some CPU cycles have been stolen if frequency scaling is not active. I think this is not an answer in the spirit of the original question. This relies on knowing that the secure world is a full-blown OS with a timer (or at least pre-emptible interrupts).
CPU exclusion and SCR
The SCR (Secure configuration register) is not available in the normal world. From the ARM Cortex-A5 MPcore manual (pg4-46),
Usage constraints The SCR is:
• only accessible in privileged modes
• only accessible in Secure state.
An attempt to access the SCR from any state other than secure privileged
results in an Undefined instruction exception.
ID_PRF1 bits [7:4].
On some Cortex-A series, the instruction,
mrc p15, 0, r0, c0, c1, 1
will get a value where bits [7:4] indicate whether the CPU supports Security Extensions, also known as TrustZone. A non-zero value indicates it is supported. Many early CPUs may not support this CP15 register . So, it is much like the SCR and handling the undefined instruction. Also, it doesn't tell you that code is active in the TrustZone mode.
Summary
It is possible that you could write a kernel module which would try this instruction and handle the undefined exception. This would detect a normal versus secure world. However, you would have to exclude CPUs which don't have TrustZone at all.
If the device is not an ARMv6 or better, then TrustZone is impossible. A great deal of Cortex-A devices have TrustZone in the CPU, but it is not active.
The combined SMC test and a CPU id, is still not sufficient. Some boot loaders run in the secure world and then transition to the normal world. So secure is only active during boot.
Theoretically, it is possible to know, especially with more knowledge of the system. There maybe many signs, such as spurious interrupts from the GIC, etc. However, I don't believe that any user space linux tool exists as of Jan 2014. This is a typical war of escalation between virus/rootkit writers and malware detection software.TZ Rootkits
You have not specified any details of the processor (A8, A9, A15?) or the execution mode (user/kernel/monitor) from where you want to detect the processor state.
As per the ARM documentation, the current state of the processor as Secure (aka the TrustZone sandbox) or Non-secure can be detected by reading the Secure Configuration Register and checking for the NS bit.
To access the Secure Configuration Register: MRC p15, 0, <Rd>, c1, c1, 0
Bit 0 being set corresponds to the processor being in non-secure mode and vice-versa.
You can check the processor's datasheet, and find those registers which behaves different between Normal world and Secure world. Generally, in Secure World, when you read these registers you will just get null. But get data in Normal world. And also, some registers that you can just access in Secure world, if you are in Secure World, you can access it, but in Normal World your access will be rejected.
Any way, there are many ways to distinguish Normal World and Secure World. JUST READ THE DATASHEET IN DETAIL.