I want to run arm's linux system in gem5's fs mode,I download related files from this address:
http://www.gem5.org/documentation/general_docs/fullsystem/guest_binaries
I was able to configure the correct file path, but finally got this output in the terminal2:
[ 0.661620] No filesystem could mount root, tried:
[ 0.661621] ext3
[ 0.661650] ext4`enter code here`
[ 0.661663] ext2
[ 0.661676] vfat
[ 0.661690] fuseblk
[ 0.661703]
[ 0.661728] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0)
And got this terrible output in the terminal1:
warn: Tried to read RealView I/O at offset 0x60 that doesn't exist
warn: Tried to write RVIO at offset 0xa8 (data 0) that doesn't exist
warn: Kernel panic in simulated kernel
I can provide my command line input, but simply adjusting the configuration inside will only lead to the same result:
./build/ARM/gem5.opt configs/example/arm/starter_fs.py --cpu="minor" --num-cores=4 --disk-image=/home/ad/GEM5/ARM_GEM5/gem5/my_image/disks/aarch64-ubuntu-trusty-headless.img --dtb=/home/ad/GEM5/ARM_GEM5/gem5/my_image/binaries/armv7_gem5_v1_4cpu.dtb --kernel=/home/ad/GEM5/ARM_GEM5/gem5/my_image/binaries/vmlinux
How can I solve this?
Regards,
Here is a full diagnostic procedure for this kind of problem: https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477
In summary, you have to ensure that:
the kernel has the config to read the disk type, for emulation usually:
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BLK=y
This seems to be the problem since there was no list of partitions given above? Please confirm. If not, the kernel can't read bytes from the disk it seems.
the kernel has the config to read the filesystem type. You kernel mentions ext2,3,4 though, so likely that's not the problem.
you are pointing the root= kernel CLI to the right partition
See also: https://cirosantilli.com/linux-kernel-module-cheat/#not-syncing That also contains a Buildroot setup that just works.
I also highly recommend that you first get it working on QEMU which boots much faster.
I had a similar problem. Trying to run a full-system emulation of both Ubuntu and Linaro minimal (from the gem5 website) under a 64-bit kernel, with the original starter_fs.py script, gives me this kernel panic:
[ 0.224367] List of all partitions:
[ 0.224394] fe00 1048320 vda
[ 0.224397] driver: virtio_blk
[ 0.224440] fe01 1048288 vda1 00000000-01
[ 0.224441]
[ 0.224480] No filesystem could mount root, tried:
[ 0.224481] ext3
[ 0.224510] ext4
[ 0.224524] ext2
[ 0.224537] squashfs
[ 0.224551] vfat
[ 0.224566] fuseblk
[ 0.224579]
[ 0.224606] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0)
[ 0.224656] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0+ #1
[ 0.224692] Hardware name: V2P-CA15 (DT)
[ 0.224717] Call trace:
[ 0.224741] dump_backtrace+0x0/0x1c0
[ 0.224765] show_stack+0x14/0x20
[ 0.224790] dump_stack+0x8c/0xac
[ 0.224812] panic+0x130/0x288
[ 0.224836] mount_block_root+0x22c/0x294
[ 0.224861] mount_root+0x140/0x174
[ 0.224884] prepare_namespace+0x138/0x180
[ 0.224910] kernel_init_freeable+0x1c0/0x1e0
[ 0.224939] kernel_init+0x10/0x108
[ 0.224961] ret_from_fork+0x10/0x18
[ 0.224987] Kernel Offset: disabled
[ 0.225009] CPU features: 0x21c06492
[ 0.225032] Memory Limit: 2048 MB
[ 0.225056] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0) ]---
Weird thing is that, few weeks ago, it worked like a charm. The problem lying into the specification of the root partition, on the kernel command line. In the starter_fs.py, change this line:
"root=/dev/vda",
By this:
"root=/dev/vda1",
You can see that before, the VirtIO block device was specified. The kernel wants a partition, not a block device. Then, you can run gem5:
build/ARM/gem5.opt -configs/example/arm/starter_fs.py --cpu="hpi" --num-cores=1 --disk-image="linaro-minimal-aarch64.img" --kernel="vmlinux.arm64"
And for me, the kernel panic is gone and I am able to boot my system again:
[ 0.228847] EXT4-fs (vda1): mounted filesystem without journal. Opts: (null)
[ 0.228906] VFS: Mounted root (ext4 filesystem) on device 254:1.
[ 0.229539] devtmpfs: mounted
[ 0.229792] Freeing unused kernel memory: 448K
INIT: version 2.88 booting
[ 0.234168] random: fast init done
Starting udev
[ 0.277039] udevd[715]: starting version 182
[ 0.411534] EXT4-fs (vda1): re-mounted. Opts: block_validity,delalloc,barrier,user_xattr
Starting Bootlog daemon: bootlogd.
[ 0.426573] random: dd: uninitialized urandom read (512 bytes read)
Populating dev cache
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
hwclock: can't open '/dev/misc/rtc': No such file or directory
Mon Jan 27 08:00:00 UTC 2014
hwclock: can't open '/dev/misc/rtc': No such file or directory
INIT: Entering runlevel: 5
Configuring network interfaces... ifconfig: SIOCGIFFLAGS: No such device
Starting rpcbind daemon...rpcbind: cannot create socket for udp6
rpcbind: cannot create socket for tcp6
done.
rpcbind: cannot get uid of '': Success
creating NFS state directory: done
starting statd: done
Starting auto-serial-console: done
Stopping Bootlog daemon:
bootlogd.
Last login: Mon Jan 27 08:00:00 UTC 2014 on tty1
INIT: no more processes left in this runlevel
root#genericarmv8:~# id
id
uid=0(root) gid=0(root) groups=0(root)
root#genericarmv8:~#
I'm trying to emulate the reMarkable tablet with Qemu in order to create a proper development environment for it, instead of cross-compiling and sending to the hardware device.
The firmware flasher repo contains the rootfs, kernel, DTB and u-boot files. I've created an .img file from the rootfs in order to boot it in Qemu with the following command:
qemu-system-arm \
-M sabrelite \
-bios "files/u-boot.imx" \
-kernel "zImage" \
-append "console=ttymxc0 rootfstype=ext4 root=/dev/mmcblk1p2 rw rootwait init=/bin/bash loglevel=8 bootmem-debug earlyprintk" \
-dtb "zero-gravitas.dtb" \
-drive file="floppy.img",format=raw,id=mmcblk1p2 \
-device sd-card,drive=mmcblk1p2
but the kernel does not seem to be starting as I have the same log whether the floppy.img file (drive+device) is given or not. The startup loops on this error:
[ 0.713093] 2020000.serial: ttymxc0 at MMIO 0x2020000 (irq = 19, base_baud = 5000000) is a IMX
[ 0.732268] console [ttymxc0] enabled
[ 0.736333] phy index low: 1, phy index high: 2
[ 240.289647] INFO: task swapper:1 blocked for more than 120 seconds.
[ 240.290160] Not tainted 4.1.28-zero-gravitas-01866-ge0b823726ea4-dirty #82
[ 240.290318] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 240.290662] swapper D 8051c44c 0 1 0 0x00000000
[ 240.292245] [<8051c44c>] (__schedule) from [<8051c73c>] (schedule+0x40/0x98)
[ 240.292473] [<8051c73c>] (schedule) from [<8051e7b8>] (schedule_timeout+0x114/0x168)
[ 240.292781] [<8051e7b8>] (schedule_timeout) from [<8051d248>] (wait_for_common+0x88/0x130)
[ 240.292953] [<8051d248>] (wait_for_common) from [<80262c74>] (imx_rng_init+0x158/0x2a8)
[ 240.293117] [<80262c74>] (imx_rng_init) from [<80262574>] (set_current_rng+0xc0/0x15c)
[ 240.293276] [<80262574>] (set_current_rng) from [<80262874>] (hwrng_register+0x190/0x1b8)
[ 240.293436] [<80262874>] (hwrng_register) from [<807c3fd8>] (imx_rng_probe+0xd4/0x134)
[ 240.293682] [<807c3fd8>] (imx_rng_probe) from [<802748e0>] (platform_drv_probe+0x44/0xac)
[ 240.293852] [<802748e0>] (platform_drv_probe) from [<802735ac>] (driver_probe_device+0x178/0x2b8)
[ 240.294009] [<802735ac>] (driver_probe_device) from [<802737bc>] (__driver_attach+0x8c/0x90)
[ 240.294158] [<802737bc>] (__driver_attach) from [<80271d50>] (bus_for_each_dev+0x68/0x9c)
[ 240.294352] [<80271d50>] (bus_for_each_dev) from [<802726bc>] (bus_add_driver+0x13c/0x1e4)
[ 240.294600] [<802726bc>] (bus_add_driver) from [<80273ed4>] (driver_register+0x78/0xf8)
[ 240.294843] [<80273ed4>] (driver_register) from [<807c434c>] (__platform_driver_probe+0x20/0x70)
[ 240.295092] [<807c434c>] (__platform_driver_probe) from [<807a9d78>] (do_one_initcall+0x118/0x1c4)
[ 240.295367] [<807a9d78>] (do_one_initcall) from [<807a9f48>] (kernel_init_freeable+0x124/0x1c4)
[ 240.295609] [<807a9f48>] (kernel_init_freeable) from [<8051883c>] (kernel_init+0x8/0xe8)
[ 240.295844] [<8051883c>] (kernel_init) from [<8000ef88>] (ret_from_fork+0x14/0x2c)
full log here
I will update this question when I have new findings, but i'm new to Qemu and I'm quite stuck and ran out of options. The repository i'm working in is here. Thanks for any input !
I haven't investigated too closely, but the fact that the backtrace shows a hang in the imx_rng_init function suggests that the problem is that QEMU doesn't have an emulation of the imx SoC's builtin RNG device, and so the guest is hanging forever waiting for a response from hardware that doesn't exist.
You'll need to either implement a model of that device, or else use a guest kernel which doesn't try to probe for that device.
More generally, running an Arm kernel that's intended for one piece of hardware on a different piece of hardware will usually not work. The sabrelite has the same SoC here, so booting works better than it would if you tried to do it with an entirely unrelated QEMU machine, but if at any time your guest code tries to access hardware outside the SoC which is specific to the reMarkable then you will find it doesn't work. If you really need to get the stock kernel for the hardware to boot you will probably at some point need to bite the bullet and implement a proper machine model of it in QEMU with the relevant devices present.
If you don't actually need to run anything on the guest that cares about the specific differences between one imx6 system and another, you might be able to get away with using a kernel and DTB for the sabrelite board plus the rootfs from the reMarkable.
I was investigating the journaling layer used in the EXT4 (JBD2) and I added some printk to see the behavior of the ext4_journal_start and ext4_journal_stop functions being called.
This is the procedure:
I first format a given partition using:
sudo mke2fs -t ext4 /dev/vdb
(I am using QEMU to run this experiment)
Then I mount it:
sudo mount /dev/vdb /mnt/mydisk
That is the normal procedure for mounting, but when I mount it, because of my printk's functions in both ext4_journal_start/stop, the dmesg shows a lot of calls to journal_stop without any journal_start.
P.S.: I should guess that it is some background behavior of EXT4 or something, but I have no idea what is it.
Here is the dmesg output:
* Restoring resolver state... [ OK ]
* Stopping System V runlevel compatibility [ OK ]
[ 124.648904] JOURNAL STOP
[ 124.778691] JOURNAL STOP
...
[... ] # it is called maybe more than 40 times
...
[ 129.641895] jbd2_journal_commit_transaction
[ 129.769132] JOURNAL STOP
...
[... ] # it is called maybe more than 40 times
...
[ 134.766164] jbd2_journal_commit_transaction
After 134 seconds, it stops these messages, and then I try to write some file into that mounting point, and it behaves as expected.
[ 624.995549] JOURNAL START
[ 624.996849] JOURNAL STOP
[ 625.000676] JOURNAL START
[ 625.001757] JOURNAL START
[ 625.002822] JOURNAL STOP
[ 625.003773] JOURNAL STOP
[ 631.004110] jbd2_journal_commit_transaction
So, it is strange that after mounting, even that I did absolutely nothing, these functions are being called (journal_stop) several times and, furthermore, after two commits (the function call jbd2_journal_commit_transaction) the dmesg gets stable, and it then follows an expected behavior.
To make it clear, my question is: what causes this several calls without any reason (the ext4_journal_stop)?
By debugging the ext4 source code, I discovered what causes the several journal operations right after mounting the file system.
The ext4 is creating the inode table, so that is why the journal is called several times right after mounting a partition.
SYNOPSIS
mount -o degraded,ro /dev/disk/by-uuid/ec3 /mnt/ec3/ && echo noerror
noerror
DESCRIPTION
mount -t btrfs fails but returns with noerror as above and only since the last reboot.
btrfs check seems clean to me (I am simple user).
btrfs restore errors out with "We have looped trying to restore files in"...
I had a lingering artifact btrfs filesystem show giving " *** Some devices missing " on the volume. This meant it would not automount on boot and I have been manually mounting (+ searching for a resolution to that)
I have previously used rdfind to deduplicate with hard links (as many as 10 per file)
I had just backed up using btrfs send/recieve but have to check if I have everything - this was the main Raid1 server
DETAILS
btrfs-find-root /dev/disk/by-uuid/ec3
Superblock thinks the generation is 103093
Superblock thinks the level is 1
Found tree root at 8049335181312 gen 103093 level 1
btrfs restore -Ds /dev/disk/by-uuid/ec3 restore_ec3
We have looped trying to restore files in
df -h /mnt/ec3/
Filesystem Size Used Avail Use% Mounted on
/dev/dm-0 16G 16G 483M 97% /
mount -o degraded,ro /dev/disk/by-uuid/ec3 /mnt/ec3/ && echo noerror
noerror
df /mnt/ec3/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/dm-0 16775168 15858996 493956 97% /
btrfs filesystem show /dev/disk/by-uuid/ec3
Label: none uuid: ec3
Total devices 3 FS bytes used 1.94TiB
devid 6 size 2.46TiB used 1.98TiB path /dev/mapper/26d2e367-65ea-47ad-b298-d5c495a33efe
devid 7 size 2.46TiB used 1.98TiB path /dev/mapper/3c193018-5956-4637-9ec2-dd5e49a4a412
*** Some devices missing #### comment, this is an old artifact unchanged since before unable to mount
btrfs check /dev/disk/by-uuid/ec3
Checking filesystem on /dev/disk/by-uuid/ec3
UUID: ec3
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
found 2132966506496 bytes used err is 0
total csum bytes: 2077127248
total tree bytes: 5988204544
total fs tree bytes: 3492638720
total extent tree bytes: 242151424
btree space waste bytes: 984865976
file data blocks allocated: 3685012271104
referenced 3658835013632
btrfs-progs v4.1.2
Update:
After a reboot (had to wait for a slot to go down) the system mounts manually but not completely clean.
Now asking question on irc #btrfs:
! http://pastebin.com/359EtZQX
Hi, I'm scratching my head and searched in vain to remove *** Some devices missing. Can anyone help give me a clue to clean this up?
- is there a good way to 'fix' the artifacts I am seeing? trying: scrub, balance. Try: resize, defragment.
- would I be advised to move to a new clean volume set?
- would a fix via a btrfs send/recieve be safe from propogating errors?
- or (more painfully) rsync to a clean volume? http://pastebin.com/359EtZQX (My first ever day using irc)
Hope it's OK to jot this down, even if I cannot accept answer immediately (and hope it's OK for SO - as there is a C patch below):
It seems I screwed up the hard disk on my desktop PC ({DRDY err}). So I wanted to run a bootable media to run fsck, but the CD on this desktop is broken, so I can only use USB flash. I have a couple of USB thumbdrives with Ubuntu and Suse - these start booting on the desktop; but during boot, udev tries to detect hard drives, and since the hard disk is screwed, it just loops there, and the respective OS never finishes booting.
So I tried to download SystemRescueCd; I have this USB thumbdrive, on which I tried to install SystemRescueCD:
# lsusb with sudo, to retrieve all info
$ sudo lsusb -v -d 058f:6387 | grep -i 'id\|iManufacturer\|iProduct\|iSerial\|bInterface'
Bus 001 Device 043: ID 058f:6387 Alcor Micro Corp. Transcend JetFlash Flash Drive
idVendor 0x058f Alcor Micro Corp.
idProduct 0x6387 Transcend JetFlash Flash Drive
iManufacturer 1 takeMS
iProduct 2 Mem-drive Mini
iSerial 3 C5E7F0CC
bInterfaceNumber 0
bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk (Zip)
# search by serial:
$ find /dev/disk/by-id/ -name '*C5E7F0CC*'
/dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0-part1
/dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0
# list and get device node
$ ls -la /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0
lrwxrwxrwx 1 root root 9 2013-03-25 20:37 /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0 -> ../../sdc
$ ls -la /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0\:0-part1
lrwxrwxrwx 1 root root 10 2013-03-25 20:37 /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0-part1 -> ../../sdc1
# it is /dev/sdc - list disk info
$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 2108 MB, 2108686336 bytes
94 heads, 29 sectors/track, 1510 cylinders
Units = cylinders of 2726 * 512 = 1395712 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003e405
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 1511 2059263+ c W95 FAT32 (LBA)
I tried to use my Ubuntu 11.04 Natty netbook to image the thumbdrive - and I used both
the recommended usb_inst.sh installer; and
I tried to use unetbootin (via sudo apt-get install unetbootin);
in both of these cases, when I try to boot the USB thumbdrive on the desktop, the boot procedure fails with:
SYSLINUX 4.02 debian-20101016 CHS Copyright (C) 1993-2010 H. Peter Anvin et al
ERROR: No configuration file found
No DEFAULT or UI configuration directive found!
boot:
.... with prompt at boot. (In fact, unetbootin fails at "Verifying DMI Pool Data", before entering syslinux - probably because it is much older than the .iso I'm trying to image).
First I checked the md5 as mentioned in No Default or UI Configuration Found!
$ md5sum ./systemrescuecd-x86-3.5.0.iso
48552b9e905872bd5061eb112b73ea20 ./systemrescuecd-x86-3.5.0.iso
... but it seems OK, as per Sysresccd-versions.
Then I tried to reformat the drive to FAT16 (via sudo gparted /dev/sdc); and repeated both usb_inst.sh and unetbootin methods - again no dice. Funny enough, in all of these cases, if I try to run the flash USB thumbdrive in the QEMU emulator:
# sudo apt-get install qemu
sudo qemu -hda /dev/sdc
... it boots fine - showing the syslinux menu and so on; however, boot always fails on the desktop.
Here I should mention, that I could write down the following from the boot screen of the problematic desktop PC:
Award Modular BIOS v6.00PG
AMDRS740 BIOS
It has a boot menu accessed via F12, and in the boot menu, among other options, these are for USB:
...
USB-FDD
USB-ZIP
USB-CDROM
USB-HDD
...
Typically, I choose USB-HDD - but I've tried the others; either the procedure freezes before even entering syslinux - or the boot fails as described above.
There is advice to rename directories/files manually from isolinux to syslinux (Trying to boot from usb - Ask Ubuntu) - when I used usb_inst.sh, only syslinux/isolinux.bin would have to be renamed. There is also advice to copy syslinux.cfg to the root of the USB flash thumbdrive (Cannot boot Live USB, Linux - Super User). But still no improvements - syslinux is still complaining that it is missing the configuration file - which apparently is the syslinux.cfg.
Then I tried to look if it is possible to somehow "debug" syslinux; found log tracing/debugging/trouble shooting in syslinux - The Syslinux Project - reboot.pro:
> Do we have specific commands to trace or log syslinux?
Being open source, one is able to compile Syslinux and enable extra debugging output.
also [SOLVED] Stuck on boot: Syslinux Problem [Archive] - Ubuntu Forums: "_
Debugging syslinux is described at http://www.syslinux.org/wiki/index.php/Development/Debugging , but effective debugging (if I recall correctly) requires recompiling it to add the debug hooks._". However, Development/Debugging - Syslinux Wiki talks about something called bochs; and I suspect that is to debug syslinux itself - not necessarily to "debug" (or query) the environment it is in.
Anyways, at last, I could see no way out but to get syslinux from source; basically, this was needed so it builds:
sudo apt-get install nasm
sudo apt-get install uuid-dev
git clone git://git.kernel.org/pub/scm/boot/syslinux/syslinux.git syslinux-git
cd syslinux-git/
make OPTFLAGS+=-DDEBUG=1
Turns out, it isn't really clear how to enable such debugging, that will show what syslinux "sees" when plugged in a given computer; given that I do load into syslinux at boot, the problem is what does it see as a filesystem. I tried to enable the DEBUG environment variable as shown above (after adding override OPTFLAGS := to the Makefile) - but that, in itself, generated no new messages during boot failure. I have used the following command to "burn" the USB thumbdrive (after unmounting it from the Gnome applet):
sudo ./linux/syslinux --stupid --directory /syslinux --install /dev/sdc1
... and I've tried both with stupid and without (and both for the source-built version, and the one from the Ubuntu package repositories for Natty).
Grepping through the source, I realized there is something called rosh (Read-Only SHell) - however, that compiles as a rosh.c32 - and one is supposed to have it as a boot kernel option in syslinux.cfg - which, as noted, I cannot load. So rosh.c32 is unfortunately not much help for my problem.
However, given that rosh implements the ls command, I tried to copy relevant portions into the code of syslinux - and trigger a ls / listing of the root when syslinux scans for the configuration file. With those changes, recorded in syslinux-e40ba60-rosh-ls.patch; now I get the following when I boot:
SYSLINUX 4.06 CHS 5-ge40ba60* Copyright (C) 1993-2010 H. Peter Anvin et al
Listing: "/"
rosh_ls_arg_dir 0 files found
Listing: "/syslinux"
Listing: ""
CurrentDirName: "/syslinux/"
confignamebuf: /syslinux/extlinux.conf; realpath -1
confignamebuf: /syslinux/syslinux.cfg; realpath -1
confignamebuf: /boot/syslinux/extlinux.conf; realpath -1
confignamebuf: /boot/syslinux/syslinux.cfg; realpath -1
confignamebuf: /syslinux/extlinux.conf; realpath -1
confignamebuf: /syslinux/syslinux.cfg; realpath -1
confignamebuf: /extlinux.conf; realpath -1
confignamebuf: /syslinux.cfg; realpath -1
ERROR: No configuration file found
No DEFAULT or UI configuration directive found!
Interestingly; for the root /, the _ls function at least returns "0 files"; the others ("/syslinux", and the empty string "") already fail at the opendir call - and so the _ls function doesn't even get called!
I would have thought that my slapstick copying of the ls function would not work as intended; but running the thumbdrive in qemu on netbook, does in fact provide a full listing of files - and given that at least for /, the function gets called and returns on the desktop - I'd suspect that it does indeed work.
However, that still doesn't solve my problem - why does syslinux, after boot, see 0 files under the root /? What else could I do to debug this problem? I wouldn't mind patching some C code into syslinux - but I just don't know what I should be looking for, that would point me to correct preparation of the USB thumbdrive for booting on the desktop machine...
OK, I got it to boot...
First, I noted there are alternative mbr's in the built git source as per Mbr - Syslinux Wiki and HowTos - Syslinux Wiki, so I tried both mbr.bin and altmbr.bin - altmbr.bin like this:
$ printf '\1' | cat mbr/altmbr.bin - | sudo dd bs=440 count=1 conv=notrunc iflag=fullblock of=/dev/sdc
... but that didn't help much.
Finally, I noted that lsusb says "bInterfaceProtocol 80 Bulk (Zip)"; and I remembered reading something about ZIP drives somewhere, so tried to look it up - and finally found this:
syslinux/doc/usbkey.txt
The proper mode to boot a USB key drive in is "USB-HDD". That is the
ONLY mode in which the C/H/S geometry encoded on the disk itself
doesn't have to match what the BIOS thinks it is. Since geometry on
USB drives is completely arbitrary, and can vary from BIOS to BIOS,
this is the only mode which will work in general.
Some BIOSes have been reported (in particular, certain versions of the
Award BIOS) that cannot boot USB keys in "USB-HDD" mode. This is a
very serious BIOS bug, but it is unfortunately rather typical of the
kind of quality we're seeing out of major BIOS vendors these days. On
these BIOSes, you're generally stuck booting them in USB-ZIP mode.
THIS MEANS THE FILESYSTEM IMAGE ON THE DISK HAS TO HAVE A CORRECT
ZIPDRIVE-COMPATIBLE GEOMETRY.
....
The script "mkdiskimage" which is supplied with the syslinux
distribution can be used to initialize USB keys in a Zip-like fashion.
To do that, calculate the correct number of cylinders (31 in the
example above), and, if your USB key is /dev/sda (CHECK THE KERNEL
MESSAGES CAREFULLY - IF YOU ENTER THE WRONG DISK DRIVE IT CANNOT BE
RECOVERED), run:
mkdiskimage -4 /dev/sda 0 64 32
(The 0 means automatically determine the size of the device, and -4
means mimic a zipdisk by using partition 4.)
So, as recommended there, first I find the number of cylinders for my thumbdrive:
$ grep 512-byte /var/log/syslog | tail -n 1
Mar 25 22:33:34 mypc kernel: [50884.608687] sd 45:0:0:0: [sdc] 4118528 512-byte logical blocks: (2.10 GB/1.96 GiB)
# get number of cylinders:
$ wcalc '4118528/(64*32)'
= 2011
... then I continue with mkdiskimage. After that was done, I tried usb_inst.sh again - and realized that it will overwrite the partition 4 that mkdiskimage made, and make a partition 1 for itself instead. That means, one should copy those files fron usb_inst.sh in a backup elsewhere, then run mkdiskimage - then finally copy the backed up files back to thumbdrive again; here is a command line log:
# mkdiskimage is present in syslinux-git:
$ ./utils/mkdiskimage
Usage: ./utils/mkdiskimage [-doFMz4][-i id] file c h s (max: 1024 256 63)
....
# ... but also in Debian/Ubuntu packaging of syslinux
$ mkdiskimage -4 /dev/sdc 0 64 32
/usr/bin/mkdiskimage: /dev/sdc: don't know how to determine the size of this device
# use sudo - note this command takes a while to complete:
$ sudo mkdiskimage -4 /dev/sdc 0 64 32
Warning: more than 1024 cylinders (2011).
Not all BIOSes will be able to boot this device.
$ ls /dev/sdc*
/dev/sdc /dev/sdc4
$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 2108 MB, 2108686336 bytes
64 heads, 32 sectors/track, 2011 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x866262cc
Device Boot Start End Blocks Id System
/dev/sdc4 * 1 2011 2059248 e W95 FAT16 (LBA)
# (make sure umounted / ejected)
# cd to usb_inst.sh directory; and
# run usb_inst.sh for /dev/sdc; note it will:
# write MBR and "Creating filesystem on /dev/sdc1..."
# and "installing boot loader on /dev/sdc1";
# regardless of the previous setup on partition 4:
sudo bash ./usb_inst.sh
# now no more partition 4:
$ ls /dev/sdc*
/dev/sdc /dev/sdc1
# ( mount /dev/sdc1 via disk applet )
$ rsync -a /media/SYSRESC /media/backup/
# ... duhh... - again now
# ( umount/eject via disk applet )
$ sudo mkdiskimage -4 /dev/sdc 0 64 32
Warning: more than 1024 cylinders (2011).
Not all BIOSes will be able to boot this device.
$ sudo ./linux/syslinux --install /dev/sdc4
# ( mount via disk applet )
$ rsync -a /media/backup/SYSRESC/ /media/31A8-40E9/
$ sudo qemu -hda /dev/sdc # works
# ( umount/eject via disk applet )
# boot on desktop - works! loads rescue64 and initram.igz...
The interesting thing is - even if there is the warning "Not all BIOSes will be able to boot this device."; somehow this problematic BIOS loads this thumbdrive without a problem (and the _ls function above lists fine). Also interesting - here I choose the USB-HDD boot option (not the USB-ZIP) and it still works ?!
So, as a partial answer - I guess the way to debug this, would be for syslinux to somehow write on the thumbdrive the CHS geometry it sees during the syslinux installation; and on boot, to query the BIOS (I guess) about which CHS geometry the BIOS sees - and then dump these two geometries to screen; if there is a mismatch, then it is likely one should run mkdiskimage (unfortunately, I wouldn't know how to code that into syslinux)
Going back to my original HDD problem - turns out also SystemRescueCD uses udev to probe for devices - and again the boot process cannot complete (even if I choose the boot option "all files to memory (docache)")... So I get messages like:
udevadm settle - timeout of 180 seconds reached, the event queue contains:
Activating dmraid (fake hardware raid) ...
Starting mdadm (linux software raid) ....
udevd[88] worker [91] unexpectedly returned with status 0x0100 ...
udevd[88] worker [91] failed while handling '/devices/pci0000:00/.../sdb/sdb1'
So, I either find a Live USB distro which does not probe for disks using udev - or I better take this HDD out, toss it into a HDD USB enclosure, and try fsck it on another computer (hopefully I'll be able to blacklist this drive from udev on a running system)
Edit Aug 24 2013: Back to this problem, I thought I'd jot down few extra notes:
Since I cannot yet afford the time to fix this PC and its faulty drive, I've used this USB thumbdrive to boot multiple operating systems: PartedMagic and SliTaz did also encounter errors on the hard disk - but apparently use different drivers to access it (so the DRDY ERR loop didn't start), and they could finish booting relatively fast. Then I tried building a custom Ubuntu 12.04 image (using ubuntu-builder) - and this one ended up in a DRDY ERR loop, which may take more than 5 minutes to complete, before the OS finishes booting. I have posted more about this in Bug #1216397 “
It should be possible to ignore (skip probing) a known bad disk partition at boot” : Bugs : “linux” package : Ubuntu.
There are a few interesting things in respect to syslinux, now that this USB thumbdrive is used to boot multiple operating systems. First of all, the thumbdrive is, still, first made bootable with syslinux --install while empty (which places a file ldlinux.sys in the partition's root) - which corresponds to the mkdiskimage step above; and only afterwards are files (like kernel images, and including /boot/syslinux/syslinux.cfg) copied to it.
Now, I'd first build the CD image ISO in ubuntu-builder, and test it using VirtualBox (as qemu on my machine is way too slow for that). Once the ISO image was shown to work as expected, then only the files under its casper directory are relevant for the USB thumbdrive thus prepared; and they can be referenced through a boot menu entry in syslinux.cfg. So, I'd edit the syslinux.cfg on the thumbdrive, and copy the casper image files (e.g. filesystem.squashfs) to the thumbdrive - and test it with qemu as above. Once this qemu step passed, I'd move the USB thumbdrive on the target PC with the broken drive - and interestingly, here I might get syslinux boot failures of multiple sorts (during different boot stages):
"No DEFAULT or UI configuration directive found!" (or sometimes a "Bad <something> ..." message), before the syslinux boot menu is shown - even if the debug, as above, would show that syslinux reads the filesystem on the thumbdrive correctly, and finds the /boot/syslinux/syslinux.cfg (which does have proper directives)!
"Invalid or corrupt kernel image", once the syslinux menu is shown, and the new kernel image (Ubuntu) chosen - even if the other images (found previously on the thumb) boot fine on the broken drive PC; and the new image boots fine from thumb in qemu on a different machine!
"/init: line 7: can't open /dev/sr0: no medium found", once the new (Ubuntu) image is chosen from syslinux menu, and it starts booting; this seems an Ubuntu specific message, appearing a few seconds after it starts booting. I still encounter it even if booting completes succesfully - when it's a problem, this message just loops repeatedly, not allowing the rest of the boot process to complete
It turns out, any of these can appear whenever I try to change and save the syslinux.cfg file on the thumbdrive; or when I make changes in the casper image files, and I rsync or copy them to the thumbdrive. Maybe the copying process (since it may change the sectors where the files are located on the thumb), "confuses" parts of the boot process - although, this shouldn't happen, since also the working procedure above starts from a blanked, syslinux'd thumbdrive, to which files are copied after; so I think this may point to failing sectors on the thumbdrive.
However, even in this state, the working procedure above seemed to be useful - because using it, I could recover the thumb back to a working state! In more detail, it goes like this:
Keep a copy of the thumbdrive files somewhere on a different disk (e.g. ~/thumbcopy) - but without the ldlinux.sys file.
Whenever you want to make a change (to syslinux.cfg or to bootable image files) - make sure this change is saved in ~/thumbcopy first
Now, say I've changed some files on the bootable thumbdrive directly, and I encounter one of the errors above. Then:
First, delete all files but the ldlinux.sys on the thumbdrive, e.g.: rm -rf $(ls -I"ldlinux.sys" /media/31A8-40E9/)
Then, rsync or copy (cp -arv ...) the files in ~/thumbcopy to the thumbdrive, e.g.: rsync -aP ~/thumbcopy/ /media/31A8-40E9/
Now, try boot the thumbdrive in the PC again - it usually boots fine!
I've encountered all three types of errors, because I'd often try to change/copy individual files directly in the thumbdrive: sometimes the change doesn't introduce a problem, so booting is fine - however, in many cases, it does introduce a problem. For some reason, using the above procedure I managed to recover the thumbdrive from either type of abovementioned problems - maybe it has to do with USB Flash delayed writes, maybe with USB Flash failing sectors, I cannot really tell... But in any case: deleting all files, and re-copying them in one go, does seem to be a worthwhile procedure to try in case of errors like that.
It's an ancient post, but in case others stumble upon this, I'll add an answer anyway.
If you're struggling to get syslinux to boot, ROSH (Read-only Shell) can be useful, as you mentioned. To start ROSH, you can simply type rosh at the boot: prompt (if you do have a working graphical menu, press escape to drop back to the boot: prompt.
Inside the shell, you have some basic commands to look around in your environment. For more documentation, see https://wiki.syslinux.org/wiki/index.php?title=Read-Only_SHell(rosh.c32)