I have a new SSD (256GB) where I got installed ESXi 6.7 (the usual bunch of standard partitions) + couple of VMFS volumes (20GB each) that contains two Linux flavors. Everything works. So, now I would like to have a backup copy of the system then I cloned the SSD into another identical SSD (in the future I wanted to have a file image .iso) but when I test the new cloned USB something is wrong. ESXi boots up and I do see the two VMFS partitions but they do not appear as datastores, they are not usable and the two virtual machines appear to be broken (most likely because of no datastore).
I do the copy, booting with a live linux and using dd:
dd if=/dev/sdb of=/dev/sda bs=512 conv=noerror, sync
of course sda and sdb are not in use when I clone, as Linux boots up from the USB.
Any idea why the exact copy delivered by dd does not work exactly the same original SSD? Is there any special settings to be used with dd?
I still don't know why a bit by bit cloned copy does not show the same but eventually mounting manually (via ssh) the VMFS partitions makes the magic and the virtual machine works.
Related
My company is selling some old AIX systems, and they want me to wipe all data from them. Not sure of the best way to do that. I'm guessing it would go something like this:
Unmount the filesystem
lookup the 'hdisk#" device in fstab - and comment it out.
use dd to copy /dev/null to the device
What I'm not sure about is whether the whole logical volume manager complicates things further than that. I.e., is it correct to treat the /dev/hdisk# device like a physical disk for this purpose?
You can boot from Standalone Diagnostics CD and wipe the drives. You have options to format and certify, format without certify and erase. You can run each option more than one time if you need.
After unmounting the file systems you should do an exportvg <your_vg>, otherwise you get complaints when rebooting the system (although, taking into account you are selling them, this is probably not your concern any more).
After removing the VG from the system you can wipe the disks using dd as you described.
(It is a common misconception that you have to wipe more than once. Just do it correctly and once is enough.)
A little background: I have tried to use rsync to backup my wife's home directory to an external usb drive with the command
rsync -va /home/wife /run/media/wife
but kept getting error messages that mkstemp failed, and that rsync failed to set times, becuase of a read-only filesystem. Worse, it seems that rsync is unable to tell when files don't need syncing, and winds up copying a lot of stuff it doesn't need to, resulting in rediculously slow backup times.
So I tried using rsync -rtvO instead, based on this guy's advice. Okay, no more warnings, but the backups still seem too slow, and esp on big media files that already exists -- i.e. it's still copying stuff unnecessarily.
Is my analysis correct?
Is there a workaround?
Will the problem be fixed if I use an NTFS drive for here backups?
I could of course use a linux filesytem, but on rare occasions she would like to be able to take the drive to work and access it from the Windows machines there.
Try using --modify-window=1
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
--modify-window=1 is useful (allowing times to differ by up to 1 second).
https://download.samba.org/pub/rsync/rsync.html
You could also try using --size-only
skip files that match in size
For rsync to FAT, this is what I use and it seems to work pretty well:
rsync -rtv --modify-window=1 source/ destination/
Source: https://serverfault.com/a/144475/58568
I have some automated test (using CUnit) which require a "disk-image"-file (raw copy of a disk) to be "mounted" in windows and explored. I have previously used a tool/library called "FileDisk-17" , but this doesn't seem to work on my Windows 7 (64bit).
Update
I should point out, that changing the image-format (to say VHD) is not at option.
Any suggestions as to other (perhaps better supported) tools or libraries for mouting the file? The project is coded in ANSI C and compiled using MinGW.
Best regards!
Søren
Edit: Searching Bing for +filedisk 64 brings up a 64bit build of FileDisk, the utility you refer to:
http://www.winimage.com/misc/filedisk64.htm
And FileDisk-15 signed for 64bit here:
http://www.acc.umu.se/~bosse/
I can't vouch for it as I have never used it and am not familiar with the author.
Alternatively:
If you have a VHD, you can mount that in windows easily:
http://technet.microsoft.com/en-us/library/cc708295(WS.10).aspx
See also:
http://www.petri.co.il/mounting-vhd-files-with-vhdmount.htm
Since you have a raw DD image not a VHD, you will need to convert it first:
http://www.bebits.com/app/4554
Or qemu-img.exe can also do this:
qemu-img.exe convert -f raw rawdisk.img -O vpc rawdisk.vhd
Alternatively, you can create an empty VHD, and use DD to copy the raw image to the VHD, by opening the VHD as a raw device.
I faced this problem recently and found ImDisk to be an extremely nice solution:
Free, with source available and a very flexible open source license
Trivial setup (I have seen filedisk64 (in the accepted answer) described as having a "technical" setup)
Straightforward GUI and command-line access
Worked on Windows 7 64-bit
Seems to happily mount any kind of filesystem recognised by Windows (in my case, FAT16)
Works with files containing
Raw partitions
Entire raw disks (i.e. including the MBR and one or more partitions; which partition to mount can be selected)
VHD files (which it turns out are just raw partitions or disks with a 512-byte footer appended!)
Also can create RAM drives -- either initially empty or based on an existing disk image! (Very neat I must say!)
I did encounter minor issues trying to unmount drives. I was unable to unmount a drive from the GUI right-click context menu as the drive appeared to be "in use" by the explorer.exe process. Closing the Explorer window and using imdisk -d -m X: also didn't work; however imdisk -D -m X: (-D "forces" an unmount, whatever that means) did. This worked even if the drive was visible in an open Explorer window, without appearing to create any problems. However even after the drive appeared to have fully unmounted, an imdisk -l to list all available devices would still report that \Device\ImDisk0 exists, and if you remount the drive later, both that and \Device\ImDisk1 will appear in the output of imdisk -l (and so on with more unmount/remount cycles). This didn't create any problems with actually using the mounted drive when I tried a few unmount/remount cycles, though it theoretically might if you perform this many times between reboots.
ImDisk was invaluable for transferring the contents of a 1.5Gb disk drive with one FAT16 DOS partition from an ancient 486 machine.
I have a folder a/ and a remote folder A/.
I now run something like this on a Makefile:
get-music:
rsync -avzru server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.
This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.
In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.
If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).
Thing is I have 3 machines in context:
desktop
notebook
home-server
So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.
I guess this is only possible with a database and track of operations :P
Any simpler solutions?
Thank you.
Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/
Syntax:
unison dirA/ dirB/
Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:
unison -auto dirA/ dirB/
unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).
Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.
Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.
It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.
FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.
On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops.
I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.
You could also try bitpocket: https://github.com/sickill/bitpocket
Try this,
get-music:
rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).
You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.
You could try csync, it is the sync engine under the hood of owncloud.
I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.
I'm now using SparkleShare https://www.sparkleshare.org/
works on mac, linux and windows.
I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.
Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.
I have two disks as .vmdk files, and four as .vdi files. I can boot virtual machines on them with Sun xMV VirtualBox, and they work just fine. However, I want to mount them on my local computer so I can read some files off of them without starting a virtual machine. I downloaded the vmware-mount utility, but I get this error:
Unable to mount the virtual disk. The disk may be in use by a virtual
machine, may not have enough volumes or mounted under another drive
letter. If not, verify that the file is a valid virtual disk file.
Thinking it's a problem with the utility, I downloaded the SDK and made my own simple program in C to try to mount a disk. It just initializes the API, connects to it, then attempts to open the disk. I get this error, once again claiming it is not a virtual disk:
**LOG: DISKLIB-DSCPTR: descriptor above max size: I64u
**LOG: DISKLIB-LINK : "f:\programming\VMs\windowstrash.vdi" : failed to open (The file specified is not a virtual disk).
**LOG: DISKLIB-CHAIN : "f:\programming\VMs\windowstrash.vdi" : failed to open (The file specified is not a virtual disk).
**LOG: DISKLIB-LIB : Failed to open 'f:\programming\VMs\windowstrash.vdi' with flags 0x1e (The file specified is not a virtual disk).
** FAILURE ** : The file specified is not a virtual disk
The files are clearly virtual disks, though, since I can actually mount and use them with a virtual machine. I tried detaching them from any VMs and trying again, but I got the same results.
Any ideas? Maybe the "descriptor above max size" is a hint?
.vdi is a VirtualBox supported format, but not supported by VMWare.
The .vmdk files are VMWare images - you should be able to load them fine using the vmware tool (VirtualBox supports these too, but the converse is not true).
However, if you are trying to just mount the VDI image somehow....
Try this blog
http://bethesignal.org/blog/2011/01/05/how-to-mount-virtualbox-vdi-image/
This is what you exactly should watch, if you wanna get this done within seconds.
I only posted this, since the other people who visit this page would find it easier.
Thanks.
In a case of removal of this blog in Future, posting the content briefly here would be helpful. So I would extract the contents and post it here as well.
Be the super user
sudo su
Load the nbd kernel module. - the network block device module
modprobe nbd
run qemu-nbd, which is a user space loopback block device server for QEMU-supported disk images. Basically, it knows all about weird disk image formats, and presents them to the kernel via nbd, and ultimately to the rest of the system as if they were a normal disk.
qemu-nbd -c /dev/nbd0 <vdi-file>
That command will expose the entire image as a block device named /dev/nbd0, and the partitions within it as subdevices. For example, the first partition in the image will appear as/dev/nbd0p1.
Now you could, for instance, run cfdisk on the block device, but you will most likely want to mount an individual partition.
mount /dev/nbd0p1 /mnt
When you’re done, unmount the filesystem and shut down the qemu-nbd service.
umount /mnt
qemu-nbd -d /dev/nbd0