A little background: I have tried to use rsync to backup my wife's home directory to an external usb drive with the command
rsync -va /home/wife /run/media/wife
but kept getting error messages that mkstemp failed, and that rsync failed to set times, becuase of a read-only filesystem. Worse, it seems that rsync is unable to tell when files don't need syncing, and winds up copying a lot of stuff it doesn't need to, resulting in rediculously slow backup times.
So I tried using rsync -rtvO instead, based on this guy's advice. Okay, no more warnings, but the backups still seem too slow, and esp on big media files that already exists -- i.e. it's still copying stuff unnecessarily.
Is my analysis correct?
Is there a workaround?
Will the problem be fixed if I use an NTFS drive for here backups?
I could of course use a linux filesytem, but on rare occasions she would like to be able to take the drive to work and access it from the Windows machines there.
Try using --modify-window=1
In particular, when transferring to or from an MS Windows FAT
filesystem (which represents times with a 2-second resolution),
--modify-window=1 is useful (allowing times to differ by up to 1 second).
https://download.samba.org/pub/rsync/rsync.html
You could also try using --size-only
skip files that match in size
For rsync to FAT, this is what I use and it seems to work pretty well:
rsync -rtv --modify-window=1 source/ destination/
Source: https://serverfault.com/a/144475/58568
Related
My company is selling some old AIX systems, and they want me to wipe all data from them. Not sure of the best way to do that. I'm guessing it would go something like this:
Unmount the filesystem
lookup the 'hdisk#" device in fstab - and comment it out.
use dd to copy /dev/null to the device
What I'm not sure about is whether the whole logical volume manager complicates things further than that. I.e., is it correct to treat the /dev/hdisk# device like a physical disk for this purpose?
You can boot from Standalone Diagnostics CD and wipe the drives. You have options to format and certify, format without certify and erase. You can run each option more than one time if you need.
After unmounting the file systems you should do an exportvg <your_vg>, otherwise you get complaints when rebooting the system (although, taking into account you are selling them, this is probably not your concern any more).
After removing the VG from the system you can wipe the disks using dd as you described.
(It is a common misconception that you have to wipe more than once. Just do it correctly and once is enough.)
A few weeks ago, I copied some directories from an old external hard drive to my iMac. I ended up with a few corrupt files due to the bad state of this ancient hard drive.
I managed to delete them, but some remain.
At the beginning, I thought it was only a question of file permissions or character encoding, but surprisingly, it's not. I've read every single posts in stackoverflow about similar issues on other unix systems.
Here are some screenshots, you'll understand easily the issue.
That is my OS and the bad files :
I can not delete anything, even after rewriting :
I can not chmod. Chmod shows me some ghost files I didn't know existed.
I can not chown either :
This is an encrypted APFS partition :
I can not run any Disk Utility repair because of this FS corruption (exit error code 8). This doesn't work either on my Mojave USB installer, same error code.
I can not decrypt the partition using FireVault, it stays stuck at 0% forever.
I never saw anything like this. What do you think we should do ?
I have some automated test (using CUnit) which require a "disk-image"-file (raw copy of a disk) to be "mounted" in windows and explored. I have previously used a tool/library called "FileDisk-17" , but this doesn't seem to work on my Windows 7 (64bit).
Update
I should point out, that changing the image-format (to say VHD) is not at option.
Any suggestions as to other (perhaps better supported) tools or libraries for mouting the file? The project is coded in ANSI C and compiled using MinGW.
Best regards!
Søren
Edit: Searching Bing for +filedisk 64 brings up a 64bit build of FileDisk, the utility you refer to:
http://www.winimage.com/misc/filedisk64.htm
And FileDisk-15 signed for 64bit here:
http://www.acc.umu.se/~bosse/
I can't vouch for it as I have never used it and am not familiar with the author.
Alternatively:
If you have a VHD, you can mount that in windows easily:
http://technet.microsoft.com/en-us/library/cc708295(WS.10).aspx
See also:
http://www.petri.co.il/mounting-vhd-files-with-vhdmount.htm
Since you have a raw DD image not a VHD, you will need to convert it first:
http://www.bebits.com/app/4554
Or qemu-img.exe can also do this:
qemu-img.exe convert -f raw rawdisk.img -O vpc rawdisk.vhd
Alternatively, you can create an empty VHD, and use DD to copy the raw image to the VHD, by opening the VHD as a raw device.
I faced this problem recently and found ImDisk to be an extremely nice solution:
Free, with source available and a very flexible open source license
Trivial setup (I have seen filedisk64 (in the accepted answer) described as having a "technical" setup)
Straightforward GUI and command-line access
Worked on Windows 7 64-bit
Seems to happily mount any kind of filesystem recognised by Windows (in my case, FAT16)
Works with files containing
Raw partitions
Entire raw disks (i.e. including the MBR and one or more partitions; which partition to mount can be selected)
VHD files (which it turns out are just raw partitions or disks with a 512-byte footer appended!)
Also can create RAM drives -- either initially empty or based on an existing disk image! (Very neat I must say!)
I did encounter minor issues trying to unmount drives. I was unable to unmount a drive from the GUI right-click context menu as the drive appeared to be "in use" by the explorer.exe process. Closing the Explorer window and using imdisk -d -m X: also didn't work; however imdisk -D -m X: (-D "forces" an unmount, whatever that means) did. This worked even if the drive was visible in an open Explorer window, without appearing to create any problems. However even after the drive appeared to have fully unmounted, an imdisk -l to list all available devices would still report that \Device\ImDisk0 exists, and if you remount the drive later, both that and \Device\ImDisk1 will appear in the output of imdisk -l (and so on with more unmount/remount cycles). This didn't create any problems with actually using the mounted drive when I tried a few unmount/remount cycles, though it theoretically might if you perform this many times between reboots.
ImDisk was invaluable for transferring the contents of a 1.5Gb disk drive with one FAT16 DOS partition from an ancient 486 machine.
I have a folder a/ and a remote folder A/.
I now run something like this on a Makefile:
get-music:
rsync -avzru server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.
This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.
In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.
If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).
Thing is I have 3 machines in context:
desktop
notebook
home-server
So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.
I guess this is only possible with a database and track of operations :P
Any simpler solutions?
Thank you.
Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/
Syntax:
unison dirA/ dirB/
Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:
unison -auto dirA/ dirB/
unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).
Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.
Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.
It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.
FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.
On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops.
I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.
You could also try bitpocket: https://github.com/sickill/bitpocket
Try this,
get-music:
rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).
You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.
You could try csync, it is the sync engine under the hood of owncloud.
I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.
I'm now using SparkleShare https://www.sparkleshare.org/
works on mac, linux and windows.
I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.
Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.
For e.g. if i am storing some files on a network server which is under many hierarchical folders.
Then i want to do backup. But i always encounter issue because the file path is more than 255?
How can i resolve this issue or work around it?
Preface: I'm assuming the OS of the machine you want to copy the files to is some flavor of Windows.
The first part of Mark Bessey's is somewhat correct, however even on modern versions of windows with modern filesystems (NTFS for example) you can still run into problems.
I suspect the limitation you're running into is due to MAX_PATH, which is a predefined limit on the length of a path that many APIs on Windows will accept.
You may try using Robocopy to do the backup as it is able to create paths longer than the MAX_PATH limitation. However, most applications will not be able to access these files.
Taring or Zipping the files may be a good plan but it seems unlikely that you'd be able to unzip or untar them to a Windows machine.
Maybe upgrade to an Operating System that's been updated in the last decade or so? Seriously, though - what OS and file system are you using? Even FAT32 supports long path and file names, though any single component of the path is limited to 255 characters.
If you've got directories with more than 255 characters in their name, then that's problematic (and a little weird). To work around that issue, you could consider using an archiving utility (tar or zip) run on the server, then ship the archive over to yuor desktop machine.
What OS, What protocol?
All I can see is: CoreFSIF with a limit of 256 chars.
What you are seeing is Win32's MAX_PATH limit (MAX_PATH is 260).
There are "escapes" on the API to be able to use NT's larger limit of 32767 characters (prepending \\?\ to the path), which however only work on some of the APIs. Files created this way tend to confuse other programs which expect all paths to fit in MAX_PATH-sized buffers.
The way to work around this (for backup/restore) is to find a program which can backup (and restore!) files with longer paths (by either always using the API "escapes" or the underlying NT API). Another option is to do the backup/restore with a different operating system.