We want HA of ActiveMQ. I explored the option of database and with Zookeeper. I wanted to check the option with Shared storage. In the documentation its mentioned about Storage Area Network. However I do not have access to SAN.
Can I use the message store to be on Shared folder on NAS ? or some shared directory on the file system ?
Regards,
JE
You could set up an NFSv4 file share and mount that directory in the kahadb persistence adapter.
Consider following in activemq.xml:
<persistenceAdapter>
<kahaDB directory="{YOUR_SHARED_STORAGE_DIRECTORY}" lockKeepAlivePeriod="5000">
<locker>
<shared-file-locker lockAcquireSleepInterval="10000" />
</locker>
</kahaDB>
</persistenceAdapter>
KahaDB works as well!
The documentation doesn't say that KahaDB also can be used,but have tried it myself KahaDB also supports locking and works well.
As KahaDB is already provided with activeMQ it will be easy to use as nothing extra will be needed.
Hope it helps,
Thanks!
You need a file system that supports file locking (ie NFS4) as described in the documentation and your filesystem needs to be very reliable. I'm not a filesystem expert, so I'm not sure what all is out there to meet that requirement, but I know that more than "SAN" does. I believe that SAN is mentioned due to the reliability, but there other options for reliable network storage out there.
Related
As part of my Research Project. I am supposed to test different DFS'S (Distributed File Systems) like HDFS, Ceph etc. For this purpose I have planned to use IBM Bluemix.
I tried doing google to get help in this regard but there is not much available in this regard. Is there any help available in this regard or any suggestion(s) please? Not getting how to start?
My understanding is you should start using Virtual Machines, get into them and install and test each File System type you want to explore: https://console.ng.bluemix.net/docs/virtualmachines/vm_index.html#vm_index
I'd like my application to be able to show a directory listing from a remote FTP (or SFTP etc) location. When a file/directory changes in the remote directory tree, the application should update its view with the relevant changes.
Because traversing the entire tree is slow and wasteful, I'd like to use something along the lines of FSEvents (inotify/kqueues on Linux), but obviously these libraries are filesystem-based, and a connection to an FTP server is not the same as a mounted filesystem.
In order to make these libraries work, I'd need to actually mount a filesystem backed by FTP/SFTP on the local machine, then attach an FSEventStream (or kqueue etc) to this local mount. I know FUSE can do this, but is there any way I can use FUSE without the user having to first install it? I mean, can I bundle it with my (Mac) application and create mounts without having to put the user through the process of actually running an installer package to copy libfuse and the kernel modules into the system? Does it assume /dev/fuse exists, or can this live outside the /dev/ path, inside my application directory?
Nice Mac applications are installed with a simple drag & drop and I'd like to keep mine this way if possible. I'm unclear on if it's possible to use libfuse directly (provided the files are included with the app), without installing it in the system paths.
Alternatively, does anyone have any other suggestions for monitoring for changes over FTP, without polling?
Unfortunately FTP and SFTP do not support any form of client notification.
Much like HTTP they are based on a request/response scheme, where each data transfer is initiated by the client. What makes things worse is that, contrary to HTTP, there is no way to ask the server to inform the client of any changes since a specific date.
This means that not only you have to use polling, but also that said polling will by no means be lightweight.
As far as FUSE is related, most FTP and SFTP modules that are available only update their view of the filesystem when the userspace applications ask for a directory listing (e.g. hitting Refresh in a file browser window). They do not perform polling on their own. Your userspace application will have to initiate the refresh by polling the directory itself.
EDIT:
To clarify a couple of things, recent versions of FUSE do support notification events. They
simply pass through the events from the modules to the kernel. The modules still have to generate them and in the case of an FTP/SFTP client module that is impossible without polling the server.
Also keep in mind that many current NFS implementations do not support change notifications either, despite the fact that NFSv4.1 has the necessary provisions. Many SMB/CIFS servers (esp. those in cheap Network-Attached-Storage embedded systems) also have limited to no support.
I have a folder a/ and a remote folder A/.
I now run something like this on a Makefile:
get-music:
rsync -avzru server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.
This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.
In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.
If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).
Thing is I have 3 machines in context:
desktop
notebook
home-server
So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.
I guess this is only possible with a database and track of operations :P
Any simpler solutions?
Thank you.
Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/
Syntax:
unison dirA/ dirB/
Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:
unison -auto dirA/ dirB/
unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).
Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.
Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.
It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.
FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.
On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops.
I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.
You could also try bitpocket: https://github.com/sickill/bitpocket
Try this,
get-music:
rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).
You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.
You could try csync, it is the sync engine under the hood of owncloud.
I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.
I'm now using SparkleShare https://www.sparkleshare.org/
works on mac, linux and windows.
I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.
Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.
I am trying to write a simple program, preferably in C, that will watch a given directory. Whenever a process accesses that directory, I just want to print out the name of that process. It seems simple, but I am coming up short for solutions on MSDN. Does anyone know which library calls I will need for this, or any helpful advice? I have considered repeatedly querying for what processes have handles on the given directory and just watching for additions to that list.This approach just seems very intensive and I am hoping there is an easier way. Thanks.
I'm not sure if there's an easier way, but one way is to use a file system filter driver. Or easier a file system minifilter driver.
You can filter, log, track, control, ... all IO.
There is no supported way to do this from user mode. You can use the FindFirstChangeNotification API to tell when a file or directory has changed, but that doesn't tell you who did it. You might be able to hook some things to obtain this information... but that is of course not supported.
If you can use a driver, you can use Event Tracing for Windows for this information. This is what Sysinternals ProcMon uses. But installation of a driver is a very invasive process, bugs in your driver cause BSODs, and installation of a driver requires administrative rights. Something to keep in mind.
Is there any way to have something that looks just like a file on a Windows file share, but is really a resource served up over HTTP?
For context, I'm working with an old app that can only deal with files on a Windows file share, I want to create a simple HTTP-based service to serve the content of the files dynamically to pick up real time changes to the underlying data on request.
WebDAV (basically) takes an existing directory, and shares it over HTTP - which sounds like the opposite of what you want.
You need something that speaks SMB/CIFS on one end, and your own code on the other. The easiest way to do that is with a userspace file system.
To that end, here's a couple of links:
WinFUSE, which is kind of a barebones CIFS/SMB server that can host your own filesystem. I've done a couple of small samples with it - and the docs are terrible, but it more or less worked.
Dokan, a userspace file driver with .NET bindings. I haven't used this one, but it looks promising. It has both .NET and Ruby bindings, so you should be able to get a POC up pretty quickly.
Callback File System - yet another userspace file system. Again, I have no experience with this one.
A Linux box with SAMBA and FUSE that shares the drive out to the Windows box.
This won't answer your question in any meaningful way, but maybe it will get you pointed in the right direction. Look into serving the "file(s)" via WebDAV--SharePoint uses this and its files can be accessed exactly as you want, as a file share where the transport mechanism is HTTP. Unfortunately I can't give any more detailed info, as I've only worked on the client end of WebDAV and not the server side of things.
I think serving up files from WebDAV might be what you're looking for.