Is there any way to have something that looks just like a file on a Windows file share, but is really a resource served up over HTTP?
For context, I'm working with an old app that can only deal with files on a Windows file share, I want to create a simple HTTP-based service to serve the content of the files dynamically to pick up real time changes to the underlying data on request.
WebDAV (basically) takes an existing directory, and shares it over HTTP - which sounds like the opposite of what you want.
You need something that speaks SMB/CIFS on one end, and your own code on the other. The easiest way to do that is with a userspace file system.
To that end, here's a couple of links:
WinFUSE, which is kind of a barebones CIFS/SMB server that can host your own filesystem. I've done a couple of small samples with it - and the docs are terrible, but it more or less worked.
Dokan, a userspace file driver with .NET bindings. I haven't used this one, but it looks promising. It has both .NET and Ruby bindings, so you should be able to get a POC up pretty quickly.
Callback File System - yet another userspace file system. Again, I have no experience with this one.
A Linux box with SAMBA and FUSE that shares the drive out to the Windows box.
This won't answer your question in any meaningful way, but maybe it will get you pointed in the right direction. Look into serving the "file(s)" via WebDAV--SharePoint uses this and its files can be accessed exactly as you want, as a file share where the transport mechanism is HTTP. Unfortunately I can't give any more detailed info, as I've only worked on the client end of WebDAV and not the server side of things.
I think serving up files from WebDAV might be what you're looking for.
Related
I'm currently working an a rather large web project which is written using C servlets ( utilizing GWAN Web server ). In the past I've used a couple of IDEs for my LAMP/PHP jobs, like Eclipse.
My problems with Eclipse are that you can either mirror the project locally, which isn't possible in this case as I'm working on a Mac (server does not run on OSX), or use the "remote" view, which would re-upload files when you save them.
In the later case, the file is only partly written while uploading, which makes this a no-go for a running web server, or the file could become corrupted if the connection was lost during uploading. Also, for changing some character, uploading the whole file seems rather inefficient to me.
So I was thinking:
Wouldn't it be possible to have the IDE open Vim per SSH and mirror my changes there, and then just :w (save) ? Or use some kind of diff-files for changes?
The first one would be preffered, as it has the added advantage of Vim .swp files, which makes it possible that others know when someone is already editing the file.
My current solution is using ssh+vim, but then I lose all the cool features I have with Eclipse and other more advanced IDEs.
Also, regarding X-Forwarding: The reason I don't like it is speed. It feels way slower than just editing locally, and takes up unneeded bandwidth, when all I want to do is basically "text editing".
P.S.: I couldn't find any more appropriate tags for the question, especially no "remote" tag, but if you know any, feel free to add them. Also, if there is another similar question, feel free to point it out - I couldn't find any.
Thank you very much.
If you're concerned about having to transmit the entire file for minor changes, the only solution that comes to my mind is running (either continuously, or on demand) an rsync job that mirrors the remote site to your local system (and back). The rsync protocol just transmits the delta information. According to Are rsync operations atomic at file level?, the change is atomic.
Another possibility: run everything in a virtual machine on your Mac. The server and the IDE/text editor are both on the same virtual machine so you don't have to fear network issues.
Because the source code on the virtual machine is under some kind of VCS the classic code → test → commit process is trivial (at least theoretically).
I am allowing users to upload photos like photo albums, and also attach files (documents for now) as mail attachments. So i assume I need some anti virus/security tool in place to scan the files first in case people upload infected stuff. So two questions:
1) Are there any 'free' or open source tools for this I can use or integrate into my environment: codeignitor php?
2) How to secure the upload area from rest of the system? Say the virus scanner fails to catch a virus and it is uploaded, how to prevent it from infecting other files? Like can the upload area be sandboxed in or something always and use that filepath for users to access the content so it does not spread to other parts of the system?
There is clamav for a free virus scanner. Install it and you could do something like:
function virus_detected($filename)
{
$clamscan = "/usr/local/bin/clamscan";
$result = exec("$clamscan -i --no-summary $filename");
return strlen($result)?true:false;
}
As for security, make sure the temporary files are uploaded to a directory outside of your web root. You should then verify the file type, rename the file to something other than it's original file name and append the appropriate extension (gif,jpg,bmp,png). I believe this should keep you fairly safe aside from exploits in php itself.
For more information about verifying file types in php check out:
http://www.php.net/manual/en/function.finfo-file.php
I know this topic hasn't been active for three years now, but, in case anyone else in the future, similarly, is looking for a PHP-based anti-virus solution, for those without an anti-virus daemon, program or utility installed on their host machine and without the ability to install an anti-virus daemon, program or utility, phpMussel, a PHP script that I've written based on ClamAV that fits the bill for what Rohit (the the original poster) was looking for (a PHP-based anti-virus to protect their CMS against malicious file uploads), may possibly be a viable solution. It certainly isn't perfect and I can't guarantee that it'll catch everything, but by far, it's certainly better than using nothing at all.
Ideally, as per already suggested above by Matt, making a call to shell to have ClamScan scan the file uploads is definitely an ideal solution, and if this is something that a hostmaster, webmaster or anyone in Rohit's situation is able to do, I'd second that suggestion wholly. What I've written, because it is a PHP script, has limitations inherent to anything that relies wholly on PHP in order to function, but, in instances where the aforementioned suggestion and/or similar suggestions aren't a possibility (such as if the host machine doesn't have an anti-virus installed and shell access is disabled; common with cheaper shared hosting solutions), that's where what I'm suggesting here could potentially step in - Something that only requires PHP to be installed (with PCRE extension included, which is standard with PHP nowadays anyhow), and nothing more.
Also remember, as Matt has already suggested, to always upload outside of your root directory, to ensure that uploaded files can't be exploited by attackers (such as in the event of an attacker attempting to compromise your system by uploading backdoors or trojans) - Viruses are not the only threat you need to worry about, and the vast majority of anti-virus solutions nowadays do not solely focus on viruses. Matt is also entirely correct in pointing out that no anti-virus solution is perfect, and for that reason, anyone allowing file uploads to their website or server needs to remain vigilant - An anti-virus solution is a must-have for anyone in that situation, but no holy grail of internet security that'll cover every possible threat exists. Also, renaming files isn't only about ensuring that they can't execute (as may be somewhat inferred by the original poster's reply comment regarding EXEs) - The risk of threats such as directory traversal attacks can be reduced by renaming files as well as the risk associated with an attacker attempting to override an already existing file on a targeted system as a means to hide their dirty-work.
Regarding the threat of files that may be malicious being missed by an anti-virus solution and then potentially infecting the system where they are being uploaded to; What a hostmaster or webmaster could potentially do in this situation is employ some sort of quick and simple encoding process that'd render the file non-executable by the system itself, but which can be easily and readily reversed by the PHP script responsible for calling that file on request, such as by way of using base64_encode(), bin2hex(), or even by just rotating a few characters and adding a salt to displace the file's magic number or something similar.
I want to create an application in C that allows two users to share a file. I'll call the person sending the file the server and the receiver the client. There are a few requirements:
The users need no identification, no "login". You could say they are unknown for my application.
The server selects a file for transfer and gets returned a simple ~10 character ID string/hash that the client can use to retrieve the file.
The same application is used for both serving and receiving.
My application must not need dedicated software running on a remote server, unless it's freely available (e.g. bittorrent trackers).
Now this sounds a lot like bittorrent and I am seriously thinking of doing this through bittorrent. I'm not sure how I would do this. Are there any good libraries for torrent creation / seeding / downloading?
Please answer this question by either:
Posing a viable alternative for bittorrent / other ideas.
Posting good libraries / snippets / implementations of the bittorrent protocol in C.
This does indeed sound like something best done with BitTorrent. Have you had a look at libbt? It's not very well documented but does include a sample client, which is btget.c in /src/.
I have now found this library: rasterbar libtorrent. It's in C++ but I don't mind (I don't know either that well anyway).
Sharing here for future reference if other people are looking for the same thing as me.
And an other solution, send the file through an IRC server (like Freenode). I came up with this solution after I had trouble with opening ports with bittorrent.
I'd like my application to be able to show a directory listing from a remote FTP (or SFTP etc) location. When a file/directory changes in the remote directory tree, the application should update its view with the relevant changes.
Because traversing the entire tree is slow and wasteful, I'd like to use something along the lines of FSEvents (inotify/kqueues on Linux), but obviously these libraries are filesystem-based, and a connection to an FTP server is not the same as a mounted filesystem.
In order to make these libraries work, I'd need to actually mount a filesystem backed by FTP/SFTP on the local machine, then attach an FSEventStream (or kqueue etc) to this local mount. I know FUSE can do this, but is there any way I can use FUSE without the user having to first install it? I mean, can I bundle it with my (Mac) application and create mounts without having to put the user through the process of actually running an installer package to copy libfuse and the kernel modules into the system? Does it assume /dev/fuse exists, or can this live outside the /dev/ path, inside my application directory?
Nice Mac applications are installed with a simple drag & drop and I'd like to keep mine this way if possible. I'm unclear on if it's possible to use libfuse directly (provided the files are included with the app), without installing it in the system paths.
Alternatively, does anyone have any other suggestions for monitoring for changes over FTP, without polling?
Unfortunately FTP and SFTP do not support any form of client notification.
Much like HTTP they are based on a request/response scheme, where each data transfer is initiated by the client. What makes things worse is that, contrary to HTTP, there is no way to ask the server to inform the client of any changes since a specific date.
This means that not only you have to use polling, but also that said polling will by no means be lightweight.
As far as FUSE is related, most FTP and SFTP modules that are available only update their view of the filesystem when the userspace applications ask for a directory listing (e.g. hitting Refresh in a file browser window). They do not perform polling on their own. Your userspace application will have to initiate the refresh by polling the directory itself.
EDIT:
To clarify a couple of things, recent versions of FUSE do support notification events. They
simply pass through the events from the modules to the kernel. The modules still have to generate them and in the case of an FTP/SFTP client module that is impossible without polling the server.
Also keep in mind that many current NFS implementations do not support change notifications either, despite the fact that NFSv4.1 has the necessary provisions. Many SMB/CIFS servers (esp. those in cheap Network-Attached-Storage embedded systems) also have limited to no support.
I have a folder a/ and a remote folder A/.
I now run something like this on a Makefile:
get-music:
rsync -avzru server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.
This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.
In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.
If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).
Thing is I have 3 machines in context:
desktop
notebook
home-server
So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.
I guess this is only possible with a database and track of operations :P
Any simpler solutions?
Thank you.
Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/
Syntax:
unison dirA/ dirB/
Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:
unison -auto dirA/ dirB/
unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).
Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.
Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.
It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.
FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.
On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops.
I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.
You could also try bitpocket: https://github.com/sickill/bitpocket
Try this,
get-music:
rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).
You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.
You could try csync, it is the sync engine under the hood of owncloud.
I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.
I'm now using SparkleShare https://www.sparkleshare.org/
works on mac, linux and windows.
I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.
Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.