We have a Java Application server installed on our client's test server. Within application server, a number of threads access the filesystem in a number of ways (using java.io classes). The application server is started as a service. After a number of hours, sometimes up to a day, the filesystem seems to have become unavailable to these threads. On looking at the folders/directories in question on the filesystem, they are still present, appear to have the correct access permissions and do not appear to have exclusive locks on them. Is this possible? Any suggestions as to clues I should look for? Most likely causes?
The Operating system in question is Windows 2003 (Service Pack 2) with 32 bit processor. Filesystem is NTFS JRE is 1.6
I have already posted this on the serverFault Q&A, but it was suggested I pose the question here instead.
Check that your applications closes streams, such as FileInputStream, correctly.
Related
I was tasked to set up an open-source icap-server to scan files that are uploaded in one of our web applications.
The files base64-encoded withing the upload method and this seems to lead to the fortigate not being able to scan it properly.
Now I managed to set up an icap-server using "c-icap" and tested it with "clamav". So far it seems to work, at least EICAR files are detected.
Now the problem is, that my boss is basically a "Windows-only" person who says that any open-source AV can´t be trusted enough.
He wants me to try to have the requests/uploads sent to the c-icap server, but scanned with Windows Defender. ß Is there any - useful - way to accomplish this? Also, since I am super new to ICAP - are there any restrictions regarding which icap-server can be used?Like, is there any way to force requests to be handled with a metadefender-icap only, and not any other icap server?
Sorry if these questions are a bit unspecific and basic, but just started with ICAP yesterday!
Thank you!
Check out this link: https://techcommunity.microsoft.com/t5/windows-it-pro-blog/metadefender-icap-with-windows-defender-antivirus-world-class/ba-p/800234 to see what Microsoft offers regarding Defender+ICAP.
I have used commercial ICAP solutions in the past (like Bluecoat) that allow plugging in multiple commercial virus engines (Kaspersky, Sophos, ...).
These are appliances, and you don't need to tell your boss their internals are probably some sort of Linux ;-)
You can probably also set up a commercial antivirus solution on your Linux system and use that.
Looking at this link: https://learn.microsoft.com/en-us/windows/security/threat-protection/microsoft-defender-atp/microsoft-defender-atp-linux it may even be possible to run Microsoft's Defender on Linux. I haven't tried it myself though.
Unlike your boss, Microsoft realizes software shouldn't be made to just run on Windows anymore (even MS-SQL Server now runs on Linux).
I'm having an intermittent problem that I'm trying to track down. Every now and then a significant portion of my src directory is being erased (like 90%+ of all files). I'll be working on my project and all of a sudden I'll get an error, look at git status and it will show nearly all of the files in my repo have been deleted. Then I have to run a bunch of git checkout -- commands and I'm lucky if I don't lose a bunch of work.
Can I use inotify or another program to watch my src directory and report which program is deleting the files? I have a feeling it's gulp but I have no evidence beyond the anecdotal, and I don't want to bother a specific project until I've nailed down the source of the problem.
OS X, by the way.
The first thing that comes to mind is to use lsof to monitor your directory and capture your output to a file (or have a terminal up.)
I tested lsof +D ~/Downloads/ -r 2 out on my OS X, and it seems to work fine.
https://unix.stackexchange.com/questions/157064/monitoring-files-continuously-with-lsof
Auditing. This is one thing that auditing is designed for.
Don't roll your own. Don't use tools designed for other purposes. Use the auditing facilities your operating system provides.
Basic tutorial for OS X is here:
OpenBSM auditing on Mac OS X
Way back in 10.3.x, Apple submitted Mac OS X and Mac OS X Server to
the National Information Assurance Partnership for Common Criteria
certification. Common Criteria certification means that the the
covered hardware and software has been tested and evaluated to make
sure that it meets an established set of requirements for security and
data protection. 10.3.6 and 10.3.6 Server were tested and were found
to meet Evaluation Assurance Level 3 (EAL3) for Common Criteria
certification.
As part of that certification effort, a new piece of software appeared
from Apple: the Common Criteria Tools audit software. This software
was OpenBSM, which is an open source implementation of Sun’s Basic
Security Module (BSM) security audit API and file format. ...
Yes, it's a pain to do properly. But it will work, and the results will be definitive.
I have a .NET winform application that requires to register a native dll during installation. My question is how can I register a dll (regsvr32 ABC.dll) as part of MSI installion process? Similary how can I un-register a dll as part of un-installation process?
Nice answer from Chris Painter, adding for reference: how to register DLL's properly in wix 3.9. And one with WiX-focus: Registering COM EXE with WIX.
Self-Registration considered harmful
The proper way to register a COM file is to extract the COM registry information from the file and add to the appropriate family of COM tables in the MSI. Most MSI tools have features to support this COM extraction, see separate section towards the end of the answer for details.
This MSI SDK article lists several variations on the general issues with self registration described below, as well as describing some further details - particularly with regards to per-user registration of COM data, and run-from-source scenarios.
Extracted COM data will ensure reliable installation of your COM server as well as support for advanced MSI features such as "advertisement", "rollback", resiliency and "elevated privileges". You can read more about these advanced MSI benefits in this summary that has become somewhat popular on serverfault.com: corporate benefits of MSI.
It is also possible to use the built-in SelfReg table in Windows installer to register the file using regsvr32.exe as part of the installation process (or even invoked as a custom action), but this is considered bad practice for a number of reasons:
Rollback: Windows Installer is unable to properly handle rollback unless the COM data is extracted and embedded in the MSI. The result is that a failed setup may not clean up its COM footprint correctly and the MSI does not put the machine back in the original state properly. The rollback of COM data really does work like "auto-magic" tracking every change in the registry whether it be addition, modification or deletion and is reliable when done right.
Security: The self registration process of a COM server may in certain cases perform unorthodox tasks such as modifying the system's network settings or perform other crazy maneuvers that have nothing to do with COM and are hard to identify and debug. I have personally seen, in disbelief I might add, COM registration change system-wide network settings without any warning, and for no obvious reason. It might have been just an optimization for an application, but this is rarely acceptable when it changes the whole system affecting all other software. Though an EXE file run in admin mode can do the same and be equally faulty, self-registration may go under the radar and be less obvious as a security issue. This is a core reason why large corporations and Microsoft best practices insist on not allowing self-registration as it may interfere with business critical systems.
Chained dependencies: Some COM files may need to be registered in a specific order on the system to register successfully. In other words file B can't register until file A has been registered. I have honestly never seen this in real life, but it is technically possible, and I have seen dependencies on language dlls (resource only dlls) cause COM extraction to fail. Though slightly different, it is still a dependency issue. MSI does not allow specification of the registration order (probably due to the database origin of MSI, rows are unordered). If you extract the registry data properly on the build computer and put it into the MSI, these chained dependencies will not cause an application error.
Permission problems: Windows Installer has advanced features to elevate the privilege level of the user installing the MSI to allow all information to be registered without permission problems (no messing about with temporary admin rights). If you use the SelfReg table you are more likely to run into registration problems caused by permission or privilege peculiarities on the local system (in my experience this is particularly evident for self-repair operations). Permission problems like these occur more and more as new versions of Windows steadily put new obstacles in place for the successful deployment of software (UAC prompts, self-repair lockdown, impersonation changes etc...).
Resiliency: If another application destroys your COM registry entries, the COM data embedded in your MSI will reinstall the COM component with all associated registry entries via self-repair if proper COM extraction is used to make the package. This means that your application should always be able to launch with its COM servers properly registered. However, this can also trigger the dreaded repetitive sequence of self repair cycles that many experienced computer users have seen (here is a simpler and shorter explanation). In other words COM extraction can be riddled with problems as well, but just using self-registration would leave your application broken, and also prone to security errors being triggered if you run repair, modify or self-repair of your product (the self registration operation may run without elevated rights and hence fail to complete if the repair is run as a restricted user). This means the errors are impossible to fix for most normal users. It is all they know how to do if the product isn't working.
Advertisement: Advertised products are available to the user via shortcuts and registry entries, but not presently installed on the machine. An "on demand" installation can be invoked in a handful of ways - referred to as advertised entry points (recommended Symantec article), one of which is the invocation of an advertised COM server. No install will be triggered unless the file is properly advertised in the registry and a crucial trigger of "self repair" is hence missing if you use self-registration.
Installation Tool Support for COM Registration
The extraction of COM data and entry into MSI tables is a fairly involved task, and most tools on the market such as Installshield, Advanced Installer, and Wise (Wise is now off-market, unfortunately) have automated solutions for this.
In Installshield you simply enable a component flag called "Extract COM data on build", and Wise has a similar flag on the component level. WiX can extract the COM registry data using a tool called heat.exe and the generated WiX-code can be inserted into your source WiX file (there may be new features for this by now that I am not aware of). I am not aware of any features in Visual Studio that will extract the COM data automatically, but it looks like Chris Painter provides a possibility in his answer.
Check out RegSpy2 if Heat doesn't work for you (Phil Wilson - the author of "The Definitive Guide to Windows Installer" wrote RegSpy and someone extended it to RegSpy2). Also check this: Register ActiveX exe server using WiX (my answer towards the bottom for regspy.exe command line use).
Erroneous COM data inserted into an MSI - particularly for repackaged applications in corporate environments - is one of the leading causes of "unexpected cyclical self-repair". Please see this long article for explanation of this issue: How can I determine what causes repeated Windows Installer self-repair? (bullet point 3 in section "Some typical self-repair problem scenarios" describes this issue).
Several other installation tools exist with similar extraction features: What installation product to use? InstallShield, WiX, Wise, Advanced Installer, etc
vsdrfCOMSelfReg is not a best practice. Try vsdrfCOM instead. This will "extract" ( or try, vdproj is a POS sometimes ) the COM metadata from the DLL and author it into the correct COM tables. This is better then hoping an out of process call to DllRegisterServer will work at install time.
Now that MSI is natively aware of your COM resources, it will handle install and uninstall for you.
Scroll down to Rule 19 in the Tao of Windows Installer to see what the MSI team said:
Using the self-registering capabilities of certain DLLs is highly discouraged. Any activity performed by the self-registration (e.g. addition of registry entries) is out of the control of the Installer, so cannot be part of advertisement, repair and is not removed on uninstall. Instead you should have the Installer manage the data for you by using the appropriate tables in the MSI database.
Select the file you want to register and in the Properties window set the Register field to vsdrfCOMSelfReg. This will author an entry in the SelfReg table which will automatically register / un-register your DLL.
http://msdn.microsoft.com/en-us/library/aa371608(VS.85).aspx
I have all my COBOL source code located on a z/OS mainframe. What is a way to migrate all this code to ClearCase?
Rational Developer for System z (RDz) is the tool you should be using for this. It's basically Eclipse with a large number of IBM proprietary plug-ins which give you access to your mainframe data sets, including those under the control of SCLM (the default z/OS source code management system).
You can use RDz to connect to the mainframe and check out your code directly into an Eclipse project. That code can then be added to any other source code management system that has an Eclipse plug-in.
There's more to it of course, such as the ability to kick off mainframe builds from the Eclipse environment, something that will be important since, no matter how hard you look, you probably won't find many distributed platform compilers that can compile mainframe source.
If you just need a one-time move, a file packing tool -- like PKZip/MVS or UnXMIT -- to bundle the source up. You can then transmit it using IND$FILE, ISPF File Transfer or FTP to your clearcase server and check it in.
If you need ongoing updates of your mainframe resources on a server based source control system, you might be better off setting up some shared DASD using samba, NFS or the like between your mainframe and your server.
Unless you plan on doing your development on PCs, I don't think Rational Developer for Z is going to be a good fix. It will do what you need, but the mainframe setup is kind of headache-y and the cost of the product is excessive if all you need is to move resources to/from your clearcase server.
IIRC, RDz costs about 6k per seat. You might spend a few days writing some procs to ftp to/from your clearcase server and check-in/check-out and save some heafty expense. Actually, IBM ought to already have those tools built already. Clearcase supports remote machines doing checkin/checkout, maybe all you need is USS and a TCP/IP connection.
I have a folder a/ and a remote folder A/.
I now run something like this on a Makefile:
get-music:
rsync -avzru server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.
This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.
In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.
If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).
Thing is I have 3 machines in context:
desktop
notebook
home-server
So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.
I guess this is only possible with a database and track of operations :P
Any simpler solutions?
Thank you.
Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/
Syntax:
unison dirA/ dirB/
Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:
unison -auto dirA/ dirB/
unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).
Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.
Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.
It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.
FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.
On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops.
I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.
You could also try bitpocket: https://github.com/sickill/bitpocket
Try this,
get-music:
rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/
put-music:
rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/
sync-music: get-music put-music
I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).
You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.
You could try csync, it is the sync engine under the hood of owncloud.
I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.
I'm now using SparkleShare https://www.sparkleshare.org/
works on mac, linux and windows.
I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.
Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.