Why no good extN drivers for Windows? - filesystems

Why are there no good drivers for Windows for reading ext2/3/4 filesystems? Googling around indicates that there's 2 or 3 out there, but all of them have problems. Is there some technical inconsistency that makes it difficult to correctly code up something that would enable me to open up My Computer and work with an extN partition just like NTFS or FAT? I thought one of the benefits of open sources and standards was that problems like this should be solved fairly quickly.

Driver signing.
Microsoft's driver signing is by its own nature incompatible with the GPL and unsigned drivers don't work anymore.

I haven't used it myself, but a coworker of mine has used Ext2 IFS for Windows without any problems.
One of the benefits of open sources and standards is that problems like this can be solved fairly quickly. If no one is sufficiently motivated to work on a problem - whether that motivation comes from money, personal need, fame, whatever - then the problem is unlikely to get solved. (The closed source world is no different.) It probably doesn't help that relatively few open source developers have experience hacking on Windows kernel mode device drivers. Writing device drivers is a specialized skill. There are developers who understand the ext2/3/4 code very well and are very willing to work on it, but odds are that the the people experienced enough at hacking on the Linux kernel to work on the ext2/3/4 drivers are probably primarily Linux users (and so don't much care about writing drivers for Windows).
With regards to driver signing: It's my understanding that, starting with Windows Vista, Microsoft doesn't have to sign or certify your drivers for them to be installed without warnings, but you do need a code signing certificate. These are somewhere in the neighborhood of $400 - $500 a year (see Verisign's web site, for example), and most non-commercial developers aren't interested in paying out that kind of money. There are methods for disabling driver signing requirements, but none of them are something the average user is likely to try, which would hinder the acceptance of a non-signed driver.
I don't know how the Ext2 IFS for Windows handles it; either its author got a certificate somehow, or it requires that you disable the driver signing requirements.
So, to summarize, the best ext2/3/4 developers probably don't have much need for Windows, and driver signing discourages would-be open source driver developers for Windows, and the availability of NTFS for Linux means that you can use NTFS instead of ext2/3/4 to share data between Linux and Windows. These three factors work together to remove a lot of the interest in developing ext2/3/4 for Windows.

Related

Creating a touch screen driver for OS X: where to start?

OK, so I recently purchased an Acer T232HL touch screen display to hook up to my Macbook Pro as a secondary monitor. To give you an idea, here's my setup.
OS X doesn't support this monitor in any way, so as you can see in the screenshot I'm actually running Windows 8 through VMware, which proxies the USB connection to Windows perfectly where the touch events are supported. But obviously this isn't ideal.
There's at least one 3rd party driver for OS X that looked sort of promising, but it doesn't seem to support multitouch from this device, it's expensive, and generally was a pain to get working to the small degree it was. There's also mt4j but best I could tell after running their examples, it doesn't support this device at all.
So here's my question: what exactly am I in for if I wanted to write a driver for this thing? I'm mostly a web developer with years of experience with Ruby, Objective-C (and a little C), Javascript, etc. I have never ventured into any kind of hardware programming, so from the surface this feels like an interesting while intimidating challenge.
I know at some level I need to read data from USB. I know this will probably mean trying to reverse engineer whatever protocol they're using for the touch events (is it possible this will be entirely custom?). However I haven't got a clue where to start - would this be a kernel extension? In C, I presume? Would love a high level overview of the moving parts involved here.
Ultimately I want to use the touch screen to drive a specialized web interface (running in Chrome), so ideally I could proxy the touch events straight to Chrome without the OS actually moving the mouse cursor to the touch location (so have the UI behave just as it would on an iPad), but regardless of whether this is technically possible, I'd love to start with just getting something working.
You're going to want to start with Apple's I/O Kit documentation. You can hope that the touchscreen isn't completely custom, though there must be some part that's not standard USB HID, or it would work already. If there are any linux (or other open source) drivers available, you'll have the advantage that somebody already did some of the reverse engineering for you. As an alternative to the I/O Kit, you might also want to look into libusb, which might make your brain hurt less when getting started. If you end up needing to write a kext, that might not help you anymore, though.
As to some of your specific questions:
would this be a kernel extension?
Maybe, maybe not. I'm not really up on the Mac OS X driver situation, but I did write some totally user-space USB code for OS X many years ago. Maybe you'll be as lucky.
In C, I presume?
Probably. I/O Kit itself is written in a subset of C++, so you can probably use that too, if you prefer.
is it possible this will be entirely custom?
Unfortunately, yes, it's possible.
Good luck!

Is 9P obsolete?

I'm interested in studying the 9P FS, currently been reading the source available from these implementations: http://9p.cat-v.org/implementations
Is 9P obsolete? Are you using it for some application?
(also I've found this, some perfomance test between 9P and NFS: http://graverobbers.blogspot.com/2007/08/v9fs-performance-versus-nfs.html)
No, 9P isn't obsolete; I don't know of a protocol that does what it does and is clean and well defined enough to be implemented correctly in almost any language that exists.
9P is used in a variety of systems. A couple of recent uses in arm-js (an ARM emulator) and 9webdraw (a GSoC project that implements the Plan 9 /dev/draw). Both are HTML5 Javascript implementations.
Just to add a bit, both the Linux client implementation and several servers are under active development, so I'd say that's a pretty clear sign that folks still have use for it. One of the areas its seen heavy use more recently is the virtio-9P (aka virtfs) which is part of qemu/kvm and can be used for direct guest to host file access. It's also been used in several experimental operating systems projects (Libra, PROSE, FusedOS) and incorporated into other operating systems (BSD, MacOSX, Windows, Linux) and hypervisors (in addition to the KVM instance above, its also been incorporated in various ways into Xen). 9P is actually being used in supercomputing deployments (both for Plan 9 and Linux, see the diod project on Sourceforge).
I think the reason is that the protocol is quite simple, so implementations also tend to be quite simple and easy to integrate elsewhere (there are several applications both inside and outside the Plan 9 world which use 9P as an interface to the application, in much the same way that some web developers use RESTful interfaces).
The protocol has a couple of different variations including the 9P.L variant which was developed specifically to match the Linux VFS API better. It adds a bit of complexity to the protocol in the addition of operations, but removes some of the complexity of mapping Linux VFS API -> 9P and vice versa.
It is used in Erlang-on-Xen both as a storage protocol for goofs http://erlangonxen.org/blog/goofs-simple-filesystem
It is the way erlang on xen instances in other ways too, see here:
http://erlangonxen.org/more/9p2000e
Also, it's used by libvirt stuff with QEMU.
http://wiki.qemu.org/Documentation/9psetup
9p, to me, is like the Scheme of network protocols. For the most part, it is very simple, but people see need to extend it to fit their environments. Luckily this is done in ways that are often backwards compatible.
In addition to everything mentioned in the other answers, Microsoft is using 9P as part of their Windows Subsystem for Linux.
They add a 9P server to each Linux distribution that is running as a guest, so that Windows can mount the Linux filesystem over 9P, and Windows processes can transparently access the files on Linux's ext4 partition.

Writing an EXT4 file system in C?

This may sound noobish, especially as I'm ( as you may have guessed ) trying to write an Operating System. At the moment I'm stuck on trying to make a file system.
What I want is a similar file system as Linux Ubuntu which is EXT4 ( at least mine is ). I want to try and also either write it in C.
Any idea's on how I can go about this? And/or any tutorials that you might have found that may help me ( I have tried searching with no luck ) :L
Thanks in advance!
Jamie.
Really smart and experienced people who have studied this problem extensively have made bugs that ate users' data. The difference between a bug in the computation layer (e.g. a kernel crash) and a bug in the storage layer is that silently eating users' data is very bad - much worse than giving wrong answers in spreadsheets (excel is buggy yet popular) or intermittently sigfaulting while preserving data on disk (this is easily mitigated by frequent autosave).
Start by studying simpler designs, like the minix filesystems from the old operating systems book [1] (the same one Linus Torvalds started with, twenty years ago).
Like others said, ext2 without journaling, extents or ACLs is a better starting point than ext4. The source code for it is in the Linux kernel and in the e2fsprogs userspace tools package[2]. The format is well documented.
As for tutorials, consider who makes them and why they spend effort on this task. Tutorials are generally made by stakeholders in platforms to bring in new people to develop using that platform, to use the network effect to grow the platform and profit from being already-established actors in a larger ecosystem.
Do you see a business model in growing the number of people who implement their own incompatible buggy[3] file systems? Only if you sell software engineering degrees. So Microsoft only writes tutorials on how to use NTFS, not on how to implement it. Same for Sun and ZFS, Red Hat and Google with EXT2/3/4, SGI with XFS, IBM with JFS, Oracle with BTRFS, etc.
If you want education instead of training, you need to read books and study smart peoples' code they use in production, not look for tutorials.
http://en.wikipedia.org/wiki/Operating_Systems:_Design_and_Implementation
http://e2fsprogs.sourceforge.net/
How much use, in how many different use pattern with it see? Consider bugs discovered in production FSs after years of use on millions of computers. It is unlikely your code will be less buggy, even if you're as smart as Matthew Dillon.
Try looking at an existing implementation, like the one in Linux.

Moving from VMS to Unix

Once upon a time, a team of guys sat down and wrote an application in C, running on VMS on a VAX. It was a rather important undertaking and runs a reasonably important back-end operation at LargeCo. This whole shebang works so well that twenty-five years later it's still chugging along and doing it's thing.
Time passes and people retire and it so happens that the Last Man Standing has turned over the keys to a new generation who - we might imagine - are less than thrilled to find themselves caretakers of a system old enough to be their younger brother. Yet, as underwhelmed as they are by the idea of dealing with Ultra Legacy Systems, they can't justify the cost of replacing the venerable application.
LMS discovered that I habla unix and put this question to me. And since I habla unix but don't speak the C I shall summarize and put it to you. Long Story Short:
LMS wants to port LegacyApp, written in C. from VMS to unix. Resources? Any books he can read? People he can talk to?
The first question I'd need to ask is why, and I'd be leading the conversation in the direction of "Do you really need to port it off of VMS". There are a number of things worth mentioning about VMS:
-> VMS is still actively developed and maintained by HP. They just release V8.4 for Field Test last week (see http://h71000.www7.hp.com/openvmsft/).
-> VMS is available on new hardware; specifically HP's Integrity servers based on the Itanium processor.
-> VMS is also available on virtual platforms via the Charon Emulation products.
-> Popular estimates are that there are about 300,000 VMS systems still in active use today. LMS may be the last man at LargeCo, but he's far from the last man standing worldwide.
-> Lots of information out there, see openvms.org for example, to see lots of current information on VMS, all from current users.
OK - you still want to port off of VMS. How do you do it? Well, it depends on lots of stuff.
-> As others have said, how standard is the code? Chances are, not very. The more VMS-isms, the more difficult the job. 'nuff said.
-> What is the database? If it's Oracle, probably not too tough to move to Oracle on some other platform. If it's some sort of custom DB based on RMS index files, then you've got more work to do, you'll need to re-create that pseudo DB, or, understand it enough to replace it with some relational DB.
-> Besides C, what else is used to create the application? What's on the front end? DECforms? FMS? Is there a transaction engine, e.g. ACMS? RTR? These things will have a huge impact on the feasibility and effort required to port to UNIX.
-> What other products are involved? Are there any 3rd party libraries being used? Are there 3rd party products in use that are critical to the application or functionality?
-> Is this system clustered? If so why? You'll need to meet those same goals with the UNIX box.
-> There are companies out there that will help you do it, and claim to have tools to make it easier, but my experience is that these companies tend to be selling you more services than products (i.e. you need to hire them to use the tools. It'll be expensive).
The book UNIX for OpenVMS Users will give the VMS novice some help in understanding VMS, but, as the title says, the book is really intended for the opposite purpose.
Everything written on VMS uses lots of VMS specific stuff it was just so convenient.
There are a few companies that sell compatibility libs to make the port easier - they wont be cheap though, VMS tended to be used where reliability mattered more than cost.
The other option is to run openVMS on some modern hardware, possibly in a VM.
I am sure Brian has made his decision by now, but for my sins of working for many years in DEC OpenVMS language support (yes, some people had this dubious honour) the real question I would have asked a customer such as Brian is: is it a real-time application or not? If it is the former, then it would be heavily dependent on many VMS system services which would rule out a 'port' and indicate a re-write. If it were the latter then the frequency of VMS system services should (possibly) be limited and make a port viable.
The acid test for me, would be to SEARCH *.c "SYS$", "LIB$" i.e. to search all of the C source files for "SYS$" and "LIB$" tags which prefix VMS system services. If the count for these are in the 10s then a port is probably likely, between 10 and 100 makes it possibly likely, but over a 100 makes a successful port highly unlikely.
Hope this helps
You have several choices.
Get the OpenVMS source, and continue to maintain Open VMS as if it were a Linux distribution. Some folks don't mind keeping up with Linux distributions and OpenVMS distributions. It can be done.
Try to recompile the VMS C into Linux. This can be trivial if the C used only standard libraries. This can be very, very difficult if the C used a lot of VMS libraries.
Once you have facts at your fingertips, you can reevaluate this course of action. Since you didn't list a bunch of VMS library methods this program uses, it's impossible to tell how entangled it is with the OS.
This may be trivial or impossible. It's difficult to tell without analysis of the source.
Write bridge libraries from VMS to Linux. If your program only does a few VMS things, this isn't very difficult. If your program does extensive VMS things, this is craziness.
The bridge -- in the long run -- is a terrible idea. Managers love it, however.
An alternative is to replace the VMS library calls with proper, portable Linux calls rather than write bridges. This is better in the long run, because it excises the non-portable features of the program.
Rewrite it from scratch in Python. That is usually simpler than trying to port the C code. It will be shorter, cleaner, simpler, and portable.
If you're willing to keep running VMS in a VM, you can look into CHARON-VAX ( http://www.charon-vax.com/ ). As previously mentioned, the ease of porting really depends a lot on how much of the VMS extensions were used; searching the source code for $ characters embedded in strings (usually with a 3-character leading substring, such as lib$gettime or dsc$descriptor or sys$foobar etc) will give you at least a basic idea of what VMS system functions are called and how likely they are to be portable, if the name is reasonably obvious.
If it ain't broke, don't fix it! Why port it or migrate the app if you don't have to? Why not run it on a current install of OpenVMS running on an HP Itanium server; that is assuming you wish to upgrade the hardware, which may not even be necessary if your VAX hardware is still running strong.
To learn C, you might as well drag it from the horse's mouth: "The C Programming Language" by its inventors, Kernighan and Ritchie.
I can recommend "The UNIX programming environment" by (again) Brian Kernighan; a more authoritative source you'll hardly find, and it teaches you both Unix/C idioms and a bit of C programming at the same time.
For more depth and detail on C, I heartily enjoyed a book by Peter van der Linden: "Expert C Programming - Deep C Secrets".
You'll also want to wrestle LMS for a library documentation of VMS-specific C functions with (of course) special emphasis on those actually used in the app. That's where your porting effort will be.
The job could be easy or difficult, depending on how much machine-specific cleverness and bit-twiddling is done, and how many VMS-specific system calls are used. It would be very good if word size was equal (in other words, if your VMS box has a word size of 32 bits, don't run the code on a 64 bit version of Unix!)
Brian, I'm not sure if LMS specified/cared to port C-code or the WHOLE process. As too often people think of languages out of scope of systems.
If there're was a process built on VMS, most likely it used at least scheduling/batch facilities, which are often scripted in DCL (rather simple and clear language, unlike shell or perl scripting).
So the cost of porting the whole process may be higher than originally perceived by your LMS. Add here the reliability aspect, given your crunches with C, which is nothing impossible, of course, with enthusiasm and determination.
If you want simply give the C-code a try, as previously posted, search it for the "$" hits. Or just cc it with all headers present, the basics of compile-link command should be enough.
Alternatively, this looks like a consultant's call, as indeed such jobs were abundant at the "exodus" time. All said VMS remains quite a robust platform (24x7 is a norm!), unless the harware dies, then there're still tons of "exodus" spares. GOOD LUCK!
About a year and a half later, maybe you've already figured out what to do. My organization has recently decided to stick with OpenVMS instead of switching to Linux even though the old guard recently left. We just couldn't argue with what we felt was a very stable and reliable system. We are currently switching from Alpha servers to Integrity servers for end of life reasons. HP has been very helpful with our transition.
For that matter, there may be Linux vendors out there who can help with the transition. Ask your new hardware vendor if they have any recommendations.
Depending on what languages you already know, C is not that hard to learn. I taught myself C in the course of learning C++ after finally prying myself loose from Pascal.
(VAX Pascal, plus Rdb/VMS, plus DCL formed a combination that was hard to beat.)
If the software is typical C, you'll spend more time learning the library functions than learning the language.
It's pretty lightweight stuff, but I went through the online tutorials for C++ that Microsoft makes available in conjunction with the express edition of Visual Studio for C++.
Here's the beginner's tutorial:
http://msdn.microsoft.com/en-us/beginner/cc305129.aspx
It's probably worth making the effort to ask why LMS wants to port the application to Unix. The answer may seem obvious, but properly exploring the reasons has its benefits. I would assume:
OpenVMS is an "ultra legacy platform", and for that reason alone is something that is not worth running an application on anymore;
It's tough to find anyone who is willing to maintain an application that runs on OpenVMS these days;
The hardware on-which OpenVMS runs is threatening to become moribund.
We have a similar challenge, but in our case the application in question not only runs on OpenVMS but is also written in COBOL. I would have to say that your situation is rosy in comparison given that your application is written in a cross-platform language.
In any case, I think if you're about to make a big decision like moving from OpenVMS to Unix it would be prudent to do a little due diligence. In your case, try to assess just how portable the code is--only then will you know what the scale of the effort is (worst case could quite easily be a multiple of best case). In C, code portability is mostly a function of the dependencies--are they "standard" or are they VMS-specific?
Our enquiries revealed that HP would be supporting OpenVMS on Itanium until at least 2022. There isn't necessarily a need to rush to another platform--perhaps you could keep things on OpenVMS whilst embarking on an effort to prepare the application for porting (make it less dependent on OpenVMS specifics).
VMS has a surprisingly healthy community and if it's the lack of Unix that's the issue, then maybe GNV could help bridge the gap?
Well u have a few options. if this code needs to be ported rather quickly, i would write a bridge library to emulate the vms libs. whener you get it back up and running on a *nix, then go through replacing the vms library calls with native/portable calls for *nix.
Also if there is a lot of optimizations in the code ie inline assembly and bit twiddling. then you will have to rewrite thi code, which will take an understanding of the VAX arch. also. be sure to check word size differences and endian differences

What is a good filesystem for embedded NAND drives?

I am working on an embedded application that uses NAND flash for storage.
As it looks now, we won't use Linux or any other RTOS. The application must handle unexpected power downs.
We have been looking on different file system solutions, including YAFFS2, JFFS2, FAT+FTL as well as solutions from HCC Embedded.
I have heard FAT+FTL is a normal choice, but I am worried about data loss in case of unexpected power downs as well as performance. Would be grateful if anyone could share insights and experience about this
FAT-FTL is a "normal choice", but not necessarily a good choice.
YAFFS2 is newer than JFFS2 and seems to be faster and more scalable for large NANDs. This presentation of the "Embedded Linux Conference Europe" compares these two and other flash file systems.
Another solution is LogFS (note: log stands here for logorithmic, not for log-structured). It should also be more scalable, but I have no idea how muture it is.
There's UBIFS. The only implementation I know of is in the Linux kernel, and it depends on the Linux kernel's UBI interface. However, the fundamental algorithms should be implementable without too much trouble in whatever environment you are using. As for production-readiness, Nokia uses UBIFS in their N900 smartphone, and plug computers based on the SheevaPlug have support for it, too. I have found the Linux implementation to be reliable, even on flaky hardware that likes to reset itself at random. Unlike JFFS2, UBIFS does not need to read the entire filesystem at startup.
You may want to reconsider your "no Linux" decision, since using Linux would make it a lot easier to use UBIFS.
There is a commercial vendor called DataLight which offers a robust flash file system, but of course it's not free.
They have an interesting white paper (PDF) (take with a grain of salt) on performances.

Resources