I am interested in creating a tool for Hackage involving analysis of the various dependencies between Hackage's packages. I am thinking this would involve downloading and syncing a mirror of all Hackage package sources, in particular the *.cabal files, on which I can run the various analyses I have in mind.
How can I access the same repositories that cabal and stack fetch packages from? Is there an API that cabal and stack use to fetch packages that my program could similarly plug into?
I'm currently working on a library that will give easy (or at least easier) access to this metadata. But the basic rule for getting the metadata on all packages from Hackage is:
Download the 01-index.tar.gz file: https://hackage.haskell.org/01-index.tar.gz
Parse each .cabal file inside of it
Note that, due to Hackage revisions, there may be multiple cabal files with the same name/version combination (known as the package identifier).
Related
I'm a recent CS grad working for a start-up. I've been tasked with adding new features to some software they bought. I've downloaded the software from AWS and have begun trying to get started but I'm having trouble with the front-end, which is built with React and bundled by Webpack.
The download left me with an old version of the front-end and minified files of the new version. I was able to use their source maps to unpack them and get the unbundled files but now I can't figure out how to rebuild it.
The old version is set up to have its dependencies installed and minified by npm. I would like to set up something similar with the new version. I've been able to add all its dependencies to the package.json file but I keep running into errors, mostly stemming from versioning conflicts. What should I do in this situation? Am I just going about this the wrong way?
Software might be a vague term to describe what you have in your hands.
It seems from your wording that you have the source code of an older version, but the production build (bundle.js, main.chunk.js, etc) of some newer version, for which you don't have (or potentially own) the source code.
React is very complex itself, it was probably built using Create React App, dealing with the bundled and ejected files is probably worse, and adding the new features without the source code sounds like a nightmare for a CS grad at a startup. This might even potentially be illegal.
I'm having the same issue as nuget feeds and promotions, eight years later!
In this case I'm talking more generically; we're using ProGet as our package manager, and have nugets, universal packages, and even some docker containers to consider in the package promotion process.
One of the ideas is to have several Nuget feeds; a ci feed where every successful integration publishes a package, a qa feed that you only publish versions you want qa to test and then a release feed, where you copy only packages from the qa feed that they successfully tested.
So, say we have a build in the ci feed that works, it's version 1.2.3-ci-xyz. We want to promote that to the QA feed, without a rebuild, and re-package it as 1.2.3-rc-1. That package passes QA and is ready to be promoted into the prod feed, with no rebuild, and ship to production. It should ship as 1.2.3. (right?)
The question is, if we're not doing any rebuilds, the package binaries will still have the version 1.2.3-ci-xyz. That'll show up anywhere a version is displayed or queried in the app.
And that's where I get stuck. What's the proper pattern here? Does it matter what version is shipped, as long as we know what it is?
meaning, we promote 1.2.3-ci-xyz from lower feeds to higher feeds, without repackaging with different versions?
Wouldn't it be incorrect for package 1.2.3 to include a binary 1.2.3-ci-xyz?
do we always build with the next 3-digit number, and forget about the ci/rc suffix?
I'll share this answer from our internal support channel :)
This is how we (Inedo) typically handle this in our libraries. The short answer is:
We set the Assembly Version to Major.Minor.Patch
We set the Assembly File Version to Major.Minor.Patch.Build
We set the Package version to Major.Minor.Patch-ci.Build (we then repackage to Major.Minor.Patch-rc.Build then to Major.Minor.Patch)
We also use the Assemble Informational Version to display a friendly version (ex: Version 6.0.0 (Build 36-v6))
This allows us to repackage without rebuilding. We also will detect these pre-release dependencies during our product builds and prevent them from being released to production. You can see our ProGet v6 build as an example: ProGet 6.0.0 Build 36.
The longer answer I feel is answered pretty well in our blog post Best Practices for Versioning NuGet Packages in the Enterprise.
I can query available packages w/ nix-env -qa [package] but how can I look for optional packages (e.g. libraries) that depend on a primary package and can be loaded or installed separately? Example: Coq (coq-8.6) has packages coqPackages_8_6.ssreflect and coqPackages_8_6.mathcomp that I can get no information about in Nix AFAIK
nix search can also be used to search for packages. Caching all available packages is slow, but searching the cache is quite fast! Check nix search --help!
I personally never use nix-env -qa because it is very slow.
Instead, when I'm looking for a top-level package (an application), I use http://nixos.org/nixos/packages.html.
When (like in your question), I'm looking for a non-top-level package, I use the auto-completion of nix repl. Run it with
nix repl '<nixpkgs>'
Now type for instance coqPackages and use auto-complete to see all the available versions. Type coqPackages_8_6. and use auto-complete to see all the available packages in this set.
I have been searching for a way to obtain a timestamp for when a package was either released for general use or possibly when first loaded on a local repository. Something in either Shell or Python would be ideal, but I'm open to other options at this point. I know packages support a changelog, but it looks like not all packages include a release date.
Thanks!
The answer depends on what exactly you are looking for, and it's not clear from the question. Before reproducible builds were introduced, the date a package was built could be retrieved from the raw ar members such as:
ar tv pkgname_version_arch.deb
If you are looking for the date the package got accepted/uploaded into a specific repository, then the answer will depend on what repository and the software used to manage it. For Debian you can get the information from UDD, from the debian-devel-changes mailing list for the maintainer uploads (but not the buildd uploads, or from package tracker, other derivatives and distributions might have different interfaces or none at all. For other private repositories perhaps there are publicly accessible logs available.
As you mention the changelog can be used for when the source package was prepared, but that might be wildly different to when it got built or even uploaded.
If a Chef recipe (or any of it's cookbook dependencies) use the package resource without specifying a version, then the latest version of the package is installed. If you want to control and test exactly what you are installing, then you must always supply the package version. What can you do when the cookbooks that you depend on do not take the same precautions?
See for example the default recipe in the ark cookbook. If this recipe is used on a production server, it could install packages that have not been tested. This is just one example (with over 5m downloads) so I am wondering how people are getting around this problem.
What can you do when the cookbooks that you depend on do not take the same precautions?
I don't think there is a simple answer. This is basically "the Chef way" ...
(Actually, I would suggest that hard-wiring package versions could do more harm than good. One of the good things about using a (good) distribution's package repo is that they regularly release updates with patches for security issues and bugs. But if you wired fixed package versions into your recipes or roles/nodes or something, you would prevent any such patches from propagating to your system.)
However, if it is critical to you that package versions are stable then maybe ...
Clone and hack the cookbooks in question to use specific versions. (Actually, you probably need to do this anyway, to avoid being bitten by unstable cookbooks!)
Use a distro (such as RHEL or its "clones") that values long-term stability, and only pushes out package updates that are "really important".
Create your own private mirror of the distro's package repos with only the "good" versions of the critical packages in it.
Modify Chef so that the package resources pick/install specified versions by default. (I don't imagine this would be easy. But if you did come up with a good solution at this level, it would be a pretty useful addition to Chef! IMO.)
UPDATE
Actually, there is a way to do this for (at least) Debian-based systems; see the apt cookbook and in particular the references to "pinning".
Or with yum, you could "lock" particular versions using "yum versionlock ..." as described here: https://www.zulius.com/how-to/yum-install-specific-package-version/
UPDATE - 2
Another possible trick would be to "inject" a version attribute into the "unsafe" package resources. Something like this:
# first, include_recipe a recipe that specifies 'package "foo"' without
# a version attribute
# then ...
r = resources("package[foo]")
r.variables['version'] = "1.2.3"
With a little ingenuity, one could create a "package version lock" recipe that pulled the versions from a databag, and dealt with missing resource exceptions and version attributes that were actually provided. But I don't know if this is "A Good Idea" (tm).
Chef's package resource uses the package manager of the node's operating system (like apt, yum, etc). These tools always install the most recent version that is available through the repositories. That's why chef's package resource also installs this version.
What the ark cookbook is that it downloads the source code and then compiles it - obvious that you can specify the version to install (through the passed URL).
So it depends on your actual need. If you want to install the version that is available through the distro's or your own package repo, then it's totally fine (and that's what most cookbooks do). If you want to compile everything from source (where you usually have the option to specify the version, the coverage of chef cookbooks supporting this is lower.
Personally, I'd suggest that you set up an own apt/yum/whatever repo for the software for which have specific version requirements.
In short : I'm not managing this.
In a more complete answer:
All distro/release go throught a validation phase before releasing new packages, I'm confident over it and it helps me keep in sync with security fixes.
As far as I know all package managers takes care of not upgrading a package in a breaking way if it is a dependencies of a package installed manually, again you have to trust the package maintainer about this.
i.e.: the package ressource without version won't update make nor gcc if it is a dependency of one package you installed with a fixed version.
For exemple under ubuntu, if you set the nagios package to manual, it will never try to update the libc package over a breaking change, and so I could break other package isntallation as dependencies are not satisfied.
If you're absolutely concerned about it, you have some choices:
Rewrite each pacakge ressource to use fixed version of packages
Fork any cookbook to fix thooses issues (you can write a foodcritic rule to help you detect package ressources without specific versions)
Have your own repos, stable and testing, and move packages inside stable repo once tested and use testing repo on your staging/QA environement.
The 3 is the most conservative as you choose what is in the repo in the stable branch and it won't change magically. The drawback is security fixes you'll have to manage.
Hope it would help.
We had the same issue in our cookbook. So we decided to use data bags.
Data bags can be easily changed, for example:
knife data bag from file my_data_bag host1
OR
knife data bag edit my_data_bag host1
Your recipe will be able to see the specified version from the data bag using the code like this:
my_bag = data_bag_item('my_data_bag', 'host1')
Chef::Log.info("You have changed the version to: #{my_bag['version']}")
package 'java' do
version my_bag['version']
action :install
end
So finally you don't need to modify Cookbook or Recipe. All you need is to pass the version to the data bag.