proj-jasper-jasperPackagingNotes1

https://news.ycombinator.com/item?id=7802005

--

lbotos 15 hours ago

link

Current Go users, What's the state of package versioning right now? Is vendorization still the answer?

reply

dayjah 12 hours ago

link

We use Godep, it is very good. As per another answer to your question: with '-copy=false' it behaves a lot like bundler.lock. Having spent a lot of time working with it we've found a few areas where you can get burned a little; particularly if you've structured your repos as a set of libraries, as seems to be the encouraged golang pattern.

When you have multiple libraries you have to be very specific about when you run godep, lest you find yourself with two libraries needing different versions of a common library, for example Main imports Foo and Bar, which both import Baz. Godep provides a mechanism for handling this: each dependency is explicitly locked into a fixed revision (e.g. commit sha, in the case of git). The pain comes about when during debugging as it can be very hard to reason which version of a library you're using.

Additionally the revision aspect is also a bit of a PITA, we use a development flow which rebases our small commits into a big commit and then merges that into our master branch; if you ran godep prior to that you're now referencing a commit that no longer exists. Given the chain of references that can exist this can go a very long way down. This same pattern also forces you into needing to push your dev branches to an origin server, as godep checks out the repos during the build, which while pretty benign a concern is a PITA if you forget and your build breaks because of it.

We're strongly considering moving to "one big repo" to help combat this issue (as well as a few others) for our internal golang repositories. Referencing "published commits" in 3rd party libraries is an acceptable level of pain. We're not entirely sold on this yet... just considering it.

reply

leef 15 hours ago

link

No need to vendor. Use Godep without copying (godep save -copy=false) to create the equivalent of a bundler.lock file and check that into source.

reply

AYBABTME 15 hours ago

link

There are other ways around, but I'd say the community is solidifying towards godep. Someone correct me if I'm wrong.

reply

chimeracoder 1 hour ago

link

There seem to be a lot of comments here recommending godep, but just to throw my experience in: none of the projects I've interacted with use godep (other than the Heroku buildpack, which was written by the author of Godep).

It seems to be a solution for some (not all) projects that are released in binary form, but that isn't relevant to most projects out there[0]. I have never felt the need for what godep provides; vendoring myself has been sufficient for the (very rare) case in which I need specific versions of dependencies other than tip/trunk.

I asked around on #go-nuts, and (though the sample size was small), the other regular contributors who idle in the channel seemed to have the same experience.

YMMV obviously.

[0] https://botbot.me/freenode/go-nuts/2014-06-19/?msg=16563763&...

reply

---

---

mimog 1 day ago

link

I like how it says "Rock-Solid Ecosystem", yet I have had the exact opposite experience trying to install even the most basic things with Cabal. I still can't get the Sublime text haskell plugin to work due to a dependency that fails to compile.

reply

rhizome31 1 day ago

link

I had a similar experience with Cabal. On one computer I haven't been able to install Yesod whereas on another one, it finally worked after I had wiped my ~/.cabal. It gave me the impression that Cabal's dependency resolution mechanism is still a bit britle.

Also I found that installing stuff through Cabal was pretty slow. It's probably partly because Haskell libraries tend to be kept narrow in scope so it's necessary to install a lot of small packages to get a piece of functionality (take for instance the dependency list of Aeson, which seems to be the recommended choice for working with JSON : http://hackage.haskell.org/package/aeson ). Another reason is that Cabal compiles Haskell code into native code.

reply

---

---

dbaupp 3 days ago

link

As others have said, having a package management system that deeply understands the language and tooling is awesome. Examples:

I'm sure all of this is possible with other systems, but it seems unlikely to be so nice to use.

reply

seabee 2 days ago

link

It's very similar to Racket, and yes, it is nice to use!

Other systems can get you much of the way there (node, Python are the only ones I'm really familiar with) but I suspect you need a little language help to achieve the same kind of convenience.

reply

---

pestaa 3 days ago

link

Curious to hear more about language-specific (though OS-agnostic!) package management systems. IMO composer is the best thing ever happened to PHP, Ruby gems are huge, Python eggs also make a very useful ecosystem.

OpenSUSE?'s Open Build System would be great to ship independent packages, but those are again heavily tied to Unices, hence leaving other platforms behind.

reply

JoshTriplett? 3 days ago

link

> Curious to hear more about language-specific (though OS-agnostic!) package management systems.

As far as I can tell, one of the main justifications for most language package management systems is "we also run on Windows/OSX, which has no package management, so we'll invent our own". As a result, users of systems that do have sane package management get stuck with multiple package management systems, one for the distro and one for every language. Even then, I find it disappointing that nobody has built a cross-platform package management system for arbitrary languages to unify those efforts.

reply

smacktoward 3 days ago

link

The other justification is generally a clash of cultures: the people who maintain distro/OS package managers generally come out of the culture of sysadmins, who value stability over feature-richness, while the people working the language communities generally come out of the culture of developers, whose priorities are the exact opposite.

When languages try to hook into existing OS-level systems, the people on the language end get frustrated by the way the people on the package-manager end don't hurry to rush out bleeding-edge versions of packages the second they hit Github. To the package-manager people, that's no big deal, their orientation towards stability and predictability makes them comfortable with waiting a little for the coffee to cool. But to the developers, who want to get their hands on the Latest and Greatest Right Now!, it feels like slogging though molasses.

So the developers eventually end up blowing their stacks and stomping off yelling "Yeah? Well fine, we'll build our own package manager then! With blackjack! And hookers!"

reply

DennisP? 3 days ago

link

Maybe OS-level package managers should default to stable, but let the user check a box to get the latest and greatest. Developers want a stable system like everyone else, but for the stuff we're hacking on, we have a legitimate need to get the most recent, so our software isn't obsolete by the time we finish it.

reply

djur 3 days ago

link

Most OS-level package managers also aren't designed to install more than one version of a package at a time. They don't tend to integrate with build systems as well, either.

reply

mercurial 2 days ago

link

That's not so simple. A distro is a fine-tuned collections of packages which work more or less well together. Debian, for instance, comes in stable/testing/unstable/experimental flavours, depending on how daring you are. But even this isn't a universal solution. If you are deploying for instance a web application, you will want to deploy a locked down number of dependencies as well, regardless of what is present on the target system. And you may need deploy multiple applications side by side. Few system package managers have an answer to this.

reply

DennisP? 2 days ago

link

So developers end up installing later versions manually. And in many cases it's no big deal. If the distro has Julia 0.2.1 and Emacs 23, I can upgrade to Julia 0.3 and Emacs 24 and it's not likely to damage anything. It'd just be nice if I could do it with the package manager instead.

But just because I'm doing that doesn't necessarily mean I want, say, the latest unstable version of the window manager.

reply

mercurial 2 days ago

link

Debian will let you do that. You can run, say, your machine on testing but get the latest Firefox from experimental if you want. This may, however, upgrade other dependencies on your system, but it's pretty much unavoidable.

reply

yxhuvud 2 days ago

link

I'd be happy enough if the OS-level packagers stopped modifying the package-level packages they packaged.

reply

kungfooguru 3 days ago

link

The problem with language package management systems is they've been used for installing user facing software. As a developer tool I think it is the perfect way to go.

And you should add Linux to your Windows/OSX as being an issue, which Linux package management tool would you build packages for? All of them?

The end user package management provided by the OS should be for installing end user packages and the language tool for installing and publishing libraries and dev tools.

reply

wycats 3 days ago

link

> The problem with language package management systems is they've been used for installing user facing software. As a developer tool I think it is the perfect way to go.

Precisely so.

reply

pjmlp 3 days ago

link

> As a result, users of systems that do have sane package management

Given the diversity of OS in the IT landscape, which systems are those?

reply

stefantalpalaru 3 days ago

link

I met only one package manager that I don't need to fight in order to get what I want: Gentoo's Portage. With a local overlay and language specific functionality concentrated in eclasses it's trivial to add new packages, do version bumps, have fine grained control over installed versions, enabled features, etc.

reply

kryptiskt 3 days ago

link

The distro only contains a small selection of the packages (even if there are hundreds or thousands of them) and the language package system is usually the source the distro maintainers use to find the packages anyway.

reply

yla92 3 days ago

link

Lately, IMHO, Gradle in Android Development(applicable for Java development as well) is a huge improvement over managing dependencies with pom.xml(ant) and linking jar files manually. Besides, you can totally customize build.gralde too.

reply

dscrd 2 days ago

link

>Disappointing to see yet another language-specific package management system (Cargo), though.

As a packager in a Linux distro, I'm disappointed every time somebody tries to cram in PL-specific packages inside distro packages.

reply

pjmlp 3 days ago

link

> Disappointing to see yet another language-specific package management system (Cargo), though

So what is the solution to have portable packages for:

reply

adrusi 3 days ago

link

The goal of the [nix](http://nixos.org/) project is to solve this, and every time anyone brings up a package manager on HN, someone has to mention nix. The reality is that nix is really nice, but isn't any better than making a new package manager until it has wide adoption, so no one is using it.

reply

steveklabnik 3 days ago

link

Nix was brought up during the discussion that led to Cargo, but no Windows support is a deal breaker.

reply

pmahoney 3 days ago

link

I would probably make the same decision, but I hope in the end Caro is easy to wrap with Nix, which is a breath of fresh air particularly when needing to mix dependencies that cross language boundaries and share those build recipes with a team.

Previously, I wrote shell scripts and worried whether everyone on the team had rsync installed, or xmlstarlet, or some other less common tool. Now I wrap those scripts in a Nix package that explicitly depends on all those and distribute with confidence. It's fantastic.

Bundler and rubygems, for example, do various things that make good support within Nix rough. Two examples: 1. rubygems has no standard way of declaring dependencies on C libraries; 2. as far as I know there is no way to ask Bundler to resolve dependencies, create a Gemfile.lock, but not install any gems (I realize github gems must be downloaded to see the gemspec...)

reply

steveklabnik 3 days ago

link

Cargo has the second, and there's a plan for the first.

That said, the reason that you want it to do the installation is that a lockfile is supposed to represent the way to do a build successfully. Without building everything, you can't actually be sure that the lockfile is correct. In theory, it should be...

reply

pmahoney 3 days ago

link

> reason that you want it to do the installation is that a lockfile is supposed to represent the way to do a build successfully

Sure, and I'd like to do that build within Nix (and someone else might want to do it with another packager), which gives a stronger guarantee than Bundler since it incorporates C library dependencies and more. Anyway, the specifics aren't relevant to this discussion, and it seems you have a grasp of the issues, so carry on!

reply

derefr 3 days ago

link

Wouldn't it still have been less effort to port Nix to Windows, than to write an entirely new package manager and then port it to every OS?

reply

steveklabnik 3 days ago

link

If that were the only downside, possibly. I don't really do Windows development, so I can't tell you how difficult porting Nix would be. There's large advantage to having a packaging system that knows your language well. It's going to have tighter integration than a generic one ever could.

reply

pjmlp 3 days ago

link

It seems to be only for GNU/Linux systems, what about all other OSs out there?

reply

BruceM? 3 days ago

link

I've been experimenting with Nix on Mac OS X lately and it works fine. I've heard that it works on FreeBSD? as well. The big gap is Windows.

The good news is that you can integrate your language-specific tools with Nix as well, such as has been done for Haskell, node.js and other things. (I'm looking at it so that we can integrate our Dylan stuff with it.)

reply

pjmlp 3 days ago

link

When this discussions happen on HN, I always see a narrow discussion of Mac OS X, GNU/Linux, Windows and with luck *BSD.

But the world of operating systems is so much bigger than the desktop under the desk.

Good work on Dylan by the way.

reply

BruceM? 2 days ago

link

I'd love to have the time and the resources to deal with more OSes. :) 20 years ago, I had to keep stuff running on Solaris and lots of other platforms. About 20 years ago, I still did some work on VMS on actual VAX hardware! It wasn't that long ago, that we had the possibility of BeOS? either. Comparatively, we have quite a monoculture (of POSIX) these days with Windows being the non-POSIX representative.

Maybe unikernels like OpenMirage? will help make things interesting.

And thanks! The work on Dylan is a lot of fun and keeps me semi-sane by keeping me busy.

reply

CMCDragonkai 3 days ago

link

Nix is much more than just a package manager though.

reply

doe88 3 days ago

link

> Disappointing to see yet another language-specific package management system (Cargo), though.

Coming from Python I find Cargo very very smart and very well thought so far, it is not feature heavy, but everything has a very clear and useful purpose. For instance today I found that if I created a file .cargo/config I could override my dependancies to make Cargo search projects on my fs instead of grabbing them on Github, while doing developments it's a big thing I think.

reply

Ygg2 3 days ago

link

> Disappointing to see yet another language-specific package management system (Cargo), though.

I don't think it is. You need support for Rust modules on various platforms Linux/Mac/Windows(possibly Android). No single tool works on all those platforms. Cargo does and it has minimal dependencies.

Not having to juggle three different configuration (CMake, Makefile, etc.) on different platform is actually pretty great.

reply

---