Bayle Shanks's website: proj-oot-ootPackagingNotes1

(mostly moved to [1]

---

" The one thing I wish it had would be that your dependencies were yours only. I think this is one of the big reasons Node.js got successful so fast. In node you can run different versions with-in the same app depending on the libraries. Elixir is more like ruby in that you can only have one. Well you can have two for swapping out without down time but that is it. I do think this is one of the limits to the Erlang VM. "

---

" Packages and build systems

In addition to GHC itself, we make use of a lot of open-source Haskell library code. Haskell has its own packaging and build system, Cabal, and the open-source packages are all hosted on Hackage. The problem with this setup is that the pace of change on Hackage is fast, there are often breakages, and not all combinations of packages work well together. The system of version dependencies in Cabal relies too much on package authors getting it right, which is hard to ensure, and the tool support isn't what it could be. We found that using packages directly from Hackage together with Facebook's internal build tools meant adding or updating an existing package sometimes led to a yak-shaving exercise involving a cascade of updates to other packages, often with an element of trial and error to find the right version combinations.

As a result of this experience, we switched to Stackage as our source of packages. Stackage provides a set of package versions that are known to work together, freeing us from the problem of having to find the set by trial and error. "

---

https://nylas.com/blog/packaging-deploying-python

summary: docker sounds cool but it's too new for us. Wound up using dh-virtualenv

discussion: https://news.ycombinator.com/item?id=9861127

discussion summary:

svieira 13 hours ago

Back when I was doing Python deployments (~2009-2013) I was:

Downloading any new dependencies to a cached folder on the server (this was before wheels had really taken off) * Running pip install -r requirements.txt from that cached folder into a new virtual environment for that deployment (`/opt/company/app-name/YYYY-MM-DD-HH-MM-SS`) * Switching a symlink (`/some/path/app-name`) to point at the latest virtual env. * Running a graceful restart of Apache.

Fast, zero downtime deployments, multiple times a day, and if anything failed, the build simply didn't go out and I'd try again after fixing the issue. Rollbacks were also very easy (just switch the symlink back and restart Apache again).

These days the things I'd definitely change would be:

Use a local PyPi? rather than a per-server cache * Use wheels wherever possible to avoid re-compilation on the servers.

Things I would consider:

Packaging (deb / fat-package / docker) to avoid having any extra work done over per-machine + easy promotions from one environment to the next.

they should have used Docker anyway (paraphrased)

 Cieplak 17 hours ago

Highly recommend FPM for creating packages (deb, rpm, osx .pkg, tar) from gems, python modules, and pears.

https://github.com/jordansissel/fpm

"http://pythonwheels.com/ solves the problem of building c extensions on installation. " "Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process)" "Also, are there seriously places that don't run their own PyPI? mirrors?"

localshop and devpi are local PyPI? mirrors, apparently

 perlgeek 5 hours ago

Note that the base path /usr/share/python (that dh-virtualenv ships with) is a bad choice; see https://github.com/spotify/dh-virtualenv/issues/82 for a discussion.

You can set a different base path in debian/rules with export DH_VIRTUALENV_INSTALL_ROOT=/your/path/here

 erikb 7 hours ago

No No No No! Or maybe?

Do people really do that? Git pull their own projects into the production servers? I spent a lot of time to put all my code in versioned wheels when I deploy, even if I'm the only coder and the only user. Application and development are and should be two different worlds.

---

"The final question was about what he hates in Python. "Anything to do with package distribution", he answered immediately. There are problems with version skew and dependencies that just make for an "endless mess". He dreads it when a colleague comes to him with a "simple Python question". Half the time it is some kind of import path problem and there is no easy solution to offer." -- van Rossum, paraphrased in https://lwn.net/Articles/651967/

arturhoo 4 hours ago

I agree with Guido that the thing I hate the most in Python is packaging in general. I find Ruby's gems, bundler and Gemfile.lock to be a much more elegant solution.

On the other hand, I really like the explicit imports (when used properly). Less magic that makes code navigation way easier.

davexunit 28 minutes ago

As a distro packager, I find Python's packages to be much better and easier to integrate than Ruby gems. I've had no shortage of troubles with Ruby gems: requiring on the git binary to build the gem (even in a release tarball), test suites not being included in the gem releases on rubygems.org, rampant circular dependencies, etc. Python's PyPI? has caused me no such issues.

---

dchesterton 2 hours ago

I find external dependencies much more reliable in the PHP world than JS. Most packages try to follow semver. Composer is one of the best dependency manager tools I've used and you can easily lock down dependency versions so you can install from a state which you know works.

People hate on PHP but there are a lot of quality packages out there and Composer is definitely considered best practice by most developers now.

---

"Other points: Cargo is underrated. Traits over primitives is a huge win over Java's boxed collections."

---

JoshTriplett? 11 hours ago

I'm curious if there were any significant responses to the tooling/ecosystem questions regarding packaging and integration with Linux distributions.

I'd like to provide an application written in Rust in major Linux distributions. "cargo install" will not work, both because it depends on network access, and because it pulls in dependencies from outside the distribution. Similarly, many distributions have policies against bundling library sources in the application. There needs to be a straightforward way to turn a source package with a Cargo.toml file into a package for Debian and Fedora that depends on other packages corresponding to its Cargo dependencies (and C library dependencies).

bluejekyll 11 hours ago

There are people working on this, here's an announcement for a deb plugin to Cargo:

https://www.reddit.com/r/rust/comments/4ofiyr/announcing_car...

wyldfire 10 hours ago

I hadn't seen this. Thank you for sharing.

steveklabnik 11 hours ago

Cargo install was never intended to be used that way. It's for things useful for Rust developers, not for people who use a program who happen to be written in Rust.

There's a few things at play here. For "build from source" distros, we've been making some changes to make it easier to package rustc. Namely, instead of relying on a specific SHA of the compiler to bootstrap, starting with 1.10, it will build with 1.9, and 1.11 will build with 1.10. This is much, much easier for distros. As for Cargo dependencies on those packages, we'll see. There's a few ways it could go, but we need the compiler distribution sorted first.

A second is that we'd like to eventually have tooling to make giving you a .deb or .rpm or whatever easier: https://github.com/mmstick/cargo-deb is an example of such a tool. This won't necessarily be good enough to go into Debian proper; I know they tend to not like these kinds of tools, and want to do it by hand. But "Hey thanks for visiting my website, here is a thing you can download and install", these kinds of packages can be made with this kind of tooling.

In general, it's complex work, as then, these issues are also different per distro or distro family. We'll get there :)

JoshTriplett?