proj-oot-ootLibrariesNotes4

---

i guess we should at least have something like this:

https://github.com/zeit/micro

---

inglor 10 hours ago

> In order to maximize your usage of this feature, you'll want to use modules from the ecosystem that expose Promise instead of just a callback.

Bluebird which is one of the most popular promise libraries converts entire callback APIs to promises in one go:

Promise.promisifyAll(require("fs")); fs is now promisified and methods return promises.

(full disclosure, I'm a bluebird contributor)

reply

---

macros in js:

http://sweetjs.org/

---

some concurrency frameworks are suggested at the end of https://gist.github.com/staltz/868e7e9bc2a7b8c1f754

---

scala's pulsar concurrency framework

---

Without digging into it, the ((Elixir)) Calendar stuff looks like a strict improvement over Erlang's tuply handling of dates, times and timestamps, which are all tuples with three numbers. This can be confusing/problematic, and something that would benefit from typing to avoid mixing up Dates with Timestamps, say.

Can anyone who knows both confirm?

reply

antipax 1 hour ago

Yes, it absolutely is an improvement over erlang-style tuples IMO, and it also vastly improves the interoperability story between the stdlib, date/time/calendar libs, and database libs.

reply

"

Elixir Logo

    Home
    Install
    Getting Started
    Learning
    Docs
    Blog
    Packages

Elixir v1.3 released June 21, 2016 · by José Valim . in Releases

Elixir v1.3 brings many improvements to the language, the compiler and its tooling, specially Mix (Elixir’s build tool) and ExUnit? (Elixir’s test framework). The most notable additions are the new Calendar types, the new cross-reference checker in Mix, and the assertion diffing in ExUnit?. We will explore all of them and a couple more enhancements below.

With this release, we also welcome Andrea Leopardi to Elixir Core Team. He has contributed greatly to this release and maintains important packages in the community, like Gettext and Redix. Language improvements

The language has been improved semantically and includes new types and APIs. Let’s see the three major features. Deprecation of imperative assignment

Elixir will now warn if constructs like if, case and friends assign to a variable that is accessed in an outer scope. As an example, imagine a function called format that receives a message and some options and it must return a path alongside the message:

def format(message, opts) do path = if (file = opts) && (line = opts) do relative = Path.relative_to_cwd(file) message = Exception.format_file_line(relative, line) <> " " <> message relative end

  {path, message}end

The if block above is implicitly changing the value in message. Now imagine we want to move the if block to its own function to clean up the implementation:

def format(message, opts) do path = with_file_and_line(message, opts) {path, message} end

defp with_file_and_line(message, opts) do if (file = opts) && (line = opts) do relative = Path.relative_to_cwd(file) message = Exception.format_file_line(relative, line) <> " " <> message relative end end

The refactored version is broken because the if block was actually returning two values, the relative path and the new message. Elixir v1.3 will warn on such cases, forcing both variables to be explicitly returned from if, case and other constructs. Furthermore, this change gives us the opportunity to unify the language scoping rules in future releases. Calendar types and sigils

Elixir v1.3 introduces the Calendar module as well as 4 new calendar types:

    Date - used to store dates (year, month, day) in a given calendar
    Time - used to store time (hour, minute, second, microseconds)
    NaiveDateTime - used to store datetimes without a timezone (year, month, day, hour, minute, second, microseconds) in a given calendar. It is called naïve because without a timezone, the datetime may not actually exist. For example, when there are daylight savings changes, a whole hour may not exist (when the clock moves forward) or a particular instant may happen twice (when the clock moves backwards)
    DateTime - used to store datetimes with timezone (year, month, day, hour, minute, second, microsecond and time zone, with abbreviation, UTC and standard offset)

The current Calendar modules and its types is to provide a base for interoperatibility in the ecosystem instead of full-featured datetime API. This release includes basic functionality for building new types and converting them from and back strings.

Elixir v1.3 also introduces 3 new sigils related to the types above:

    ~D[2016-05-29] - builds a new date
    ~T[08:00:00] and ~T[08:00:00.285] - builds a new time (with different precisions)
    ~N[2016-05-29 08:00:00] - builds a naive da"

---

http://stackoverflow.com/questions/15995/useful-code-which-uses-reduce-in-python/282206#282206

---

i hear great things about the Scala collections library

also the operations in the LINQ section of Self:proj-plbook-plChData look like a good core set of collection operations, with special attention to those that they deigned to give C# LINQ syntax to.

Note that that core set is pretty much just a slightly expanded version of Python list (generator, actually) comprehensions.

---

numerical computing

you often hear about the popular fortran libraries BLAS and LAPACK.

LAPACK is a layer above BLAS (it calls BLAS). There are various implementations ("typically ATLAS + LAPACK, or MKL, or ACML" [1]). ATLAS contains BLAS and a subset [2] of LAPACK.

So maybe first provide the subset of LAPACK that ATLAS provides?

as of this writing, these are (from [3]):

" [S,D,C,Z]GESV [S,D,C,Z]GETRF [S,D,C,Z]GETRS [S,D,C,Z]GETRI [S,D,C,Z]TRTRI [S,D,C,Z]POSV [S,D,C,Z]POTRF [S,D,C,Z]POTRS [S,D,C,Z]POTRI [S,D,C,Z]LAUUM "

S,D,C,Z prefixes indicate single precision real, double precision real, single precision complex, double precision complex.

Here's what they do:

Solve AX = B:

Factorization:

Matrix determinant and inverse:

Products of special matrixes:

seems to me like we can start by just providing:

then:

then looking at BLAS, then LAPACK (the union of http://www.netlib.org/lapack/lug/node25.html and http://www.netlib.org/lapack/lug/node37.html appear to list the things that LAPACK can do, but the latter list is more numerics-y so avoid it for now), then see if MKL and ACML provide anything extra

then numpy

then scipy

---

yoavm 23 hours ago

D3 has the reputation of being super-complicated because of all the libraries that are based on it, "simplifying" it so that everyone can use it. In the past year I wanted to create pretty unique type of data visualisation, so I dived into D3 and discovered it a makes a lot more sense than I though. Of course, if you only want a regular bar chart, you'll do better with things like C3, nvd3 etc'. But if you want anything a bit special, D3 itself is very powerful and the documentation in pretty good - there's no reason to avoid using it directly.

Definitely looking forward to try the new release.

reply

minimaxir 22 hours ago

To add to that, if you are a complete newbie to any data visualization, do not start with d3. If you want to make pretty charts programatically, using R/ggplot2 or Python/Seaborn is good enough. Even Excel is fine if you tweak the defaults.

D3 is good if your visualization benefits from interactivity, either with dynamic data adjustment or rich tooltips. But static visualizations are important too. (I recently restructured my workflow so I can output static images and interactive charts with the same code, which makes it the best of both worlds.)

reply

danso 21 hours ago

What is your static+interactive workflow now, if I can ask? Also, is it fairly easy to build a workflow that generates static visualizations via D3 (i.e. making savable SVGs)?

reply

minimaxir 21 hours ago

I make charts with R/ggplot2. Standard workflow is to construct chart and save as static file. (PNG/SVG etc.) But with the plot.ly bridge, I can convert to an interactive chart w/ rich tooltips with very very little tweaking. (See this notebook: https://github.com/minimaxir/interactive-facebook-reactions/...)

To convert D3 to SVG, I believe all you need is a helper function but that is not my area of expertise.

reply

monfera 19 hours ago

There's also Shiny, Jupyter, Bokeh...

reply

minimaxir 19 hours ago

Those aren't updated clientside (have to ask the server for the new chart instead of just the data), which is a different architecture that does not scale.

reply

---

https://github.com/cht8687/You-Dont-Need-Lodash-Underscore

(Lists of JavaScript? methods which you can use natively )

"

frostik 8 hours ago

There's a huge gotcha: _.map, _.forech, ... can iterate over objects (often used as associative arrays) - Native map can't.

At some point you'll need lodash and co again - at least for convinience.

reply "

" Alternate title for the article: "Lists of lodash features replaced by ES6 if you don't mind throwing an exception when given invalid input such as null or undefined".

All kidding aside, a lot of our lodash code ends up looking something like this:

    function (xs) {
        return _(xs).pluck('foo').filter().value();
    }"

"

The need for Lodash and Underscore is diminishing, but not because of JS getting native support for a subset of utilities, but because there are already better alternatives. Utility libraries like Ramda[1] or pointfree-fantasy[2] with related modules which offer a better support for FP idioms and more useful higher-order functions are what made me drop Underscore.

I think that, with a language with such tiny stdlib, there always be a need for some kind of utility belt-style library ("if only for convenience", as another commenter write, which is actually the main point of such libraries). It's just a matter of choosing the best library for your coding style and project at hand. It may be a hassle, but an alternative is an stdlib the size of Python's - and I'm not sure if it's a good direction.

[1] http://ramdajs.com/0.21.0/index.html [2] https://github.com/DrBoolean/pointfree-fantasy

reply "

---

imatix SFL (legacy but this guy liked it [7] )

http://imatix-legacy.github.io/sfl/index.htm

---

dir(__builtins__)

['ArithmeticError?', 'AssertionError?', 'AttributeError?', 'BaseException?', 'BlockingIOError?', 'BrokenPipeError?', 'BufferError?', 'BytesWarning?', 'ChildProcessError?', 'ConnectionAbortedError?', 'ConnectionError?', 'ConnectionRefusedError?', 'ConnectionResetError?', 'DeprecationWarning?', 'EOFError', 'Ellipsis', 'EnvironmentError?', 'Exception', 'False', 'FileExistsError?', 'FileNotFoundError?', 'FloatingPointError?', 'FutureWarning?', 'GeneratorExit?', 'IOError', 'ImportError?', 'ImportWarning?', 'IndentationError?', 'IndexError?', 'InterruptedError?', 'IsADirectoryError?', 'KeyError?', 'KeyboardInterrupt?', 'LookupError?', 'MemoryError?', 'NameError?', 'None', 'NotADirectoryError?', 'NotImplemented?', 'NotImplementedError?', 'OSError', 'OverflowError?', 'PendingDeprecationWarning?', 'PermissionError?', 'ProcessLookupError?', 'RecursionError?', 'ReferenceError?', 'ResourceWarning?', 'RuntimeError?', 'RuntimeWarning?', 'StopAsyncIteration?', 'StopIteration?', 'SyntaxError?', 'SyntaxWarning?', 'SystemError?', 'SystemExit?', 'TabError?', 'TimeoutError?', 'True', 'TypeError?', 'UnboundLocalError?', 'UnicodeDecodeError?', 'UnicodeEncodeError?', 'UnicodeError?', 'UnicodeTranslateError?', 'UnicodeWarning?', 'UserWarning?', 'ValueError?', 'Warning', 'ZeroDivisionError?', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']

>>> len(dir(__builtins__)) 151 >>> len(['abs', 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']) 72

---

how many functions are in the Python standard library?

i don't know a quick way to find that out.

the official tutorial has a 'brief tour of the standard library' at https://docs.python.org/2/tutorial/stdlib.html and https://docs.python.org/2/tutorial/stdlib2.html

they list sections and modules (with approximate number of contained functions in parens):

(total of numbers in parentheses: 622)

not counting the ones listed under 'Batteries Included', that's 35 modules, which are:

os, shutil, glob, getopt, argparse, sys, re, math, random, urllib2, smtplib, datetime, zlib, gzip, bz2, zipfile, tarfile, timeit, profile, pstats, doctest, unittest, repr, pprint, textwrap, locale, string, struct, threading, logging, weakref, array, collections, heapq, decimal

(numbers in parentheses were obtained using:

def count_namespace(ns, already_seen = None): if already_seen is None: already_seen = set()

  items = [x for x in dir(ns) if [types.FunctionType, types.BuiltinFunctionType, types.UnboundMethodType].__contains__(type(ns.__getattribute__(x))) and x[0] != '_']
  items = set(items).difference(already_seen)
  already_seen = already_seen.union(items)
  count = len(items)
  for subns in [ns.__getattribute__(x) for x in dir(ns) if (type(ns.__getattribute__(x)) == type) and x[0] != '_']:
    try:
      subns2 = subns()
      countIncrement, already_seen = count_namespace(subns2, already_seen=already_seen)
      count += countIncrement
    except TypeError:  # if the constructor for subns required arguments
      pass
    except ValueError:  # no such test method in <class 'unittest.case.TestCase'>: runTest
      pass
      
  return count, already_seen

already_seen = set() for n in ['os', 'shutil', 'glob', 'getopt', 'argparse', 'sys', 're', 'math', 'random', 'urllib2', 'smtplib', 'datetime', 'zlib', 'gzip', 'bz2', 'tarfile', 'timeit', 'profile', 'pstats', 'doctest', 'repr', 'pprint', 'textwrap', 'locale', 'string', 'struct', 'threading', 'logging', 'weakref', 'array', 'collections', 'heapq', 'decimal']: from importlib import import_module; m=import_module(n); count, already_seen = count_namespace(m, already_seen = already_seen); print n, count

  1. zipfile omitted b/c 'CRC' in dir(zipfile.ZipInfo?()), but zipfile.ZipInfo?().CRC yields an AttributeError?
  2. unittest omitted b/c it did something weird when i ran this

note: this is an underestimate, because we are only counting one when multiple functions in different modules have the same name; eg there is both decimal.Decimal.log10 and math.log10 but we only count one.

)

old:

def count_namespace(ns, already_seen = None): if already_seen is None: already_seen = set() print dir(ns)[0] print ns.__getattribute__(dir(ns)[0])

  items = [x for x in dir(ns) if [types.FunctionType, types.BuiltinFunctionType, types.UnboundMethodType].__contains__(type(ns.__getattribute__(x))) and x[0] != '_']
  print 'AS: ', already_seen
  items = set(items).difference(already_seen)
  already_seen = already_seen.union(items)
  print ns, items
  count = len(items)
  for subns in [ns.__getattribute__(x) for x in dir(ns) if (type(ns.__getattribute__(x)) == type) and x[0] != '_']:
    countIncrement, already_seen = count_namespace(subns, already_seen=already_seen)
    count += countIncrement
  return count, already_seen

already_seen = set() for n in ['os', 'shutil', 'glob', 'getopt', 'argparse', 'sys', 're', 'math', 'random', 'urllib2', 'smtplib', 'datetime', 'zlib', 'gzip', 'bz2', 'zipfile', 'tarfile', 'timeit', 'profile', 'pstats', 'doctest', 'unittest', 'repr', 'pprint', 'textwrap', 'locale', 'string', 'struct', 'threading', 'logging', 'weakref', 'array', 'collections', 'heapq', 'decimal']: from importlib import import_module; m=import_module(n); count, already_seen = count_namespace(m, already_seen = already_seen); print n, count

note: this i

---

https://docs.python.org/2/library/index.html#library-index lists about 25 main library sections, here they are with the number of modules in each section:

7. String Services: 11 8. Data Types: 19 9. Numeric and Mathematical Modules: 9 10. File and Directory Access: 12 11. Data Persistence: 13 12. Data Compression and Archiving: 5 13. File Formats: 6 14. Cryptographic Services: 4 15. Generic Operating System Services: 17 16. Optional Operating System Services: 9 17. Interprocess Communication and Networking: 7 18. Internet Data Handling: 16 19. Structured Markup Processing Tools: 15 20. Internet Protocols and Support: 25 21. Multimedia Services: 10 22. Internationalization: 2 23. Program Frameworks: 2 24. Graphical User Interfaces with Tk: 7 25. Development Tools: 6 26. Debugging and Profiling: 7 27. Software Packaging and Distribution: 2 28. Python Runtime Services: 16 29. Custom Python Interpreters: 2 30. Restricted Execution: 2 31. Importing Modules: 7 32. Python Language Services: 13

---

nostrademons 3 hours ago [-]

It's because languages are communication tools as well as compilers, and the stdlib is part of the common vocabulary that the rest of the language ecosystem shares.

Without a decent stdlib, you end up with the pre-STL C++ situation, where every library & framework declares its own string, vector, and hashmap classes and you need slow, verbose, and error-prone routines to convert between them. The main benefit of having these in the stdlib isn't that you don't have to re-implement them, it's that every third-party lib knows exactly which class represents the concepts of text, sequences, and dictionaries.

The same goes for many other types, eg. Promises, Paths, URIs, Dates, Input/OutputStreams?, Iterators, etc. Indeed, when one of the stdlib types is misdesigned (eg. java.util.Date), you get an utter mess as the community converges on an alternative (eg. JodaTime?) and then the alternative API is folded back into the stdlib (Java 8).

Luckily, we're learning just which types need to be in the stdlib and which can be outsourced to a third-party package manager. In particular, a lot of serialization/parsing formats are better off in a package manager, as long as the language defines an annotation system that can be used to define which fields must be saved. And giant systems like webservers, webframeworks, or RPC formats are often better off as a third-party library.

reply

jackmott 10 hours ago [-]

F# isn't much like C#, and in fact it does suffer a bit for it, as some of the back end causes compromises in the F# language. Mutable fields in records have secret baggage, functions with arithmetic operators can't be automatically genericized unless you mark them inline, etc

reply

__s 6 hours ago [-]

My biggest let down with F# is that NullReferenceException? is still a thing. Also a C# function is called as if they're typed to take tuples but then you can't create a tuple & pass it along

reply


the HLA Standard Library

http://www.plantation-productions.com/Webster/HighLevelAsm/HLADoc/HLAStdlib/1_HTML/HLAStdlib.htm (see http://www.plantation-productions.com/Webster/HighLevelAsm/HLADoc/index.html for other formats)

---

mb the YASM 'libyasm' module (YASM is a ground-up rewrite of NASM), although that seems to really be just for assembler (assembly language compiler)-related stuff:

https://github.com/yasm/yasm/wiki/Libyasm http://yasm.tortall.net/ (section Key Internal Features )

---

https://github.com/fantasyland/fantasy-land https://github.com/hemanth/functional-programming-jargon

---

https://www.haskell.org/platform/contents.html

---

tibbe 14 hours ago [-]

I wouldn't recommend following the Haskell approach ((small stdlib core with a 'platform' of recommended other libs; https://www.haskell.org/platform/ )). It hasn't worked well for us. (I took part in creating the Haskell Platform and the process used to add packages to it. I also used to maintain a few of our core libraries, like our containers packages and networking).

Small vs large standard library:

A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries.

Here's a number of examples of problems we currently have:

Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.

reply

---

"who doesn't have `mtl` in their applications at this point" -- Haskeller, in a comment (yes, mtl is in the Haskell Platform)

---

pcwalton 2 hours ago [-]

> But if you have a large standard library and want to break the API, you can.

We have a policy of no breakage for stable libraries post 1.0 (as does Python, and Go, etc.). So no, we can't.

The size of the standard library has nothing to do with it.

reply

((this was in reply to a comment proposing a larger stdlib, rather than a small core stdlib, so that the language could atomically make changes in various libs without breaking other libs; pcwalton is saying that regardless of if the atomic change would fit inside the stdlib, a no-breaking-changes-to-stdlib policy prevents this anyways))

---

rtpg 16 hours ago

parent flag favorite on: The Rust Platform

...I think the failure of the Haskell Platform has a lot more to do with how Haskell deals with dependencies, and the difficulties it entails, than with the "batteries included" approach itself.

((this guy is on a thread arguing about whether a small core stdlib with a larger recommended stdlib is a good idea; e is arguing that it is; the thread had previous cited Haskell Platform as a failure of this sort of thing, and e is rebutting this by saying the failure was actually with Haskell dependency management))

hyperhopper 14 hours ago

parent flag favorite on: The Rust Platform

But even then, languages that have great, thriving easy to use dependency systems and package managers with small standard libraries still run into problems

see: javascript

---

"Python codebases that are resilient don't use much of "core", arrow for time, requests for http, simplejson, etc. "

---

pjmlp 11 hours ago [-]

I disagree.

In the enterprise space it is quite common that we only get to use what it is in the computer and access to anything else is strictly controlled by IT.

So if it isn't in the standard library or some internal library mirror, we don't get to use it, as simple as that.

reply

---

" The 'std lib is where libraries go to die' was invented by Python. The libs are shallow, don't break backwards compat and provide a substandard experience. Things that continue to improve provide an out of tree alternative package name. "

---

---

One of the things I love above all about Python and Ruby are the kitchen-sink standard libraries. The node ecosystem is deeply frustrating in this respect.

reply

krylon 2 hours ago [-]

It has been a while since I did anything with Python, but I did like its standard library. It was reasonably comprehensive without feeling bloated, and the documentation was pretty good (mostly).

Having a good standard library also makes deployment easier. (In Go, OTOH, I tend to care less, even though its standard library is quite good, because thanks to static linking, deployment is always easy, no matter how many third-party libraries I use.)

reply

---

steveklabnik 11 hours ago [-]

Rust will already allow you to have multiple versions of transitive dependencies.

reply

bassislife 2 hours ago [-]

Which I guess is normal since it does not create any dependency cycle. A new version might as well be thought as a completely different package (of perhaps similar functionality).

reply

sitkack 5 hours ago [-]

This needs to be screamed from the hill tops!

reply

---

https://luarocks.org/stats/this-week

---

kibwen 12 hours ago [-]

  > Empirically, languages that have large standard 
  > libraries (e.g. Java, Python, Go) seem to do better than 
  > their competitors.

You seem to be overlooking the ultimate counterexample: C. :P

reply

brandonbloom 11 hours ago [-]

You were being down voted, maybe for perceived snark, but I think you raise an interesting point.

To me, C did have a standard library: Unix. It's a runtime system too! Due to the nature of the original C bootstrapping process it just happens to be possible to remove this standard library, and Windows was evidence of this.

There is another interesting potential counter example: Lua. It's minimalistic standard library is part of what makes it so attractive for embedding, eg. in game engines. However, Lua's embedding API is so good, you could almost say that it comes with a large standard library too: Your existing C code!

...

 pjmlp 6 hours ago [-]

> To me, C did have a standard library: Unix. It's a runtime system too!

Fully agree. We just ended up with ANSI C + POSIX, because the standard bodies refused to put everything into the same bag.

In the early days, most C compilers were anyway shipping partial UNIX APIs on top of their K&R and ANSI implementations.

reply

 ---

ufo 10 hours ago [-]

Lua's lack of a stdlib is also a curse. I can't imagine how many incompatible versions of string.trim and OOP libraries are out there in the wild right now...

Things have been getting better lately because of Luarocks but its still an uphill battle.

reply

String trim is just:

  foo:gsub("%s*$", "")

or

  foo:gsub("^%s*", "")

The standard idiom for OOP in Lua is a one-liner:

  return setmetatable(self, mt)

where mt.__index has all the methods. How you assign to mt.__index can vary across modules according to style, but that's a _purely_ asethetic issue. The mechanics are identical. Using a module to accomplish it creates a useless dependency.

There are many criticisms one could make of Lua, but I don't think those two particular criticisms are legit. They're classic bikeshedding.

reply

JoshTriplett? 6 hours ago [-]

If you're reading a pile of string processing code, seeing

    s.rstrip()

helps make code self-documenting, compared to

    s:gsub("%s*$", "")

I don't want to argue for a massive standard library (for instance, I don't think Python should have shipped modules for dbm, bdb, sqlite, or XML-RPC), but simple string processing seems like a good thing to standardize.

reply

---

comex 6 hours ago [-]

> You seem to be overlooking the ultimate counterexample: C. :P

I think one reason (of many) that C++ has replaced C almost completely for new development is the STL. Of course, the STL fundamentally depends on the language feature of templates, which you can only approximate in C, but considering that Java and Objective-C, among other languages, lasted pretty long with no generics and only non-type-safe containers, I think C could have benefitted greatly from basic things like resizable arrays, hash tables, trees, better strings, etc. in the standard library. Now it is probably too late for it to matter (which most people consider a good thing).

reply


krylon 2 hours ago [-]

FWIW, the last time I cooked up something in C, I liked Judy very much: http://judy.sourceforge.net/

It has slightly awkward but very simple API, and it's very fast.

reply

---

bitmadness 9 hours ago [-]

This is exactly the main problem with Haskell. A stunning language with a lousy standard library. In my opinion, Haskell should offer arrays and maps as built-ins (like Go) and ship with crypto, networking, and serialization in the standard library (I know serialization is already there, but everyone seems to prefer Cereal, so...)

reply

---

kibwen 12 hours ago [-]

Go isn't immune to the problem either. See the `flag` package, which is something that new users are encouraged to avoid in favor of e.g. https://github.com/jessevdk/go-flags .

reply

...

the_mitsuhiko 6 hours ago [-]

Everybody I have ever seen do CLI applications in Go will recommend heavily against it ((the flag package))

reply

---

..."Python simplejson vs json. Most working Python developers I know try simplejson first...because simplejson got much faster as it evolved outside of the standard library[0]"

---

steveklabnik 13 hours ago [-]

In the Ruby world, very few people use the standard library because it's got so many flaws, and they can't be fixed. So you end up with Nokogiri rather than REXML, all the various HTTP libs rather than net/*, etc. So it just ends up being bytes sent over the wire, wasting disk and bandwidth...

reply

anamoulous 9 hours ago [-]

I wouldn't say very few people use the standard library?

CSV, logger, json, fileutils, tempfile, pp, and on and on are used all the time...

reply

steveklabnik 9 hours ago [-]

There are some parts that are good, and some parts that are bad, for sure.

reply

---

shadowmint 9 hours ago [-]

Sql?

It does all the wrong things; singletons, no testability, cgo for implementations, side effects and you have to use every database differently based on their individual semantics.

Virtually everyone I've ever spoken to either uses a high level wrapper around the sql library or a no-sql solution.

That's the definition of 'stdlib is where packages go to die'.

It's not that the API is unusable, it's just basically not used by the community because there are other better things out there...but you're stuck with it forever, because it's there and some people do use it, and changing or removing it would be a breaking change.

Anyhow, we're just speculating. Does anyone actually collect metrics about the usage of different parts of the stdlib for any language?

Without hard data to back it up, you couldn't really make a strong argument either way.

reply

saysjonathan 1 hour ago [-]

I didn't see anyone actually mention sql so I'll just assume your first line is to be interpreted as "sql is the counterexample of why Go's standard library is not as great as it may seem."

>Virtually everyone I've ever spoken to either uses a high level wrapper around the sql library or a no-sql solution.

How does that reflect the quality of the std lib implementation? All the high-level wrappers I've seen still utilize database/sql, they just provide convenience methods on top of the existing functionality. Are people using NoSQL? databases because database/sql is so bad or merely because that technology fits their project's requirements?

>That's the definition of 'stdlib is where packages go to die'.

steveklabnik's example of Ruby XML parsing libraries is a better example of this, if only because the std lib implementations are almost completely ignored by all other gems. Go's database/sql is actively used outside of the std lib to great affect, whether in wrappers and ORMs or in implementing other SQL databases (like Postgres).

reply

---

rcfox 9 hours ago [-]

Out of curiousity: what do the 5 string types do differently?

reply

geekingfrog 9 hours ago [-]

String: Linked list of Char. Nice for teaching, horrible in every other aspects. Text and lazy Text: modern strings, with unicode handling and so on. ByteString? and lazy ByteString?: these are actually arrays of bytes. Used to represent binary data. Because haskell is lazy by default, and sometimes you want strictness (mostly for performances), there are two variants of Text and ByteString?, and going from one flavor to the other requires manual conversion.

reply

tibbe 8 hours ago [-]

Risking to go off-topic a bit, I think the lazy versions of Text and ByteString? wouldn't have been needed if we had nice abstractions for streams (lists are not, they cause allocation we cannot get rid of) so that you don't need to implement a concrete stream type (e.g. lazy Text and lazy ByteString?) for every data type.

Rust does this well with iterators, for example.

reply

wyager 8 hours ago [-]

The problem is that streams actually have very complicated semantics when they interact with the real world. What does it mean to traverse an effectful stream multiple times? Can you even do that?

Data.Vector provides a very efficient stream implementation for vector operation fusion, but it's unsuitable for iterators/streams that interact with the real world. Pipes, on the other hand, combined with FreeT?, provides good, reasonable semantics for effectful streams.

As with many other things, Haskell forces you to be honest with what your code is actually doing (e.g. streaming things from a network) and this means that there's no one-size-fits-all implementation we can stuff everything into.

reply

tibbe 7 hours ago [-]

Just sticking with the pure types there's currently no generic stream model that works well. No stream fusion system fuses all cases (even in theory) and they also fail to fuse the cases they're supposed to handle too often in practice.

I haven't looked at pipes, but I'm guessing it doesn't all fuse away either.

reply

wyager 7 hours ago [-]

You're right, I believe Haskell's fusion framework could be greatly improved (although it is the best production solution I'm aware of). However, how would you go about solving this? I don't think there's any generalized solution to the problem of creating no-overhead iteration from higher-level iterative combinators.

reply

losvedir 58 minutes ago [-]

> Haskell's fusion framework could be greatly improved (although it is the best production solution I'm aware of). However, how would you go about solving this?

Given that we're in a rust thread... are you familiar with rust's iterator fusion [0]? Basically there are three components: iterators (something like a source), iterator adapters (where all manipulations happen), and consumers (something like a sink). LLVM will compile all the iterator adapters into a single manipulation such that the underlying stream/vector/whatever only goes through it once.

I personally like it much better than Haskell's. With rust the fusion is guaranteed to happen, although it makes the types a little verbose and tricky to work with, but with, e.g., Haskell's Text's stream fusion I was never really sure that it was working, or if I could do something to prevent it. It seems like in Haskell it's more of a behind the scenes optimization that you hope kicks in, rather than designed into the types. Or do I misunderstand? I only dabbled in Haskell.

[0] https://doc.rust-lang.org/book/iterators.html

reply

---

catnaroek 13 hours ago [-]

Haskell's actual problem isn't the lack of a comprehensive standard library, but rather the presence of core language features that actively hinder large-scale modular programming. Type classes, type families, orphan instances and flexible instances all conspire to make as difficult as possible to determine whether two modules can be safely linked. Making things worse, whenever two alternatives are available for achieving roughly the same thing (say, type families and functional dependencies), the Haskell community consistently picks the worse one (in this case, type families, because, you know, why not punch a big hole on parametricity and free theorems?).

Thanks to GHC's extensions, Haskell has become a ridiculously powerful language in exactly the same way C++ has: by sacrificing elegance. The principled approach would've been to admit that, while type classes are good for a few use cases, (say, overloading numeric literals, string literals and sequences), they have unacceptable limitations as a large-scale program structuring construct. And instead use an ML-style module system for that purpose. But it's already too late to do that.

reply

---

yoshuaw 12 hours ago [-]

If a language is to play the long game, they must be conservative on what they add. Even a minimal runtime like Node is still wounded by the addition of a few broken interfaces into the core platform (even emitters, streams, domains to name a few). These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust.

For a language to remain relevant in the long term, a system must be capable of evolving. Going out there and blessing potentially conceptually incorrect packages such as Mio is therefore not a good idea.

---

rcthompson 12 hours ago [-]

> These cannot be stripped out of the platform because they're "blessed" and now everyone will forever have a bad time. I suggest we don't do that for Rust.

It sounds like the proposal in the OP avoids this problem by having versioned platforms that are independent of the Rust version. So if something turns out to be a bad idea, it can be stripped out of later platform versions and replaced with something better without disrupting users of the older platforms.

reply

---

erickt 11 hours ago [-]

> By keeping platforms / package collections in userland we can get organic collections of stuff that make sense to bundle together. Imagine a "server platform", an "event loop platform", a "calculus platform", a "programming language platform". It would be detrimental to creativity to have all these collections live in the shadow of the "rust committee blessed platform".

That's exactly what we want to do. We don't have the domain expertise in many of these areas, but we want to enable those ecosystems to help develop their own platforms. It's one of our goals to make sure any infrastructure we develop for the "Rust Platform" is usable throughout our community.

reply

---

pcwalton 14 hours ago [-]

C++ has innumerable de facto "platforms" and "ecosystems". You have to choose one to get anything done, whether that be various Boost libraries, POSIX, Win32, Cocoa, Qt, GTK(mm), even stuff like XPCOM…

Helpfully, many of these platforms reinvent basic things like strings [1] and reference counted smart pointers [2] in incompatible ways.

Wouldn't it be better if there were just one platform?

[1]: http://doc.qt.io/qt-4.8/qstring.html

[2]: http://doc.qt.io/qt-4.8/qshareddatapointer.html

reply

---

saysjonathan 14 hours ago [-]

Some languages have also seen great success with having a standard platform but still allowing users to replace it as needed. The many Haskell Preludes and Jane Street's ocaml core are two such examples.

I think the ability to opt out of the rust-platform metapackage is a great feature.

reply

---

dikaiosune 14 hours ago [-]

...

What other installers are you referring to, exactly? The old rustup.sh which couldn't support multiple toolchains installed alongside each other? multirust which didn't work on Windows? rustup is a massive improvement over both, IMO.

reply

---

tdicola 10 hours ago [-]

Go watch recent talks by Herb Sutter and Bjarne Stroustrup--they both lament the fact that C++ never developed as strong a standard library as Python, etc. With C++14 and beyond the C++ working committee is actively trying to make the language and libraries more complete and comparable to larger libraries out there.

reply

---

zanny 14 hours ago [-]

Qt / Boost / .net are C++ platforms. The difference is you can choose one or none, and the OP actually explicitly talks about how important it is not to have one absolute blessed platform like Java has.

reply

majewsky 4 hours ago [-]

When I was developing KDE applications, I refered to my programming language as "C++/Qt".

reply

pjmlp 6 hours ago [-]

> C++ does not have a "platform". Nor does it need one.

Sure it does, it is called C++ Standard Library and ISO C++ catching up to what other languages already offer.

https://isocpp.org/std/status

reply

---

jonstewart 1 hour ago [-]

The relationship between Boost and the C++ standard library might provide a good example (especially in recent years, as there's been sustained focus on expanding stdlib). Boost has a stringent peer review and comment period. Sometimes I think they let in designs that are too clever, but they rarely let in the clunkers that are seen in Python's standard library.

People then gain experience with it and sometimes subtle issues emerge. Those issues can be fixed, either in Boost itself or when libraries move from Boost to stdlib (and sometimes the issues inform language features).

While C++'s standard library is not "batteries included" like Python's, it's been very gratifying to see it expand slowly and surely over the last few years.

reply

---

hsivonen 6 hours ago [-]

Having used Java and having experienced how you learn to replace the JDK URL parser with something else, the JDK HTTP client with something else, the JDK encoding libraries with something else, etc., I'm worried about entrenching first-published libraries beyond their natural first-mover advantage and making it even harder for better later-comers to be adopted.

OTOH, compared to e.g. C++, where even strings aren't standard across multiple real large code bases (e.g. Gecko and Qt) having standard types especially for foundational stuff can be very useful if the foundation is right or right enough. Java having a single (though not optimal) foundational notion for Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and spec-wise correct (though not design-wise optimal) XML API (SAX) was a key reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was compiled!).

Still, at this early stage, I'd point to crates.io in general as the "batteries" instead of creating a layer of official approval on top. But I'm biased, because lately, I've been writing a second-to-market crate for a topic for which a crate already exists.

reply

---

themihai 5 hours ago [-]

You have modules, use, extern, cargo.toml, cargo.lock. It looks quite verbose and redundant to me. I expected it to simplify but moving more to config files seems a backwards step. I may be biases but I find the Go import/packing vastly superior in terms of usuability(i.e using a single import statement you make the import explicit and ready to use). I think the trick is to build the tools on top of the source code not on top of config files(i.e ala C/C++).

reply

---

qwertyuiop924 14 hours ago [-] Good work! The idea of dropping extern crate is worrying, however. Most of the ways that that would be done would add more irregularity, complexity, and implied environment to the language (Rust Platform is always there, whether or not you want it), all of which are opportunities for bugs to creep into code.

reply ...

gpm 14 hours ago [-]

> rustc already requires passing an --extern flag for each 'extern crate' in the source

Only if it can't find the crate otherwise. Generally it just searches your library path and any directories you specify with -L, e.g. most cargo executables can be recompiled (the first compilation via cargo used to compile the dependencies) via `rustc src/main.rs -L target/debug/deps`

reply

---

https://github.com/Ericson2314/rfcs/blob/cargo-libstd/text/0000-cargo-libstd-awareness.md

Currently, all packages implicitly depend on libstd. This makes Cargo unsuitable for packages that need a custom-built libstd, or otherwise depend on crates with the same names as libstd and the crates behind the facade. The proposed fixes also open the door to a future where libstd can be Cargoized. ... Currently, Cargo doesn't know what parts of the standard library packages depend on. By giving it this knowledge, we can make cross compilation and exotic platform development easier, simplify rustbuild, and allow anyone to easily specify bounds on the standard library for their packages. This will allow building parts of the standard library from source, but in order to not disrupt existing workflows, the binaries that come with rustc will still be used by default. ...

---

dikaiosune 13 hours ago [-]

Pedantically, the standard library (strings, basic networking, containers/collections, etc.) is already well defined and quite small. The Rust Platform idea seems more aimed at providing access to curated, stabilized versions of community-developed librares for common higher-level tasks (async I/O, serialization/deserialization, etc.).

Personally, I think that letting users (especially beginners) opt in to a larger set of starting libraries could be very beneficial for adoption.

reply

---

moogly 3 hours ago [-]

For contrast, with .NET Core, Microsoft decided to cut up .NET's quite extensive stdlib (BCL + FCL) into packages. On paper, it looks pretty good, but it remains to see how well the versioning will work in the long run.

reply

---

coolsunglasses 15 hours ago [-]

Haskell Platform is the last thing you should take inspiration from. Many of us have been doing our best to kill it off. Maybe the downsides involved wouldn't affect Rust in the same ways.

My suggestion, look at how Stack (the tool) and Stackage (the platform) work:

https://docs.haskellstack.org/en/stable/README/

https://github.com/commercialhaskell/stack

https://www.stackage.org/

reply

saysjonathan 15 hours ago [-]

Most of your arguments linked in a comment below are unrelated to the application of metapackages to cargo. Cargo already includes a good portion of the behaviors found in Stack and Stackage. The idealogical battle of Stack vs Haskell Platform is irrelevant to this proposal.

Haskell Platform is a perfectly adequate example to take high-level inspiration for the core of this idea: use Cargo (which is like Stack/Stackage for rust) to help bootstrap Rust libraries when using rustup to install and upgrade (mostly equivalent to `stack setup`).

reply

Ericson2314 14 hours ago [-]

The one thing stack does do nicely that this proposal doesn't is allow curated comparable versions without forcing course dependencies. In plainer English, stack can pick the versions while you still opt in pack by package. I beleive that this is crucial to not slow down the evolution of the ecosystem.

In Cargo jargon, a solutuon would be for metapackages could double as sources: `foo = { metapackage = bar }` to use foo from bar.

reply

quodlibetor 13 hours ago [-]

To make this more concrete, you're suggesting something like

   [dependencies]
   hyper = { metapackage = { rust-platform = "2.7" } }

I like the general theme of having the platform be just a set of known-compatible versions, but on the other hand this feels like it loses out on many of the ease-of-use advantages of just specifying a platform version and knowing that you have all the crates inside of it.

reply

Ericson2314 12 hours ago [-]

Yeah that's a proper example.

I don't think we need to pick one way exclusivity. My specific use-case is making PRs for packages to work in kernelspace / in unikernels. The library night initially be packaged the easy way but then I'd use this. I don't want the PR recipient to also worry they might get out of sync with the platform as a side effect.

reply

---

" LTS Haskell: Version your Ecosystem ... Stackage is a stable source of Haskell packages. We guarantee that packages build consistently and pass tests before generating nightly and Long Term Support (LTS) releases. "

---

" New people do not understand:

1. ghc-pkg

2. The global vs. user package databases

3. Cabal

4. What Platform does to their build environment

New Haskell users are least equipped to understand errors caused by:

1. Package version conflicts

2. Something innate to their install, rather than something wrong with their package constraints.

Part of the problem is that the people making decisions on these matters are not the people on the frontlines helping new people in IRC or otherwise in the wild with sufficient regularity to see how often this catches people up. It's less common than it used to be, but that's due in large part to people following the install instructions on https://github.com/bitemyapp/learnhaskell and using things like MinGHC? and Stack. "

-- https://mail.haskell.org/pipermail/haskell-community/2015-September/000014.html

---

coolsunglasses 14 hours ago [-]

>so many things Haskell are hard to build/install for no-good reason.

Were. Were hard. It's quite fine now with Stack. I know users of all kinds of languages that all miss Stack when they're working on their non-Haskell projects.

reply

---

seagreen 14 hours ago [-]

Not sure why you're being downvoted. Haskell tooling definitely started turning around with Stack. I've used a fair amount of package managers (including cutting edge stuff like Nixpkgs) and Stack is by far my favorite.

EDIT: To elaborate on why I like Stack:

+ Fully declarative. I don't run commands to edit my Stack environment, instead I modify the relevant stack.yml file. This means that the current environment can always be easily examined, committed to Git, etc.

+ Easy NixOS? integration which means I can also describe the non-haskell dependencies for a project, and enforce that _absolutely_ nothing else on my system gets used. This is amazing.

+ I like that it uses Stackage by default, and that I can pin projects to LTS Stackage releases. This means that many users of my projects will probably already have most dependencies installed -- especially cool for small things like tutorials where users won't have the patience for long build times.

+ It reuses the already existing `.cabal` file format so it's easy to make libraries compatible with cabal-install, the other package manager in the Haskell ecosystem.

reply

saysjonathan 14 hours ago [-]

You're clearly very into Stack. If you're interested though, cabal-install 1.24 introduced nix-style local builds.

reply

seagreen 13 hours ago [-]

Hey! So I'm deeply appreciative towards cabal-install for historical reasons. The situation before cabal-install was . . . not the best (https://www.gwern.net/Resilient%20Haskell%20Software) so I think we can credit cabal-install with a lot of the success the ecosystem has had over the last decade.

That said, nix-style builds don't actually address my issues with cabal-install. My (very personal) preference is to always have the current dependencies being used reflected exactly in a local file. I picked up this attitude from NixOS?, but it's an extremely useful way of doing things -- the knowledge that as long as I don't lose that file I can always rebuild my current environment exactly is just too awesome.

I think that if you need to track version bounds as well (say you're developing a library) they should be kept in a separate file, and you should always have a master file that describes what you're currently using locally. Happily this is exactly what Stack does with my-project.cabal and stack.yml.

That said, I appreciate that others like cabal-install, so I do what I can to make my code easy to use with both package managers (the main thing is keeping aggressive upper bounds on the few libraries I maintain, which especially helps cabal-install users since they do dependency solving more).

---

coolsunglasses 15 hours ago [-]

Haskell Platform is the last thing you should take inspiration from. Many of us have been doing our best to kill it off. Maybe the downsides involved wouldn't affect Rust in the same ways.

My suggestion, look at how Stack (the tool) and Stackage (the platform) work:

https://docs.haskellstack.org/en/stable/README/

https://github.com/commercialhaskell/stack

https://www.stackage.org/

reply

---

https://github.com/rust-lang/cargo/issues/2064

Cargo should understand distinctions between public/private dependencies #2064

...

If a dependency is an implementation detail (i.e. no types are exposed) then it can be easily duplicated in the dependency graph without worry (other than concerns like binary size). In other words, Cargo's safe to have multiple semver-incompatible versions of this dependency because they won't talk with one another.

If a dependency is part of the public API, however, then it's critical that any crate which can share this type it needs to all be of the same version. In other words, Cargo's logic of allowing multiple semver-incompatible versions is almost always guaranteed to go awry.

---

Cargo is overeager to pull in multiple copies of a crate

Say we have a crate with the following Cargo.toml:

[package] name = "foo" version = "0.1.0" authors = ["Steven Fackler <sfackler@gmail.com>"]

[dependencies] postgres = "0.9" r2d2_postgres = "0.9"

r2d2_postgres version 0.9.3 depends on postgres version 0.10, while version 0.9.2 and lower depend on postgres version 0.9. Cargo would ideally pick r2d2_postgres version 0.9.2 and postgres version 0.9.6, but it instead picks r2d2_postgres version 0.9.3, which pulls in postgres version 0.10, as well as postgres version 0.9.6. This ends up totally blocking any use of these dependencies.

This issue rather significantly impedes crates' abilities to upgrade their dependencies.

It seems like Cargo's resolution logic should have the property that if there exists a resolved dependency graph with no more than one copy of each dependency, it should never pull in more than one copy of a dependency. Unfortunately, questions like "does such a resolved dependency graph exist" are NP-complete, but I suspect that it's tractable in practice (we don't expect crates to be randomly permuting their dependencies every minor version, for example).

In the meantime before someone has a chance to integrate a SAT solver into Cargo, I think there are some things we can do to help users correct Cargo when it fails to resolve things "properly". For example, cargo update could warn whenever it has to pull in multiple copies of a crate, identifying the portion of the object graph that forced that to happen. This should provide users an easier starting point to poking the graph into shape with cargo update --precise.

see https://github.com/rust-lang/cargo/issues/2064

---

steveklabnik 15 hours ago [-]

I actually cross-posted this to /r/haskell to get explicit feedback. Someone else mentioned stack/stackage. In my understanding, Cargo already does this specific behavior. Can anyone who's more familiar with both confirm this?

reply

coolsunglasses 14 hours ago [-]

I checked that other thread, I agree completely with jeremyjh's summary.

The problem with Haskell Platform came down to a set of intersecting issues:

These problems led to beginners being confused by seemingly spurious build failures because Platform would fall out of date with the rest of the ecosystem. Cabal would be unable to find sets of compatible dependencies and say it couldn't build the package.

What non-beginners were doing to avoid these problems was:

1. Install the bare compiler, no Platform

2. Use package database sandboxes for each project

All of these (UX and technical) problems were solved without compromising dependency conflict enforcement via Stackage and Stack.

Speaking hypothetically, if Cargo behaves like Maven or Ivy and pulls in two dependencies who want conflicting versions (1.1 and 1.2, say) of a particular library and just picks a winner, then you'll never see something like this.

reply

---

https://aturon.github.io/blog/2016/07/27/rust-platform/

" Distribute a wide range of artifacts in a single “Rust Platform Package”, including: The compiler, Cargo, rust-lang crates (e.g. std, libc), docs Best-of-breed libraries drawn from the wider ecosystem (going beyond rust-lang crates) Best-of-breed Tools drawn from the wider ecosystem (e.g. rustfmt, clippy, NDKs, editor plugins, lints) Cross-compilation targets “Language bridges”: integration with other languages like JS, Ruby, Python, etc. ... In general, rustup is intended to be the primary mechanism for distribution

"

--- " With the platform metapackage, things are quite different. On the one hand, we can provide an experience that feels a lot like std (see below for more on that). But it doesn’t suffer from the deficits of std. Why? It all comes down to versioning:

    Stability: Doing a rustup to the latest platform will never break your existing code, for one simple reason: existing Cargo.toml files will be pinned to a prior version of the platform metapackage, which is fundamentally just a collection of normal dependencies. So you can upgrade the compiler and toolchain, but be using an old version of the platform metapackage...

allowing metapackage dependencies to be overridden if they appear explicitly in the Cargo.toml file. So, for example, if you say:

[dependencies] rust-platform = "2.7" regex = "3.0"

" --- " Approaching std ergonomics

There are several steps we can take, above and beyond the idea of a metapackage, to make the experience of using the Rust Platform libraries approximate using std itself.

    cargo new. A simple step: have cargo new automatically insert a dependency on the version of the platform associated with the current toolchain.
    Global coherence. When we assemble a version of the platform, we can do integration testing against the whole thing, making sure that the libraries not only compile together, but work together. Moreover, libraries in the platform can assume the inclusion of other libraries in the platform, meaning that example code and documentation can cross-reference between libraries, with the precise APIs that will be shipped.
    Precompilation. If we implement metapackages naively, then the first time you compile something that depends on the platform, you’re going to be compiling some large number of crates that you’re not yet using. There are a few ways we could solve this, but certainly one option would be to provide binary distribution of the libraries through, much like we already do for std. Likely this would work via a general mechanism in Cargo and crates.io.
    No extern crate. Getting a bit more aggressive, we might drop the need for extern crate when using platform crates, giving a truly std-like feel."

---

Versioning and release cadence

I’ve already alluded to “major versions” of the platform in a few senses. Here’s what I’m thinking in more detail:

First off, rustc itself is separately versioned: the Rust Platform 5.0 might ship with rustc 1.89.

With that out of the way, here’s a plausible versioning scheme and cadence:

    A new minor version of the platform is released every six weeks, essentially subsuming the existing release process. New minor releases should only include minor version upgrades of libraries and tools (or expansions to include new libs/tools).
    A new major version of the platform is released roughly every 18-24 months. This is the opportunity to move to new major versions of platform libraries or to drop existing libraries. It also gives us a way to recognize major shifts in the way you write Rust code, for example by moving to a new set of libraries that depend on a major new language feature (say, specialization or HKT).

...

Two-level curation

So far I’ve focused on artifacts that officially ship as part of the platform. Curating at that level is going to be a lot of work, and we’ll want to be quite selective about what’s included. (For reference, the Haskell Platform has about 35 libraries packaged).

But there are some additional opportunities for curation. What I’d love to see is a kind of two-level scheme. Imagine that, somewhere on the Rust home page, we have a listing of major areas of libraries and tools. Think: “Parsing”, “Networking”, “Serialization”, “Debugging”. Under each of these categories, we have a very small number of immediate links to libraries that are part of the official platform. But we also have a “see more” link that provides a more comprehensive list.

That leads to two tiers of curation:

    Tier one: shown on front page; shipped with the platform; highly curated and reviewed; driven by consensus process; integration tested and cross-referenced with the rest of the platform.
    Tier two: shown in “see more”; lightly curated, according to a clearly stated set of objective criteria. Things like: platform compatibility; CI; documentation; API conventions; versioned at 1.0 or above.

By providing two tiers, we release some of the pressure around being in the platform proper, and we provide valuable base-level quality curation and standardization across the ecosystem. The second tier gives us a way to motivate the ecosystem toward common quality and consistency goals: anyone is welcome to get their crate on a “see more” page, but they have to meet a minimum bar first.

...

The rust-lang crates

One small note: our previous attempt at a kind of “extended std” was the rust-lang crates concept. These crates are “owned” by the Rust community, and governed by the RFC process, much like std. They’re also held to similar quality standards.

Ultimately, it’s proved pretty heavy weight to require full RFCs and transfer of control over the crates, and so the set of rust-lang crates has grown slowly. The platform model is more of a “federated” approach, providing decentralized ownership and evolution, while periodically trying to pull together a coherent global story.

However, I expect the rust-lang crates to stick around, and for the set to slowly grow over time; there is definitely scope for some very important crates to be completely “owned by the community”. These crates will likely become part of the platform as well, though there may be some crates that the community maintains that aren’t high-profile enough to appear there. "

---

 "how does curation work? What I envision is something that fits into the Roadmap planning proposal. In a given quarter, we set out as an initiative to curate crates in a few areas – let’s say, networking and parsing...."
 

---

https://github.com/aturon/rfcs/blob/rust-lang-crates/text/0000-rust-lang-crates.md

" First, two additional github organizations are proposed:

    rust-lang-nursery
    rust-lang-deprecated

New cratess start their life in a 0.X series that lives in the rust-lang-nursery. ...If the library team accepts a crate into the nursery, they are indicating an interest in ultimately advertising the crate as "a core part of Rust", and in maintaining the crate permanently. ... Eventually, a nursery crate will either fail (and move to rust-lang-deprecated) or reach a point where a 1.0 release would be appropriate. The failure case can be decided at any point by the library subteam.

If, on the other hand, a library reaches the 1.0 point, it is ready to be promoted into rust-lang proper. To do so, an RFC must be written ... Once a "1.0 RFC" is approved by the libs team, the crate moves into the rust-lang organization ... When a std-destined crate has reached sufficient maturity, the libs subteam can call a "final comment period" for moving it into std proper ... Part of the reason for having rust-lang crates is to have a clear, short list of libraries that are broadly useful, vetted and maintained. But where should this list appear?

This RFC doesn't specify the complete details, but proposes a basic direction:

    The crates in rust-lang should appear in the sidebar in the core rustdocs distributed with Rust, along side the standard library. (For nightly releases, we should include the nursery crates as well.)
    The crates should also be published on crates.io, and should somehow be badged. But the design of a badging/curation system for crates.io is out of scope for this RFC.

...

Plan for existing crates

There are already a number of non-std crates in rust-lang. Below, we give the full list along with recommended actions: Transfer ownership

Please volunteer if you're interested in taking one of these on!

    rlibc
    semver
    threadpool

Move to rust-lang-nursery

    bitflags
    getopts
    glob
    libc
    log
    rand (note, @huonw has a major revamp in the works)
    regex
    rustc-serialize (but will likely be replaced by serde or other approach eventually)
    tempdir (destined for std after reworking)
    uuid

Move to rust-lang-deprecated

    fourcc: highly niche
    hexfloat: niche
    num: this is essentially a dumping ground from 1.0 stabilization; needs a complete re-think.
    term: API needs total overhaul
    time: needs total overhaul destined for std
    url: replaced by https://github.com/servo/rust-url

---

python 'struct'

---

https://en.wikipedia.org/wiki/Ncurses

---

subjective 'most common subset' of J primitives: http://code.jsoftware.com/wiki/User:Devon_McCormick/MinimalBeginningJ

all J primitives: http://code.jsoftware.com/wiki/NuVoc

also could skim the 'J primer' and note all the fns mentioned there (i'm translating here into my own idiom, usually python):

range repmat and reshape ('$' in J) shape rand exp sin cos tan print with numeric precision argument (or environment dynamically scoped variable) + - * div arithmetic negation (and boolean negation) reciprocal list variants of everything (in syntax) elementwise variants of everything (in syntax) plot names (list of names in a namespace) index (into list) append (append item onto end of list) len [] (empty list literal) variable assignment (local and 'global') 9 control structures: if. T do. B end. if. T do. B else. B1 end. if. T do. B elseif. T1 do. B1 elseif. T2 do. B2 end. try. B catch. B1 end. while. T do. B end. whilst. T do. B end. for. T do. B end. for_i. T do. B end. select. T case. T0 do. B0 case. T1 do. B1 fcase.T2 do. B2 case. T3 do. B3 end.

<= < >= >

double halve ^2 sqrt tokenize string number literal (including 'e' eg 1e9, including complex, including infinity, negative infinity (and undefined)) string literal bool literal (in J this is just 0 1) complex conjugate or and gcd lcd nor

note: J's repmat/reshape can even do this:

   2 3 $ 7 8	NB. cycle through atoms to get enough
7 8 7
8 7 8

   3 4 $ 'abcdefghijklmnopqrstuv'
abcd
efgh
ijkl

this list is so far only thru http://www.jsoftware.com/primer/more_primitives.htm

todo add the stuff from the rest of the Primer

related:

https://en.wikipedia.org/wiki/APL_syntax_and_symbols

---

https://twitter.com/fylwind/status/549342595940237312

Learning curves of popular Haskell concepts

low:

high:

---

the APL keyboard:

https://scottlocklin.files.wordpress.com/2013/07/aplkeyb.gif?w=720

---

J has a cool 'generalized outer product' called 'table' (i say 'generalized' b/c it is outer product by any function, not just by multiplication): http://www.jsoftware.com/primer/table_adverb.htm

that page shows how it can be useful when plotting a function

---

https://adambard.com/blog/clojure-batteries-included/

---

https://github.com/bluss/petgraph