Bayle Shanks's website: notes-computer-programming-programmingLanguageDesign-prosAndCons-python

" My own experience with Python, after being a heavy Perl and Ruby user isn't that good. Especially if you are coming from a language like Perl, Python as a language has an interesting trend. At first you feel very happy that things are so simple with it. Then slowly once you learn the language and start getting deeper into it, you try to use it for day to day work. Suddenly you realize that text processing sucks badly, something that is very big problem on Unix based environments. Writing a single regex takes a 10's of lines of exception handling. You try to use it for quick dirty scripting, again the language runs out of steam very soon. Its just too verbose for scripting class.

Now coming to more important reasons. Sacrificing power for readability might be great for newbie's but isn't good experienced programmers. Lack of multiline lambda's, no good support for TOC, GIL, barely there object system(compared to Moose), lack of extensibility, lack of syntax plugins. "

" Consider bash:

    if [ `which foo` == "/usr/bin/foo" ]; then
      do_stuff
    fi

compared to Ruby:

    if `which foo`.strip == "/usr/bin/foo"
      do_stuff
    end

compared to Python:

    import os
    if os.popen("which foo").read().strip() == "/usr/bin/foo":
      do_stuff()

  if [ -e '/bin/bash' ];
  then
      do_stuff
  fi

(I don't know Ruby.)

Python:

  import os.path
  if os.path.exists('/bin/bash'):
      do_stuff()

" lobster_johnson 7 days ago

link

... three core similarities/differences:

Ruby has bashisms like `foo`; Python requires a function call.
Ruby has a nesting syntax similar to bash with if/end; Python is whitespace-sensitive.
Ruby has method calls without parentheses, similar to bash commands, whereas Python requires them.

> Although if your goal is to determine if e.g. bash is located in /bin,

Actually, that was not the goal at all. The goal (which, again, is irrelevant to my point) was to determine if the PATH-resolved executable binary "foo" was in a specific location. Rest assured that I would not use "which" in a real-life app. :-)

(Thanks for commands.getoutput(), my Python is rusty.)

reply "

https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx

" What is async I/O? • Do something else while waiting for I/O • It's an old idea (as old as computers) • With lots of approaches – threads, callbacks, events... • I'll come back to this later Why async I/O? • I/O is slow compared to other work – the CPU is not needed to do I/O • Keep a UI responsive – avoid beach ball while loading a url • Want to do several/many I/O things at once – some complex client apps – typical server apps Why not use threads? • (Actually you may if they work for you!) • OS threads are relatively expensive • Max # open sockets >> max # threads • Preemptive scheduling causes races – "solved" with locks Async I/O without threads • select(), poll(), etc. • asyncore :-( • write your own • frameworks, e.g. Twisted, Tornado, zeroMQ • Wrap C libraries, e.g. libevent, libev, libuv • Stackless, gevent, eventlet • (Some overlap)

Downsides • Too many choices • Nobody likes callbacks • APIs not always easy • Standard library doesn't cooperate

Let's standardize the event loop • At the bottom of all of these is an event loop – (that is, all except OS threads) • Event loop multiplexes I/O • Various other features also common Why is the event loop special? • Serializes event handling – handle only one event at a time • There should be only one – otherwise it's not serializing events • Each framework has its own event loop API – even though the functionality has much overlap What functionality is needed? • start, stop running the loop – variant: always running • schedule callback DT in the future (may be 0) – also: repeated timer callback • set callback for file descriptor when ready – variant: call when I/O done Interop • Most frameworks don't interoperate • There's a small cottage industry adapting the event loop from framework X to be usable with framework Y – Tornado now maintains a Twisted adapter – There's also a zeroMQ adapter for Tornado – I hear there's a gevent fork of Tornado – etc. Enter PEP 3156 and Tulip

I know this is madness • Why can't we all just use Tornado? • Let's just import Twisted into the stdlib • Standardizing gevent solves all its problems – no more monkey-patching – greenlets in the language • Or maybe use Stackless Python? • Why reinvent the wheel? – libevent/ev/uv is the industry standard Again: PEP 3156 and Tulip • I like to write clean code from scratch • I also like to learn from others • I really like clean interfaces • PEP 3156 and Tulip satisfy all my cravings What is PEP 3156? What is Tulip? • PEP 3156: – standard event loop interface – slated for Python 3.4 • Tulip: – experimental prototype (currently) – reference implementation (eventually) – additional functionality (maybe) – works with Python 3.3 (always) PEP 3156 is not just an event loop • It's also an interface to change the event loop implementation (to another conforming one) – this is the path to framework interop – (even gevent!) • It also proposes a new way of writing callbacks – (that doesn't actually use callbacks) But first, the event loop • Influenced by both Twisted and Tornado • Reviewed by (some) other stakeholders • The PEP is not in ideal state yet • I am going to sprint Mon-Tue on PEP and Tulip Event loop method groups • starting/stopping the loop • basic callbacks • I/O callbacks • thread interactions • socket I/O operations • higher-level network operations Starting/stopping the event loop • run() # runs until nothing more to do • run_forever() • run_once([timeout]) • run_until_complete(future, [timeout]) • stop() • May change these around a bit Basic callbacks • call_soon(callback, *args) • call_later(delay, callback, *args) • call_repeatedly(interval, callback,

args) • call_soon_threadsafe(callback, *args) • All return a Handler instance which can be used to cancel the callback I/O callbacks • add_reader(fd, callback, *args) -> Handler • remove_reader(fd) • add_writer(fd, callback, *args) -> Handler • remove_writer(fd) • Not all fd types are always acceptable • fd may be an object with a fileno() method UNIX signals • add_signal_handler(sig, callback,
args) -> Handler • remove_signal_handler(sig) • Raise RuntimeError? if signals are unsupported Thread interactions • wrap_future(future) -> Future • run_in_executor(executor, callback,
args) -> Future • Used to run code in another thread – sometimes there is no alternative – e.g. getaddrinfo(), database connections • Threads may use call_soon_threadsafe() Socket I/O operations • sock_recv(sock, nbytes) -> Future • sock_sendall(sock, data) -> Future • sock_accept(sock) -> Future • sock_connect(sock, address) -> Future • Only transports should use these High-level network operations • getaddrinfo(host, port, ...) -> Future

getnameinfo(address, [flags]) -> Future • create_connection(factory, host, port, ...) -> Future • start_serving(factory, host, port, ...) -> Future • Use these in your high-level code Um, Futures? • Like PEP 3148 Futures (new in Python 3.2): – from concurrent.futures import Future – f.set_result(x), f.set_exception(e) – f.result(), f.exception() – f.add_done_callback(func) – wait(fs, [timeout, [flags]]) -> (done, not_done) – as_completed(fs, [timeout]) -> <iterator> • However, adapted for use with coroutines Um, coroutines? • Whoops, let me get back to that later What's a Future? • Abstraction for a value to be produced later – Also known as Promises (check wikipedia) – Per wikipedia, these are explicit futures • API: – result() blocks until result is ready – an exception is a "result" too: will be raised! – exception() blocks ands checks for exceptions – done callbacks called when result/exc is ready Futures and coroutines • Not the concurrent.futures.Future class! • Nor exactly the same API • Where PEP 3148 "blocks", we must use... Drum roll, please PEP 380: yield-from

I cannot possibly do this justice • The best way to think about this is that yield-from is magic that "blocks" your current task but does not block your application • It's almost best to pretend it isn't there when you squint (but things don't work without it) • PS. @coroutine / yield-from are very close to async / await in C# How to think about Futures • Most of the time you can forget they are there • Just pretend that: data = yield from <function_returning_future> is equivalent to: data = <equivalent_blocking_function> ...and keep calm and carry on • Also forget about result(), exception(), and done-callbacks Error handling • Futures can raise exceptions too • Just put a try/except around the yield-from: • try: data = yield from loop.sock_connect(s, h, p) except OSError: <error handling code> Coroutines • Yield-from must be used inside a generator • Use @coroutine decorator to indicate that you're using yield-from to pass Futures • Coroutines are driven by the yield- from • Without yield-from a coroutine doesn't run • What if you want an autonomous task? Tasks • Tasks run as long as the event loop runs • A Task is a coroutine wrapped in a Future • Two ways to create Tasks: – @task decorator (instead of @coroutine) – f = Task(some_coroutine()) • The Task makes sure the coroutine runs "

codygman 1 day ago

link

Unicode support, concurrency/parallelism, and monkey patching were very big reasons for me using Go rather than python.

aba_sababa 1 day ago

link

The only reason I'm not doing that is because I'm anchored by Python's stats libraries.

visitorx 1 day ago

link

seconded, I doubt many a sane person would use Python for IO bound concurrency, they should definitely be looking at Rust (in the future slightly) and Erlang (currently the defacto standard for easy-of-use concurrency).

reply "

Ixiaus 2 days ago

link

I used gevent at first for a project that needed Async I/O and it worked really well, but then I switched to Erlang and I realized how poor a choice Python is for such tasks. The language really needs to be designed from the start for it (like Go, Rust, Erlang &c... Haskell wasn't designed from start for it but because of it's functional purity, bolting it on was "natural" but - it isn't so for Python IMHO).

craigyk 2 days ago

link

Yeah, if you're going to have to do async to be performant than it better be pretty pervasive throughout all the libraries. Bonus points if the language supports syntax to make async easier as well. Node is beating out Python for server stuff not simply because it is "async", but because it is so much FASTER. The speed of V8 vs. CPython is a big part of that. In fact, vanilla JS doesn't have much to make async programming particularly easy: it has verbose function declaration and no yield mechanisms. Even library-level solutions like promises are merely 'OK'.

Still, it is easier to build a fast server that can handle streams in Node than it is in Python. Async Python? I'll just stick to async JS in that case. "

" nashequilibrium 1 day ago

link

The reason why you getting so many downvotes is because your comment is not just silly but flat wrong. Twisted, Tornado were around before nodejs. There are also async frameworks that make writing async code the same as synchronous code. I like tornado but I am using nodejs for my current app because of the libraries. This is where nodejs really shines, the community and libraries are awesome. Twisted has a lot of libraries but it has so much going on that many developers find it too complex. Tornado is a much simpler async framework to adapt to and allows you to run twisted libraries.

Twisted's inlinecallback's and tornado's gen module gets rid of all the async spaghetti code. This is hard to do with nodejs but I still chose nodejs because the available libraries made my project quicker to develop.

reply "

rdtsc 1 day ago

link

Sorry I didn't express myself correctly (see my reply to akuchling below).

Basically yes, Python had Twisted for years, it had Diesel, Monocle, Tornado, and some other ones. I am aware of those and as you've read my comment you saw that I used Twisted enough to know its ins and outs (5 years).

> There are also async frameworks that make writing async code the same as synchronous code.

Yes there is inlineCallbacks and I used. Node.js also has async (https://github.com/caolan/async). But you don't address the main problem that I raised -- fragmentation of libraries. Python is great because it comes with batteries, and then you can find even more batteries everywhere, _except_ if you use an async framework like Twisted, which, percolates all the way through you API code. Once your socket.recv() returns a Deferred(), that deferred will bubble up all the way to the user interface. So you now you end up searching or recreating a parallel set of libraries.

> Twisted has a lot of libraries but it has so much going on that many developers find it too complex.

It is too complex with too many libraries for those who want to take it up but it is not complex and doesn't have enough libraries if you are in it already -- every library you use has to be Twisted now. That's the danger of inventing a new framework.

Yes it will be standard, but there is already a practical standard -- eventlet and gevent. This is somethin Node.js doesn't have. I will personally take monkey-patching and the danger that my function will context switch inside while doing IO over using Twisted. I saw a practical benefit from it at least. "

pjscott 2 days ago

link

YES. Thank you for saying this. Some of the async code that I maintain uses Twisted and some of it uses Eventlet, and the difference between them is night and day. The code using Eventlet is so much cleaner, so much easier to maintain, and (oddly enough) so much less magical than the Twisted stuff. This was written by the same people, and they're all really good programmers, so the obvious confounding variables are not an issue here. Eventlet and Gevent are just so much better.

Worried about monkey-patching? Then only monkey-patch the parts you need to be asynchronous. Worried about magic that you don't understand? Have a look at the code; the magic is actually pretty straightforward after you've paid a little attention to the man behind the curtain.

If you're interested in async stuff for Python, I urge you to have a look at Eventlet or Gevent. "

eurleif 1 day ago

link

One thing I love about gevent is that you can share code between async and non-async. Most of my project benefits from async IO, but there's one part that needs to use a lot of CPU within a single process. So that part uses multithreaded Jython, the rest uses gevent, the common code is shared, and it all just works.

rdtsc 1 day ago

link

Exactly, I was surprised how in the whole "ideas" mailing list discussion Guido had and in other forums that is dismissed as "meh" not even mentioned.

Discussions quickly turn theoretical and academic. "But you don't know when your green threads will switch, man, so I'll add yields in there for you". Yes, and then also make sure there is a complete universe of libraries.

Python is awesome not just because it is fun to write little hello world examples in (so is Logo), it is awesome because it is easy to GetShitDone?(TM) quickly. The big part of GetShitDone?(TM) quickly is reusing libraries not rewriting everything from scratch.

Using an exotic database for some reason -- great. Found a Python library to interact with it -- great. Oh but my framework is based on Deferreds and this one was written without Deferreds or this one returns Futures. Sorry, go write your own from scratch.

This has been the story of my life for 5+ years search or re-writing Twisted version of already existing libraries.

Now at least just adopt Twisted and go with it if they are going this route. But now, they are 'standardizing' on something new. I think had they done this in 2007, yeah rock on, that would have made sense. They didn't. What saved and kept Python on the back-end during the past 5 or so years was greenlet (eventlet and gevent). Guido is kicking all those people in nuts and saying, "no", we'll do Twisted now (with some changes).

reply "

rdw 1 day ago

link

As someone who's contributed to Eventlet, I've always felt that the only way that it could get over this scary hurdle (and it is legitimately scary) is for it to be integrated within Python itself. Almost all of the weird problems come from fighting with the baked-in assumptions of the Python runtime. Eventlet does try to alleviate the scariness a little bit by allowing you to import "greened" modules rather than changing the global versions, but that has its own problems.

If it were integrated with Python, there would be no monkeypatching, no special magic, it would be just how things work. That said, I'm not at all surprised that Guido doesn't favor a coroutine-based solution; his opposition to general coroutines is as famous as his opposition to anonymous functions. (to clarify: I don't think the @coroutine decorator creates "real" coroutines, any more than generators were already coroutines) "

fzzzy 1 day ago

link

From a certain perspective it is a rational decision. Because the CPython API relies so heavily on the C stack, either some platform-specific assembly is required to slice up the C stack to implement green threads, or the entire CPython API would have to be redesigned to not keep the Python stack state on the C stack.

Way back in the day [1] the proposal for merging Stackless into mainline Python involved removing Python's stack state from the C stack. However there are complications with calling from C extensions back into Python that ultimately killed this approach.

After this Stackless evolved to be a much less modified fork of the Python codebase with a bit of platform specific assembly that performed "stack slicing". Basically when a coro starts, the contents of the stack pointer register are recorded, and when a coro wishes to switch, the slice of the stack from the recorded stack pointer value to the current stack pointer value is copied off onto the heap. The stack pointer is then adjusted back down to the saved value and another task can run in that same stack space, or a stack slice that was stored on the heap previously can be copied back onto the stack and the stack pointer adjusted so that the task resumes where it left off.

Then around 2005 the Stackless stack slicing assembly was ported into a CPython extension as part of py.lib. This was known as greenlet. Unfortunately all the original codespeak.net py.lib pages are 404 now, but here's a blog post from around that time that talks about it [2].

Finally the relevant parts of greenlet were extracted from py.lib into a standalone greenlet module, and eventlet, gevent, et cetera grew up around this packaging of the Stackless stack slicing code.

So you see, using the Stackless strategy in mainline python would have either required breaking a bunch of existing C extensions and placing limitations on how C extensions could call back into Python, or custom low level stack slicing assembly that has to be maintained for each processor architecture. CPython does not contain any assembly, only portable C, so using greenlet in core would mean that CPython itself would become less portable.

Generators, on the other hand, get around the issue of CPython's dependence on the C stack by unwinding both the C and Python stack on yield. The C and Python stack state is lost, but a program counter state is kept so that the next time the generator is called, execution resumes in the middle of the function instead of the beginning.

There are problems with this approach; the previous stack state is lost, so stack traces have less information in them; the entire call stack must be unwound back up to the main loop instead of a deeply nested call being able to switch without the callers being aware that the switch is happening; and special syntax (yield or yield from) must be explicitly used to call out a switch.

But at least generators don't require breaking changes to the CPython API or non-portable stack slicing assembly. So maybe now you can see why Guido prefers it.

Myself, I decided that the advantages of transparent stack switching and interoperability outweighed the disadvantages of relying on non-portable stack slicing assembly. However Guido just sees things in a different light, and I understand his perspective.

  [1] http://www.python.org/dev/peps/pep-0219/
  [2] http://agiletesting.blogspot.com/2005/07/py-lib-gems-greenlets-and-pyxml.html"

-- https://news.ycombinator.com/item?id=5395385

---

(has this been fixed by http://www.python.org/dev/peps/pep-0342/ ? )

http://stackoverflow.com/questions/346365/why-cant-yield-return-appear-inside-a-try-block-with-a-catch

---

> (1) It's a whole lot easier to debug a problem with gevent than with > anything > which uses yield / Deferreds / asyncore / whatever. With gevent, you get a > standard stack trace. With anything else, the "where did this call come > from" > information is not part of the call chain and thus is either unavailable, > or > will have to be carried around preemptively (with associated overhead). >

mb relevant to http://lambda-the-ultimate.org/node/4592 , see my Filtered1 note.

http://eventlet.net/

http://sdiehl.github.com/gevent-tutorial/

http://blog.gevent.org/2010/02/27/why-gevent/

---

While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. " -- https://news.ycombinator.com/item?id=3751510

" ...Some of the async code that I maintain uses Twisted and some of it uses Eventlet, and the difference between them is night and day. The code using Eventlet is so much cleaner, so much easier to maintain, and (oddly enough) so much less magical than the Twisted stuff. This was written by the same people, and they're all really good programmers, so the obvious confounding variables are not an issue here. Eventlet and Gevent are just so much better. "

" I am surprised that gevent was dismissed. Ok, there is also eventlet, if someone doesn't like gevent. Monkey patching is scary? Is it really _that_ scary? Most sane and portable IO code probably runs on that today. Why? Because there is no need to create a parallel world of libraries. Write a test does it pass? Does it handle your use case? I'll take not knowing if my green threads switch tasks and add another green thread lock, instead of doubling my code size with yields, callbacks and futures.

Let's talk about Twisted (sorry couldn't resist WAT video reference). I remember for years searching for parallel libraries to parse exotic protocols. Regular Python library is there, but no, can't use that, sorry. Gotta go find or write one that returns Deferreds. You add a single Twisted module in your code, good luck! -- it ripples all the way to the top through your API and you are doomed being locked into the Twisted world forever.

When gevent and eventlet came around it was like a breath of fresh air. This is what sane concurrent IO looks like in Python:

http://eventlet.net/doc/examples.html

My fear is that many will just say fuck it, I'll just use Go/Rust/Erlang for IO bound concurrent problems. "

eurleif 2 days ago

link

codygman 2 days ago

link

Unicode support, concurrency/parallelism, and monkey patching were very big reasons for me using Go rather than python.

" visitorx 2 days ago

link

reply "

>> "@coroutine / yield-from are very close to async / await in C# 5" "

---

" Maybe it's because I had a very similar feeling about Python (indentation matters? Having to pass self explicitly to methods? No ++? No assignment in if, while, etc.? I hated all this things at first, but after understanding the rationale and using then in real work, it works great!). "

---

1] What does this program do?

    print object() > object()

raverbashing 9 hours ago

link

About your code snippet, it prints a "random" boolean value

You're creating two objects (with random addresses, which affect the __str__ method result, which in turn result in a string comparison that returns False or True)

ptx 6 hours ago

link

  >>> print(object() > object())
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: unorderable types: object() > object()

Yet another reason to upgrade to Python 3! :)

---

"Friday, March 23, 2012 PyCon?: Keynote: Guido Van Rossum Take the survey at svy.mk/pycon2012survey.

Videos are already being uploaded to pyvideo.org.

Guido is wearing a "Python is for girls" T-shirt.

He showed a Python logo made out of foam on a latte. Yuko Honda really did it.

He had some comments about trolls.

Troll, n:

    A leading question whose main purpose is to provoke an argument that cannot be won.
    A person who asks such questions.

Guido said that Python, Ruby, and Perl are exactly the same language from 10,000 feet. They should be our friends. Lisp is not all that different from Python, although you might not notice it. Ruby is not kicking Python's butt in any particular case.

They added unicode literals back to Python 3.3 just to make compatibility with Python 2.7 easier.

"I should have used a real presentation tool instead of Google Docs." (Doh! He's a Googler!)

More and more projects are being ported to Python 3.

People are actually capable of using one code base for Python 2.7 and Python 3.3. There's a helpful library called "Six".

The Python 3 migration is painful now, but it's going to be worth it.

If PyPy? is faster than cPython, why not abandon cPython? No one is using PyPy? in production. PyPy? is still hard for people writing C extensions. cPython is still a great glue system (for Fortran, C, and random binary code). "PyPy? still has a long way to go. Speed is not everything." PyPy? and cPython will live together in slightly different contexts for years. cPython will remain the reference implementation.

People still criticize dynamic typing, saying it's not safe. "The type checking in compilers is actually really weak." For instance, most of the time, a type checker can't tell you if an int represents inches or centimeters. The only way to create good code is to test, audit, hire smart people, etc. Test! A type checking compiler can only find a narrow set of errors; no one has conquered the problem of writing perfect software.

A large program written in a statically typed language will often have a dynamically typed, interpreted subsystem, done via string manipulation, or something else.

Dynamic typing is not inherently inferior to static typing.

Guido hopes people are "doing interesting things with static analysis."

Guido pronounces GIL as "Jill".

"OS level threads are meant for parallel IO, not for parallel computation." Use separate processes for each core. One team was using Python on a machine with 64,000 cores. "You have to be David Beazley to come up with the counterexample."

Guido has complaints about each of the async libraries, but he doesn't have a solution that he recommends. It's not a problem he has to deal with. gevent is interesting. He can't add stackless to Python because it would make other implementations much more complicated or impossible. He doesn't want to put an event loop in the stdlib because not everyone needs it. Some things are better developed outside of Python. He's not a fan of callback-style approaches; he would prefer the gevent approach. He likes synchronous looking code that under the hood does async, without using threads. All these things deserve more attention.

We might want to make Python 4 the new language to solve the concurrent programming problem. He wants to see lots of alternatives.

Introducting a new language to replace JavaScript? is such a hard, political problem.

He was told running Python on Android is "strategically not important." He was asked not to work on it anymore. Other people are going to have to do it.

"I don't think Python is a functional programming language. map, filter, lambda are not enough to make a functional programming language. Python is very pragmatic. It lets you mess around with state, fiddle with the OS, etc. Those things are really hard in a beautiful language like Haskell. When you have to do messy IO in Haskell, all the messiness becomes even messier.

Learn a functional language, get excited about the ideas, and let it impact your Python code. You don't have to do everything with a functional style.

"From data to data" really fits the functional paradigm. You can use that in Python.

He likes functional programming as a challenge and as an idea. However, 30 years from now, we won't all be writing functional programs all the time.

You're much better off not being in the stdlib. If you have a broken API, you can never fix it. Python has a very slow release schedule. It's much better to be third-party. Don't feel like you're not successful unless you're in the stdlib. It's better to own your own release cycle. Things in the stdlib should be things that enable everything else; like a context manager.

Python has reference counting and a garbage collector that resolves cycles, etc. PyPy? is the place to play with better memory allocators. "I'm very happy with cPython's memory allocator."

Let's set our expectations low about adding new features to the language since we just finished Python 3. If you really have a desperate need for an extension to the language, think about using a preprocessor [my term] of some sort. Import hooks can let you do the transformation automatically. Posted by Shannon Behrens at 6:12 PM Labels: pycon2012, python

---

Pxtl 2 days ago

link

This isn't intrinsic to Matrices vs Number-Arrays. The matrix-mulitplication issue is just a mathy version of the plus-as-sting-concat troubles ("Foo: " + 1 + 1 makes "Foo: 11" while 1 + 1 + " Foo" makes "2 Foo"). There are always holy wars about whether "+" should be string-concat because of that.

Both approaches have their merits.

ngoldbaum 2 days ago

link

Check out the Julia pycall library. It allows arbitrary python calls from inside Julia with nice autogenerated bindings from python objects to Julia types.

jamesjporter 2 days ago

link

This already exists as a prototype as part of IJulia, see Leah Hanson's awesome blog post about it here: http://blog.leahhanson.us/julia-calling-python-calling-julia...

The collaboration between the scientific Python and Julia communities in recent months has been awesome to watch.

avodonosov 2 days ago

link

The reason to bet on Julia is disassembling a function? This is a standard feature in Common Lisp (ANSI standardized in 1994)

  CL-USER> (defun f(x) (* x x))
  F
  CL-USER> (disassemble 'f)
  L0
           (leaq (@ (:^ L0) (% rip)) (% fn))       ;     [0]
           (cmpl ($ 8) (% nargs))                  ;     [7]
           (jne L33)                               ;    [10]
           (pushq (% rbp))                         ;    [12]
           (movq (% rsp) (% rbp))                  ;    [13]
           (pushq (% arg_z))                       ;    [16]
           (movq (% arg_z) (% arg_y))              ;    [17]
           (leaveq)                                ;    [20]
           (jmpq (@ .SPBUILTIN-TIMES))             ;    [21]
  L33
           (uuo-error-wrong-number-of-args)        ;    [33]

kisielk 2 days ago

link

This is also possible to a degree in Python, though you only get the bytecode:

    >>> def f(x):
    ...     return x * x
    ...
    >>> import dis
    >>> print dis.dis(f)
      2           0 LOAD_FAST                0 (x)
                  3 LOAD_FAST                0 (x)
                  6 BINARY_MULTIPLY
                  7 RETURN_VALUE

sitkack 2 days ago

link

And the bytecode is just calling polymorphic methods. All the real work is done in the object implementations of type(x). I was very bummed years ago to realize how shallow the bytecode representation in Python is. There is no sub-terpreter, just C.

steveklabnik 2 days ago

link

You can get the bytecode in Ruby too: http://www.ruby-doc.org/core-2.1.0/RubyVM/InstructionSequenc...

http://stevenloria.com/python-best-practice-patterns-by-vladimir-keleshev-notes/

http://sahandsaba.com/thirty-python-language-features-and-tricks-you-may-not-know.html

" Thoughts on Go

As the adage says, programmer time is much more valuable than processor time. So you should use Go.

    Go is a high-level language. It is easy to write in just like Python.
    It has a rich standard library just like Python.
    Go’s unit testing framework is phenomenal.
    Unlike Python, Go is statically typed, which makes refactoring large codebases unbelievably easier.
    Like Python, Go supports a form of duck-typing, through interfaces. Interfaces are the bees knees.
    Like Python, Go is garbage collected.
    Unlike Python, Go frees you from worrying about distinctions between synchronous and asynchronous calling conventions or threadpool switching.
    Like Twisted, Go is evented for I/O, but unlike Twisted, Go stack traces are much more manageable and Go’s profiling tools provide much greater clarity about execution order.
    Unlike Python, Go is fast, compiled, and leaves the runtime execution to only the work that needs to be done.

It’s a shame Go didn’t exist when Space Monkey started. We’re very lucky to have it now.

" article on dumping Python for Go

" Regarding lexical scope, Python has ridiculous gotchas too http://stackoverflow.com/questions/5218895/python-nested-fun... "

http://www.toptal.com/python/top-10-mistakes-that-python-programmers-make

one guy's suggestions for python: http://c2.com/cgi/wiki?PythonThreeThousand

http://docs.python-guide.org/en/latest/writing/style/

http://web.archive.org/web/20100310233706/http://www.python.org/dev/culture

" Compare these two programs … which is more human-readable?

First, in C++:

include <iostream.h> void main() { cout 1> append_one()

    >>> append_one()

    >>> append_one()

---

pyre 1 day ago

link

Python doesn't seem to agree with you :^)

  In [1]: a = []

  In [2]: a.append(a)

  In [3]: a
  Out[3]: [[...]]

  In [4]: a[0]
  Out[4]: [[...]]

  In [5]: a[0][0]
  Out[5]: [[...]]

  In [6]: a[0][0][0]
  Out[6]: [[...]]

  In [7]: a[0][0][0][0]
  Out[7]: [[...]]

  In [8]: a.append(a)

  In [9]: a
  Out[9]: [[...], [...]]

  In [10]: a[0][1][0] is a
  Out[10]: True

  In [11]: id(a)
  Out[11]: 4547140064

  In [12]: id(a[0][1][0])
  Out[12]: 4547140064

sophacles 23 hours ago

link

Yeah, the whole infinite loop thing. Wasn't fully thinking when I wrote my reply. Good catch.

---

https://news.ycombinator.com/item?id=8001337

Python is now the most popular introductory language at top U.S. universities (acm.org)

blahedo 3 days ago

link

This methodology is suspect, and the headline is extremely deceptive. Note that Guo has conflated CS0 (meaning: CS for non-majors) with CS1 (meaning: intro to CS major). He is upfront about this fact and he does it intentionally, but comments about "first officially-sanctioned exposure in college" and such make it sound like he's unaware that most CS majors do not ever take a CS0 course; and among instructors of CS, phrases like "introductory language" without further qualification is usually understood as referring to CS1.

If we then look at the table of actual data to filter a little bit, the story changes. There are certainly some schools, big ones, that use Python in CS1. But a lot of the Python representation is coming from CS0 at schools where CS1 is taught using Java.

(I can also clarify one of his other comments: he says of the AP exam using Java that "it's unclear whether that will change anytime soon." This is untrue. It will not change in the next 3-4 years at the very minimum---and that would be the time frame if the College Board said right now today that a change was happening. In fact, though, although the idea's been mentioned, there has been no study or apparent motion on that front, so a more realistic earliest-date-of-change would be around 2019 or 2020. And I wouldn't count on it, given how many CS1 courses are still taught in Java.)

pgbovine 3 days ago

link

Author here. Thanks for your comments blahedo. I'm a bit short on time right now, but I will convert some parts of this response into more coherent form and edit my article accordingly to clarify.

I'll address each of your paragraphs in turn:

1.) Re: CS0 vs. CS1. Like you mentioned, I was very upfront about the methodology for this analysis. And I am very aware that most CS majors do not ever take a CS0 course. But please also be aware that there are a minority of CS majors that do take CS0 before moving onto CS1, usually because they feel ill-equipped to take CS1 right away. I harp on this point because even though this is a minority population, it's a very important one since those are students who, without CS0, would not major in CS. These students are more likely to be female or from underrepresented minority groups. On a related note, a larger and larger fraction of non-CS majors are taking CS courses because they recognize the value of programming and computational thinking. (Again, these non-CS majors are more likely to be female and from underrepresented minority groups.) So I would argue that CS0 is just as important as an "introductory programming" course as CS1, if not more important, due to the rise of the non-software-engineers-who-want-to-learn-programming population. [Edit: And I think that a CS department's choice of language to offer to the CS0 student population is still very significant, and a decision that is not made hastily.]

2.) Re: Python CS0 and Java CS1. You are absolutely right, and in my original article, this sentence appears in the discussion section: "Some schools have fully switched over to Python, while others take a hybrid approach, offering Python in CS0 and keeping Java in CS1."

3.) Re: AP exams using Java. Thanks for the clarification. I will remove "it's unclear whether that will change anytime soon." from the article.

[Also, as to cyorir's comment that the chosen data may have been cherry-picked, please rest assured that I tried my best NOT to cherry pick. If I had cherry picked, wouldn't I have made Python look even better? ;) Any incomplete data is strictly due to human error or oversight on my part, and I gladly welcome any correction requests via email.]

---

candeira 3 days ago

link

I work at Monash University in Melbourne, Australia. Last year we moved our foundational course in data structures and algorithms from Java to Python, with great success. This is a course that used to be taught in C, then moved to Java. I tutored it as a Java course, and was in charge of adapting the tutorial (classroom exercises) and laboratory (programming assignments) materials when we started using Python to teach the unit.

The main content of the course hasn't changed. Complexity, Big O notation, sorting and searching are introduced in the first two or three weeks. Students work through the usual implementations of arrays, linked lists, stacks and queues (both array and linked-list based), trees, heaps, tries, etc. All these data structures are explained first as chunks of data with external functions for the operations; then a second time as objects with methods for operations, once object orientation is introduced in week 5 or 6. Iterators are introduced as an abstract interface to list-, array- and tree- walking. Where relevant, we explain every operation twice, showing the iterative and recursive approaches. The course ends with two weeks of lower level work, where the students hand-compile pseudo-Python to working MIPS assembly.

The move to Python gained us, among other things, the freedom not to have to explain Java's generics and access modifiers. Many of our students have never programmed with a text editor before our unit, and dropping them into the big boilerplate world of Java micro-management was not good for them or for the course. We used to spend too much time on syntax and other details ("native types are like this, reference types are like htat") that we should have been spending on data structures and algorithms.

With Python, we even have time to explain idiomatic Python iterators and generators, and to discuss Python scoping rules (which come up again when we work on local variables in their hand-compilation tasks).

mandalar12 2 days ago

link

I have a different experience tutoring (exercises and labs) in an engineering school in France for programming / CS beginners.

We moved from Java to python this year and I miss a few things. Due to the absence of explicit typing the students don't bother to understand types anymore and it leads them to a lot of mistakes and misunderstandings: confusion between simple types, lists, dictionaries, instances of a user created class, etc. Besides I think the verbose and rigid syntax of Java forced them to understand what their wrote and for a first language that was a good thing.

Overall I found that since it is easier to write code in python they rush to writing anything without understanding the algorithmic problem first. Thus they are less able to decompose a problem in sub-problems and write atomic functions to solve each of them.

Note that I think teaching python as a first language is a viable option but in our case the course needs to be intensively rewritten and the labs adapted. For instance my lattest point about algorithms is not a problem with the language: it could be resolved by having a part of the lab being pure algorithmics and then an implementation part.

bicubic 3 days ago

link

I'm so happy to hear that!

I graduated from Monash Clayton a few years ago when the Java train was in full effect and the focus on Java was a big source of frustration for me. I distinctly remember an algorithms & data structures assignment about Markov chains which had to be done in Java. I spent a good 40 minutes writing what was effectively boilerplate to load the provided files into a meaningful representation and then just snapped and did the entire assignment in Python in maybe 15 minutes. Its crazy how much Java gets in your way for small projects.

Even though it was made clear that Python submissions won't be accepted, my tutor was very understanding and gave me full marks after I stepped him through the code. If that tutor was you, I owe you.

---

jarrett 3 days ago

link

My take on Ruby vs Python is this:

While Python has a healthy web dev ecosystem, Ruby's feels much larger to me. That's almost certainly because Rails is so wildly popular. And Rails is an excellent, mature framework. So for web dev, I would consider Ruby the winner.

Python is the clear winner for scientific computing. That's not really due to anything inherent in the language. It's an ecosystem thing. If you were using Fortran before, you might be working in a problem domain where Python dominates.

Both are excellent for miscellaneous scripting work. E.g. reading in a CSV file and doing something with each row; batch-processing a bunch of images; renaming 1000 files according to some ruleset; gathering some system data and sending nightly status emails.

In terms of syntax and features, they're very very similar. Python has meaningful whitespace, which you may like or dislike. (I think it's good for enforcing proper formatting, but you're free to disagree.) Ruby has multiple ways of expressing the mathematical concept of a function (methods, blocks, and procs), which has its pros and cons. Both have metaprogramming facilities, though I find Ruby's more pleasant. If I remember correctly, it was in large part the metaprogramming that made DHH pick Ruby for Rails.

toyg 3 days ago

link

> While Python has a healthy web dev ecosystem, Ruby's feels much larger to me.

It's funny because I actually feel the opposite: Ruby is mostly a niche language for the web community, whereas Python is now mainstream in so many different areas (graphics, scientific, sysadmin, financial...).

I've yet to see a non-webdev tool embedding Ruby as a scripting language, while Python is now featured in office suites, spreadsheets and enterprise software. Anywhere simple and clear syntax is more valued than metaprogramming features, Python is likely to appear at some point.

jarrett 3 days ago

link

I think we have almost the same opinion, actually. I said Ruby's "web dev ecosystem...feels much larger to me." I agree that Ruby's strongest niche is web dev. And I think it has an edge of Python there. Outside web dev, I don't see Ruby dominating any particular niche.

pmontra 3 days ago

link

Ruby is perhaps more dominant than Python in web development. Not that Rails is necessarily better than Django, but it seems there are more Rails developers than Django ones. However Ruby seems to be pretty much confined to web development and deployment tools. It also makes great DSLs thanks to its block syntax. However in general Python is more popular than Ruby.

One reason is that Python is used for system programming (e.g: the Linux apt system), desktop applications, big data (thanks to numpy) so you might find it a better investment for general purpose software development.

Another reason, which maybe explains the first one, is that Python looks more similar to C++/Java. Basically it's a simplification of those languages (the {} pair traded for a single colon, no type declarations, etc) plus some handy shortcuts. Ruby is from a somewhat different lineage (Smalltalk) and it shows in many places (e.g. the do

variable

... end blocks). It can be used to write more natural language looking programs and combining that with some metaprogramming sometimes both amazes and unsettles programmers that come close to the language for the first time.

I suggest that you learn a little of both of them, working on the problem you want to solve, and decide by gut feeling what you would like to work with. I went the Ruby way because I can't stand some of the syntax of Python, not that Ruby is perfect.

simonw 3 days ago

link

I think C makes a great second language. Use Python to teach if statements, variables, control structures etc - then later use C to teach low-level algorithms, pointers and how computers actually work.

I don't think there's any harm in learning Python first. I think they may be harm in learning C first - it's much more likely to frustrate people and potentially even put them off programming.

This is the same reason I dislike Java as a teaching language: having to tell people "don't worry about what public static void main(String[] args) means just yet" when teaching "hello world" isn't a great introduction to programming.

---

why have nested list comprehensions? to avoid having to use nested loops instead of list comprehensions:

" Some common excuses for not using a list comprehension:

    You need to nest your loop. You can nest entire list comprehensions, or just put multiple loops inside a list comprehension. So, instead of writing:

        words = ['her', 'name', 'is', 'rio']
        letters = []
        for word in words:
            for letter in word:
                letters.append(letter)

    Write:

        words = ['her', 'name', 'is', 'rio']
        letters = [letter for word in words
                          for letter in word]

    Note that in a list comprehension with multiple loops, the loops have the same order as if you weren't making a list comprehension at all." -- http://lignos.org/py_antipatterns/

---

" Loops

Generally speaking, in Python the scope of a name is wider than one might expect given other languages. For example, in Java, the following code will not even compile:

Get the index of the lowest-indexed item in the array that is > maxValue for(int i = 0; i < y.length; i++) { if (y[i] > maxValue) { break; } } Oops, there is no i here processArray(y, i);

However, in Python the equivalent will always compile and often produce the intended result:

for idx, value in enumerate(y): if value > max_value: break

processList(y, idx)

This will work in all cases except when y is empty; in that case the loop never runs and the call to processList will raise a NameError? because idx is not defined. If you use Pylint, it would warn you about "Using possibly undefined loop variable idx."

The solution is to always be explicit and set idx to some special value before the loop, so you know what to look for if the loop never runs. This is called the Sentinel Pattern. So what value should you use for a sentinel? Starting with C or earlier, back when int ruled the Earth, a common pattern for a function that needed to return an "expected error" result was to return -1. For example, let's say you want to return the index of an item in a list:

def find_item(item, alist): # None is arguably more Python than -1 result = -1 for idx, other_item in enumerate(alist): if other_item == item: result = idx break

    return result

In the general case, None is a better sentinel of choice in Python, even if it isn't used consistently by Python's standard types (e.g., str.find). See the style section for recommended ways to test for None. " -- http://lignos.org/py_antipatterns/

---

the outer scope is like a global namespace in the ways that it can lead to bugs:

" bugs like the following:

import sys

See the bug in the function declaration? def print_file(filenam): """Print every line of a file.""" with open(filename) as input_file: for line in input_file: print line.strip()

if __name__ == "__main__": filename = sys.argv[1] print_file(filename)

If you look closely, you'll see that the definition of print_file names its argument filenam, but the body of the function references filename. However, this program works just fine. Why? In print_file when a local variable named filename isn't found, the next step is to look at the global namespace. Since the code that calls print_file lives in the outer scope (even though it's indented), the variable filename declared there is visible to print_file.

So, how do you avoid problems like this? First, don't set any values in the outer scope that aren't IN_ALL_CAPS. Things like parsing arguments are best delegated to a function named main, so that any internal variables in that function do not live in the outer scope.

This also serves as a reminder about the global keyword. You do not need the global keyword if you are just reading the value of a global name. You only need it if you want to change what object a global variable name refers to. See this discussion of the global keyword on Stack Overflow for more information. " -- http://lignos.org/py_antipatterns/

" Testing for empty

If you want to check whether a container type (e.g., list, dictionary, set) is empty, simply test it instead of doing something like len(x) > 0:

numbers = [-1, -2, -3]

This will be empty positive_numbers = [num for num in numbers if num > 0] if positive_numbers:
Do something awesome

If you want to store this result somewhere, use bool(positive_numbers); bool is what is called to determine the truth value of the target of if. " -- http://lignos.org/py_antipatterns/

" Testing for None

As I mentioned previously, None makes a good sentinel value. How should you check for it?

If you are specifically testing for None and not just other things that evaluate as False (e.g., empty containers, 0) use is:

if x is not None: # Do something with x

If you are using None as a sentinel, this is the desired pattern; you want to distinguish None from 0, for example.

If you are just testing for whether you have something useful to work with, a simple if pattern is usually good enough:

if x: # Do something with x

For example, if x is expected to be a container type, but could be None based on the result of another function, this handles all the cases that you care about at once. Just be aware that if you change the value that goes into x such that True or 0.0 are useful values, this may not behave the way you want. "

-- http://lignos.org/py_antipatterns/

---

http://lignos.org/py_antipatterns/ summary:

use for (which acts like foreach), zip, enumerate instead of using a for..range to loop over index values
use list comprehensions instead of loops
if you need to repeatedly check for membership in a list, it will be faster to use a set instead
loop variables are available after loop termination. But if the loop didn't run even once, they may be unassigned, so using them this way is dangerous unless you preassign them to a default before the loop.
variables assigned to in the outer scope are available in the inner scope. So don't do much in the outer scope. Use vars in all caps for outer scope names that are intended to be used everywhere.
globals can be read from, just not written to, even without the 'global' keyword.
use 'if container:' instead of 'if len(container) > 0:'
use 'if x:' when possible, e.g. unless you want to distinguish between 0, emptiness, False, and None (bayle: not sure i like this one)

summary from the discussion https://news.ycombinator.com/item?id=8008944 :

beware of bare 'except:' statements (these even catch cntl-C!)
watch out for mutable default function arguments, as they are only initialized once

---

hcarvalhoalves 1 day ago

link

Split without argument is equivalent to:

    >>> filter(None, " quick   hack for  split".split(" "))
    ['quick', 'hack', 'for', 'split']

why does Python not re-initialize mutable default function arguments?

cabalamat 1 day ago

link

> Mutables as default function/method arguments

It would really make sense to change the semantics of Python to fix this issue.

bkeroack 1 day ago

link

Change them how, to no longer have functions be first class objects? The behavior of mutable default arguments is clear if you know how Python treats function objects. Any "fix" would handicap the language.

jcampbell1 1 day ago

link

Can you clarify?

    def foo(default_arg = []):

Why can't that just be shorthand for:

    def foo(default_arg = ParamNone):
        if default_arg == ParamNone:
            default_arg = []

How would that break first class functions?

dalke 1 day ago

link

As a minor point, use "default_arg is ParamNone?", since "==" probably won't do the right thing.

What breaks is something like:

  def foo(default_arg = slow_f()):
    pass

Under the shorthand gets turned into:

  ParamNone = object()

  def foo(default_arg = ParamNone):
    if default_arg is ParamNone:
        default_arg = slow_f()
    pass

This is fine, since everyone would know that the shorthand means to not put slow code there. Instead, people will start writing it as:

  _foo_arg = slow_f()
  def foo(default_arg = _foo_arg):
    pass

Of course, then what happens with:

  _foo_arg = slow_f()
  def foo(default_arg = _foo_arg):
    _foo_arg = 5

? Under expansion it becomes:

  _foo_arg = slow_f()
  def foo(default_arg = ParamNone):
    if default_arg is ParamNone:
        default_arg = _foo_arg
    _foo_arg.add(5)

This violates Python's scoping rules, because _foo_arg is now being used in local scope instead of global scope. Eg:

  >>> def f(x=None):
  ...   if x is None:
  ...     x = spam
  ...   spam = 3
  ... 
  >>> spam = 9
  >>> 
  >>> f()
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 3, in f
  UnboundLocalError: local variable 'spam' referenced before assignment

Which means you now need a new scoping rule, just to handle default parameters without making things more confusing.

It also turns what was a simple O(1) offset into a precomputed list into a globals() lookup for many cases.

---

geertj 1 day ago

link

Agreed on all counts. However I do find myself using mutables as default arguments sometimes because the generated documentation is clearer.

For example, this is a real method in one of my projects:

  def listen(self, address, ssl=False, ssl_args={}):
      pass

I like the way this turns up in the docs because it's immediately clear that ssl_args needs to be a dict. Otherwise I have to describe it in words.

jamiis 1 day ago

link

You should not ever use mutable default args. The documentation benefit is drastically outweighed by the potential for nasty bugs. http://pythonconquerstheuniverse.wordpress.com/2012/02/15/mu...

kunstmord 1 day ago

link

In case you're using Python 3, why not write ssl_args: dict, ssl: bool=False?

---

sophacles 1 day ago

link

The only thing I disagree with is "use nested comprehensions" thing. In my mind: x = [letter for word in words for letter in word]

is inside-out or backwards or backwards. I want the nested for being the less specific case:

   x = [letter for letter in word for word in words]

makes more sense in my mind.

(It's also my first answer to the "what're some warts in the your language of choice).

njharman 1 day ago

link

I'm in the camp that if your list comp needs more than one for clause, it's complicated enough to be broken out into actual for loop.

---

jules 1 day ago

link

Even clearer:

    x = [for word in words: for letter in word: letter]

This also has the advantage of being readable left to right without encountering any unbound identifiers like all other constructs in Python.

---

ggchappell 1 day ago

link

This is a nice little article, but I wonder about some of the design decisions. In particular:

> The simplifications employed (for example, ignoring generators and the power of itertools when talking about iteration) reflect its intended audience.

Are generators really that hard? (Not a rhetorical question!)

The article mentions problems resulting from the creation of a temporary list based on a large initial list. So, why not just replace a list comprehension "[ ... ]" with a generator expression "( ... )"? Result: minimal storage requirements, and no computation of values later than those that are actually used.

And then there is itertools. This package might seem a bit unintuitive to those who have only programmed in "C". But I think the solution to that is to give examples of how itertools can be used to create simple, readable, efficient code.

---

omegote 1 day ago

link

Point 3 of the iteration part is not good advice. With [1:] you're making a copy of the list just to iterate over it...

omaranto 1 day ago

link

You're right. I still wouldn't recommend looping over indices, but rather using itertools.islice(xs, 1, None) instead of xs[1:].

zo1 1 day ago

link

You're right... But at least it's a shallow copy.

---

wodenokoto 1 day ago

link

I use python for datamining, and most of my work is done exploring data in iPython.

> First, don't set any values in the outer scope that > aren't IN_ALL_CAPS. Things like parsing arguments are > best delegated to a function named main, so that any > internal variables in that function do not live in the > outer scope.

How do I inspect variables in my main function after I get unexpected results? I always have my main logic live in the outer scope because I often inspect variables "after the fact" in iPython.

How should I be doing this?

msellout 16 hours ago

link

I'd try to test smaller chunks of code for validity. If any block of code is longer than 12 lines, I get nervous that I don't understand what it's doing. Refactor your code into functions as you confirm the code behaves as expected in the interpreter.

It's very difficult to write automated tests when all logic is in outer scope rather than chunked into functions.

hyperion2010 1 day ago

link

If you are using the interpreter directly then that particular bit of advice is hard to follow since you basically live in global all the time. For that reason I would say that this advice applies mainly to .py files.

LyndsySimon? 1 day ago

link

Agreed.

There's a big difference between "scripting" and "writing software" in terms of best practices.

If you're writing some ETL scripts in an IPython notebook, it would be overkill to encapsulate everything to keep your global scope clean.

---

noobermin 1 day ago

link

Question, how do you over multiple long lists (in python 2) especially if zip itself takes a long time to zip them, for example.

takeda 1 day ago

link

You use Python 3.

J/K, while this is technically a limitation of Python 2, there actually is izip in itertools package which is a generator and works in similar way to zip in python 3.

---

orf 8 hours ago

link

I switched to Python 3 this year and I haven't looked back -there are some niggles, but overall its a great improvement.

A great feature that's not really talked about is the __prepare__ function in metaclasses: you can supply a custom type that stores all class members. You could whip up multiple-dispatch using this (it lets you handle classes with duplicate property names) in conjunction with signature annotations, which I think is pretty neat.

---

"...the frequent complaints I hear about `len()` being a function rather than a method on `list` and `str` types." -- https://news.ycombinator.com/item?id=8187158

---

philh 2 hours ago

link

    >>> original = 42
    >>> class FooProxy:
    ...  def __getattr__(self, x):
    ...   return getattr(original, x)
    ...
    >>> proxy = FooProxy()
    >>> proxy
    42

For anyone else who found this as confusing as I did ("wtf, how can proxy actually be 42?"), what's going on here is that it's calling `proxy.__repr__()` when it attempts to display `proxy`, which in turn calls `42.__repr__()`. (Similarly, `proxy + 1` calls `42.__add__(1)`.)

---

GnarfGnarf? 1 hour ago

link

I'm learning Python and I must say it is a wonderful language. Being able to concatenate strings by simply saying "a + b" is a great productivity boost (coming from C++). Python libraries are powerful. I can read an Excel spreadsheet with one line of code. I can create plots in PDF format with a half-dozen lines. Amazing!

However, I am disappointed with the difficulty of turning a program into a Windows EXE. I wrote a small program (couple of thousand lines), tried Py2exe which failed to handle the Numpy (or Matplotlib, I forget which) imbroglio.

PyInstaller? works, except that the EXE is 85MB, and takes one minute to start up. Not practical for customer distribution. I can't expect my customers to install the Python runtime. In contrast, my 500KLOC C++ program, with all its third-party libraries, is 19MB. Yes, I know, Python needs everything including the kitchen sink. Still, 85MB is not practical.

Too bad. Not all programs run on a server.

http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/ talks about this in Python a bit (search for "immutability of builtin types"). Some pros:

disallow confusing monkeypatching of buildin types
allow the interpreter to statically optimize operations on builtin types, instead of dynamically look them up

---

that essay, http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/ , also talks about how atomicity guarantees in the builtins lead to not using dynamic attribute lookup in their implementations (because any dynamic lookup could trigger eg a lazy import), which leads to reimplementations of things rather than delegation:

" A good example are collections. Lots of collections have convenience methods. As an example a dictionary in Python has two methods to retrieve an object from it: __getitem__() and get(). When you implement a class in Python you will usually implement one through the other by doing something like return self.__getitem__(key) in get(key).

For types implemented by the interpreter that is different. The reason is again the difference between slots and the dictionary. Say you want to implement a dictionary in the interpreter. Your goal is to reuse code still, so you want to call __getitem__ from get. How do you go about this?

A Python method in C is just a C function with a specific signature. That is the first problem. That function's first purpose is to handle the Python level parameters and convert them into something you can use on the C layer. At the very least you need to pull the individual arguments from a Python tuple or dict (args and kwargs) into local variables. So a common pattern is that dict__getitem__ internally does just the argument parsing and then calls into something like dict_do_getitem with the actual parameters. You can see where this is going. dict__getitem__ and dict_get both would call into dict_get which is an internal static function. You cannot override that.

There really is no good way around this. The reason for this is related to the slot system. There is no good way from the interpreter internally issue a call through the vtable without going crazy. The reason for this is related to the global interpreter lock. When you are a dictionary your API contract to the outside world is that your operations are atomic. That contract completely goes out of the window when your internal call goes through a vtable. Why? Because that call might now go through Python code which needs to manage the global interpreter lock itself or you will run into massive problems.

Imagine the pain of a dictionary subclass overriding an internal dict_get which would kick off a lazy import. You throw all your guarantees out of the window. Then again, maybe we should have done that a long time ago. "

an insightful (but perhaps too negative) description about why Python uses these slots:

hosay123 7 hours ago

link

This post is surprisingly confused, it is phrased as a complaint about the language, then immediately degrades into CPython implementation specifics that have little bearing on the usability of the language itself.

Ronacher should also know better than to post microbenchmarks like the one provided here, especially without corresponding (C) profiler output. At the C level, slots allow the implementation constant-time access to the most common code paths for an object, and especially when you have C code calling other C code via the type system (IMHO the primary use for Python, and still its strongest use case), "interpreter overhead" is reduced to a few extra memory indirection operations.

In the alternative world, sure, perhaps some microbenchmark may behave faster, but now systemically, and for e.g. "reduce(operator.add, range(1000))" requires more hash table lookups than I can count.

Python is all about providing a lightweight way to compose bits of fast code (the kernel, network stack, NumPy?, MySQL?, whatever). Unfortunately somewhere along the way solutions like Django got popular, which are almost the antithesis to this old viewpoint. Ronacher seems to be advocating that we should punish the CPython implementation's traditional strong areas in favour of.. actually, he didn't even describe any vision that was impacted by his complaints. He just seems to want the CPython implementation to suffer.

Perhaps his complaint about new-style __getattribute__ would be better suited as a bug report, it seems the only substantial observation made about the language itself in this post.

my summary: Python's design goal is to be a lightweight way of composing C code, not to be a 'leakless abstraction' extremely dynamic language.

---

" You mention "a good async story" -- that has already been addressed with asyncio. A better multi-processing API has also been addressed in the concurrent.futures module. It's true that "multi-core" is still unsolved in Python, but most practitioners work around this problem by just using process-oriented parallelism instead.

Nonetheless, the frequent complaints about multi-core haven't fallen on deaf ears; they will probably come next! It's pretty much the primary focus of the pypy project and recent development work has begun on a Software Transactional Memory implementation that would remove the GIL.

GvR? at PyCon? 2014 said that he thinks pypy is the future of core Python development -- and that at some point in the future, pypy might become the reference implementation rather than CPython. " -- https://news.ycombinator.com/item?id=8187180

---

shadowmint 7 hours ago

link

Well... I think javascript is a terrible broken language, but you can't deny that it's been chosen as the flagship language by Microsoft, or that at least 3 major players (Microsoft, Google, Apple) are investing serious time and money in the runtimes for it; or that those runtimes are actively deployed on the future on computing: mobile devices.

As a full time python developer I'm seriously concerned about how viable it is as a platform going into the future; this 'everything is fine' attitude is the problem.

WAKE UP.

Python is not doing fine.

It's growing as the target for scientific computing (which is great~), but I'd argue that's masking a downturn in the use of python for serious software engineering tasks, where the people who traditionally used it (webdevs, backend system tool makers, like disqus) are turning to tools with better performance and distribution tooling (like Go, node-webkit, etc) and less drama (py3).

rattray 7 hours ago

link

Thanks; I should clarify.

I agree Python is, in my subjective opinion, the best language spec and community in programming. It's simply the most fun, productive environment.

But people are leaving; important people and newcomers and loyal stalwarts. Im on the go right now so can't access links, but there have been a handful of posts here in the past 6mo or so of prominent members leaving, or sharing their disenchantment. Many other "I'm moving to go" posts. And the velocity of npm/go are undeniable (look at the stats). Remember, npm is largely for the server, where js shouldn't have an advantage.

I love Python. I want it to win. Maybe "lose" is too strong of a word, but it's certainly slipping.

pixelmonkey 2 hours ago

link

I am reminded of Yeats:

    Things fall apart; the centre cannot hold;
    Mere anarchy is loosed upon the world,
    The blood-dimmed tide is loosed, and everywhere
    The ceremony of innocence is drowned;
    The best lack all conviction, while the worst
    Are full of passionate intensity.

zak_mc_kracken 6 hours ago

link

I see a lot of Python developers switching to Go, and it's hard to deny that Python 3's adoption has been... underwhelming.

I think the future of programming languages clearly belongs to statically typed languages, and Python will not be part of this future unless it evolves in that direction.

vhost- 52 minutes ago

link

I was very, very critical of Python programmers switching to Go for a while.

I recently got fed up with python and have been doing C, Go and Lua instead. I've started to rediscover programming and I'm enjoying myself much more. Go is very addicting.

It all started when I tried to get one of my projects that requires a lot of concurrent IO working in both Python 2.7 and 3.x. Gevent, of course only works in Python 2, so I tried to switch to the asyncio family. This was a disaster.

I'm sad to see it go, but I've decided to mostly drop Python. I'll probably still use it for Flask apps (I seriously love Flask). But anything that requires heavy lifting will probably be done in Go or C.

---

the_mitsuhiko 7 hours ago

link

> We have PyPy? now; why not use the tremendous advances there in the next Python?

Because PyPy? as cool as the project is, is terrible to develop on unless you are a PyPy? person. I would consider myself a reasonable programmer but PyPy? makes me go crazy. Slow iteration times, super complicated code.

wyuenho 6 hours ago

link

compiling pypy also takes about half a day...

rguillebert 6 hours ago

link

your days must be really short, it takes an hour on my machine

---

notastartup 1 hour ago

link

I moved from Javascript to Python 2.7 and haven't looked back since. I now strictly use Python in all of my projects, Javascript only because browser doesn't support it.

I wish that Python would introduce a few things from the Javascript world. Easy async, and a real-time Meteor like framework.

http://lucumr.pocoo.org/2014/8/24/revenge-of-the-types/

https://news.ycombinator.com/item?id=8219506

(current-kill 0)

---

http://nothingbutsnark.svbtle.com/how-to-argue-for-pythons-use

---

the Go/Python 3 comparison shows that Python can accomplish solutions in less code than Go every time http://benchmarksgame.alioth.debian.org/u64q/benchmark.php?test=all&lang=python3&lang2=go&data=u64q

---

i asked a friend why he didn't like python, i probably forgot some of his reasons, but generally they were second-order annoyances:

significant whitespace
single-line constraint on anonymous functions
library dependency (and possibly packaging/deployment) annoyances (i dont know exactly what his annoyances were here)
python version 2 vs version 3

btw this same friend used to like Node.js but now he's switched to Scala, and he loves it. He's now a big fan of 'functional, static, immutable'.

---

" I should mention that I have some experience in this area: Google's App Engine (to which I currently contribute most of my time) provides a "secure" variant of Python that supports a subset of the standard library. I'm putting "secure" in scare quotes here, because App Engine's security needs are a bit different than those typically proposed by the capability community: an entire Python application is a single security domain, and security is provided by successively harder barriers at the C/Python boundary, the user/kernel boundary, and the virtual machine boundary. There is no support for secure communication between mutually distrusting processes, and the supervisor is implemented in C++ (crucial parts of it live in a different process).

In the App Engine case, the dialect of the Python language supported is completely identical to that implemented by CPython. The only differences are at the library level: you cannot write to the filesystem, you cannot create sockets or pipes, you cannot create threads or processes, and certain built-in modules that would support backdoors have been disabled (in a few cases, only the insecure APIs of a module have been disabled, retaining some useful APIs that are deemed safe). All these are eminently reasonable constraints given the goal of App Engine. And yet almost every one of these restrictions has caused severe pain for some of our users. "

---

" Where is Rust Positioned?

The three languages where I do most of my work in are Python, C and C++. To C++ I have a very ambivalent relationship because I never quite know which part of the language I should use. C is straightforward because the language is tiny. C++ on the other hand has all of those features where you need to pick yourself a subset that you can use and it's almost guaranteed that someone else picks a different one. The worst part there is that you can get into really holy wars about what you should be doing. I have a huge hatred towards both the STL and boost and that has existed even before I worked in the games industry. However every time the topic comes up there is at least someone who tells me I'm wrong and don't understand the language.

Rust for me fits where I use Python, C and C++ but it fills that spot in very different categories. Python I use as language for writing small tools as well as large scale server software. Python there works well for me primarily because the ecosystem is large and when it breaks it's a super straightforward thing to debug. It also keeps stays running and can report if things go wrong.

However one interesting thing about Python is that unlike many other dynamic languages, Python feels very predictable. For me this is largely because I am a lot less dependent on the garbage collector that in many other languages. The reason for this is that Python for me means CPython and CPython means refcounting. I'm the guy who will go through your Python codebase and break up cycles by introducing weak references. Who will put a refcount check before and after requests to make sure we're not building up cycles. Why? Because I like when you can reason about what the system is doing. I'm not crazy and will disable the cycle collector but I want it to be predictable.

Sure, Python is slow, Python has really bad concurrency support, the interpreter is quite weak and it really feels like it should work differently, but it does not cause me big problems. I can start it and it will still be there and running when I come back a month afterwards.

Rust is in your face with memory and data. It's very much like C and C++. However unlike C and C++ it feels more like you're programming with Python from the API point of view because of the type inference and because the API of the standard library was clearly written with programmer satisfaction in mind. ...

If you try, you can still make terrible decisions for concurrency in Rust, but you really need to go out of your way to do it. The language forces you to think more and I believe that's a good thing. I don't want to become another "objective oriented programming is the billion dollar mistake" preacher but I do think that the language decides largely what code people write. Because subclassing is easy in C++, Java, Python and many more languages this is what we write. And then birds are instances of animal classes. If you take that tool away you start thinking differently and that's a good thing. CPUs stop getting faster and looking at one object at the time really no longer makes any sense at all. We need to start reasoning a lot more about collections of things and what transformations we actually want to do. " -- http://lucumr.pocoo.org/2014/10/1/a-fresh-look-at-rust/

---

"I read an article by Bruce Eckel saying that he liked Python because he could remember the syntax. He said that despite teaching Java and writing books on Java, he could never remember the syntax for opening a file in Java, for example. But he could remember the corresponding Python syntax...I stand by my initial impression that Python is plain, but I now think that’s a strength. It just gets out of my way and lets me get my work done." -- http://www.johndcook.com/blog/2009/05/08/plain-python/

"But isn’t the same true for ruby? ;)" -- mark

"Not the same for ruby. I stopped using python for ~1.5 years and it was very easy to pick back up. But if I stop using ruby for a few months, it can be nontrivial to refresh. It’s a more complex language with odd conventions that make it harder to use at a low cognitive load (e.g. opening classes). Some things in ruby are brilliantly designed and very easy to pick back up, like map/select/etc block chaining. but overall it’s harder." -- Brendan O'Connor

"As someone who has been trying to make the same switch, I have to say, I keep returning to Perl because I keep forgetting the Python syntax!" -- Kyle

Ken 9 May 2009 at 00:37

“Memorable syntax increases productivity. You don’t have to break your train of thought as often to reach for a reference book.”

I find it increases the rate at which I can write lines of code, for this reason. That doesn’t necessarily correlate with productivity, especially long-term productivity.

In Common Lisp, for example, I can never remember all the args to with-open-file (though I can remember the couple I use all the time, and Emacs can tell me the rest). But my programs end up naturally OAOO. If I need something for a purpose other than how I coded it, I rarely have to refactor to use it. It’s already in a data structure I can use.

In Python, I can crank out the lines of code faster, but then after a month I have … a lot of lines of code. Then I have to go back and refactor, especially when I want to use something in a different way than originally intended.

I wonder if there’s a way to build a system with the best of both worlds. There’s got to be something better than Common Lisp for memorization, while keeping its benefits.

Abhishek Ghose 9 May 2009 at 02:33

Was delighted to know that I am not the only one who has a hard time remembering syntax to open files in Java :) I am a Rubyist, but I am planning to shortly look at Python because of its alleged performance edge over Ruby (I say ‘alleged’ since I am yet to see a conclusive comparison of Python with Ruby 1.9)

Waldir 28 June 2009 at 07:59

This post seems to resonate much with (and, in a way, summarize) Eric S. Raymond’s “Why Python?“. It’s much longer, but worth reading. That post delivered the final blow in convincing me to give up my plan to learn Perl (no matter how much I liked working with regex) and try Python instead. haichit 24 July 2009 at 12:46

I am not CS student but I need to write code for my research. C/C++ is powerful with its high speed but it’s very difficult to learn for me. The first time I met Python, I was very surprised how quick I could learn it. Thanks for your article.

Farhat 29 December 2009 at 01:34

I followed the same path with Python. Was a Perl code for several years, I dipped my toe into the Python pool a few times but never waded in. Then I was asked to teach a course in computing and thought since teaching is the best way to learn, I taught in Python and was amazed at how much easier it was do many things in Python as compared to Perl or C. And for tasks that require OOP, Python is head and shoulders above Perl. Developer 28 March 2011 at 04:18

I used several programming languages such as Assembly, Pascal, Delphi,Visual Basic, Matlab, C++, Fortran etc for a long time but recently I am experiencing Python. When I need to produce a nice look GUI rich application the best for me is Visual Basic yet. For me all experience with python GUI producers were not good enough. If I want to evaluate a numerical algorithm with immediate graphics and of course with enjoyment I use Python. When I need to improve the computation speed I use Fortran to generate library to be used in Python. … PJ Brunet 26 April 2011 at 11:21

I was just thinking last night, this is why I miss Pascal.

I don’t mind lots of extra typing either, because if the code is readable I save time reading/writing comments.

People new to programming assume WRITING the code is time-consuming, but thinking through the problem is typically where you’re burning the most calories. Not saving a few keystrokes.

Anyway, I agree with the above comment about PHP’s popularity due to its “www” birthright, but please don’t hate on PHP because you have a Ruby syntax fetish ;-) Too many programmers get hung up on the craft and forget the raison d’etre.

---

"Python had been my favourite language for about five years, but although I still work in Python professionally (I’d enjoy changing that to Rust, by the way) I find myself missing features from Rust more and more, and it really hasn’t worked the other way except with regards to libraries that haven’t been written in Rust yet. Rust has really spoiled Python for me. " -- http://chrismorgan.info/blog/rust-fizzbuzz.html

---

https://pythonconquerstheuniverse.wordpress.com/2009/10/03/python-java-a-side-by-side-comparison/

---

Python vs. java: example where java is only a litle bit faster, and much more complicated:

http://ubuntuforums.org/archive/index.php/t-1593005.html

---

http://punchplace.blogspot.com/2012/06/c-vs-java-vs-python.html

---

http://benchmarksgame.alioth.debian.org/u32q/compare.php?lang=java&lang2=python3 implies a

PEP 0484 – Type Hints is accepted (python.org) https://www.python.org/dev/peps/pep-0484/

bkeroack 2 days ago

One of the reasons why I use Python 3.x exclusively is for the new function annotations, which allow use of this: https://github.com/prechelt/typecheck-decorator

This allows me to write Python in a design-by-contract style with full, rich runtime type checks. My bug count has gone down dramatically, and when I do find a bug it's virtually always via a typecheck exception.

ak217 2 days ago

Thanks for the link. I developed the annotation+decorator-based type checker in https://github.com/kislyuk/ensure before I saw typecheck-decorator (which looks like it has some cool features). One thing I'm proud of with ensure's @ensure_annotations decorator is that we actually went through a few rounds of optimization and are fairly confident in the decorator's runtime performance.

Optional type checking as a first-class citizen is a great tool to have, and it's good to see the syntax standardized.

But as alluded to in the Reddit thread, there are serious issues with this PEP. The use of comments and the lack of runtime type checking from the start not only limits the potential of this particular PEP, but undermines future efforts to bring that functionality in.

IMO it would have been better to put in the work and properly integrate type checking decorations into the interpreter and the grammar. The tack-on comments at the end of the line are especially un-Pythonic and jarring.

Edit: Just saw the string literal based forward references. Ugh. Another problem in the making.

---

https://lwn.net/Articles/651967/

---

" Quick, what does the following literal represent semantically?

'abcd'

If you're a Python 3 user you would say it's the string consisting of the letters "a", "b", "c", and "d" in that order.

If you're a Python 2 user you may have said the same thing. You may have also said it was the bytes representing 97, 98, 99, and 100. And it's the fact that there are two correct answers in Python 2 for what the str object represents that led to changing the language so that the single Python 3 answer was the only answer. ... The simplification of the language and the removal of the implicitness of what a str object might represent makes code less bug-prone. The Zen of Python points out that "explicit is better than implicit" for a reason: ambiguity and implicit knowledge that is not easily communicated code is easy to get wrong and leads to bugs. By forcing developers to explicitly separate out their binary data and textual data it leads to better code that has less of a chance to have a certain class of bug. "

---

" Sadly concurrent running in the same interpreter won't work because Python 3 has different types for strings (unicode only) and a bytes type, while Python 2 has a str type (pretty much like bytes but can also be treated as a string) and a separate unicode type. Python 2 had code that tried to do the right thing automagically with str, such as automatic promotion to unicode where it looks like you really meant a string and that is what is needed. Automatic promotion can fail, be unexpected, is hard to test and numerous other issues. That is why Python 3 was needed and why it needed to be incompatible. To have Python 2 and 3 code interoperate in the same interpreter would require another layer of automatic promotion/demotion and heuristics when data goes between the two. " -- https://news.ycombinator.com/item?id=10889471

---

Footnotes:

 "Hello world" << endl;}

Now, in Python (v.2.7):

print "Hello world" "

" Analyzing 30 programs written in Java and 30 written in Python by novice programmers (in Finland, aged 16–19), Mannila et al. (2006) studied the errors found in them to identify those that could be attributed to the language. The students in both groups had the same teacher and studied the same contents in the same environment; only the language changed.

The study categorized errors as relating to understanding (logic) or arising from features of the language (syntax). Four criteria were also applied to the programs as a whole: execution, satisfying specs, error handling, and structure.

Of all the syntax errors found, only two appeared in Python programs while 19 were found in the Java programs (missing brackets or semicolons, uninitialized variables, etc.). And the logic errors were also significantly fewer in the Python programs, compared to Java (17 to 40). Also, more Python programs ran correctly and satisfied the specifications, and more included error checking/handling, compared to the Java programs. "

" Studying how people use a natural language to describe a task to another human gives clues. In such descriptions, people don’t define iterations, they instead put into words set operations; they are not explicit about loops terminating; people use constraints, event-driven tasks and imperative programming, but they never talk about objects. And when these natural-language instructions are given to other participants, they have no problem following them. Processing a set of data until it's finished is natural, but incrementing an index is not.

How is this related to Python? It so happens that the language's core looping idioms can often replace index manipulation, making it more like plain English. The following examples were given by Raymond Hettinger (core Python developer) in a keynote in 2013. To get the square of the numbers from 0 to 5, you might write Python code like this:

for i in [0, 1, 2, 3, 4, 5]:

print i2

But the convenient function range() makes it easy to iterate over longer lists.

for i in range(6):

print i2

This has the disadvantage of creating the list in memory (not good if the list is very big). So in Python 2.7, a better way is with the xrange() function (which in Python 3 dropped the x):

for i in xrange(6):

print i2

Now, suppose you want to print the colors in a list like this:

colors = ['red', 'green', 'blue', 'yellow']

You might write this to loop over all the colors:

for i in range(len(colors)):

print colors[i]

But Python let's you do this instead, which looks more natural, and like Raymond says, more beautiful:

for color in colors:

print color

In summary, Python is a lot more like English than other programming languages, and reduces the cognitive load in learning to think computationally. "

" Matlab has bad string manipulation Indexing: Python indexing goes as it does in C … starting from 0 + Python indexing is done using brackets, so you can see the difference between an indexing operation and a function call. "

-- http://lorenabarba.com/blog/why-i-push-for-python/

---

" My perspective is as a researcher in academia. From here Python is far from needing to be revived, it is gaining ground. Also, Python 2 is still king and I suspect will remain popular - researchers rarely have the luxury/incentive to go back and improve it to comply with a language update.

A few reasons Python is gaining ground in academia. Firstly, Python is _great_ for teaching undergraduate programming (see [0] for a good perspective), and many institutions are only now catching on and migrating from Matlab. Secondly it is a charitable language allowing scientists to concentrate on their research and not the CS implementation they are using. It is easy to learn, flexible, has lots of scientific libraries, interfaces to fortran/C libs. ... Python often gets a bad rap for speed, but this is rarely the problem it is made out to be. Python is best when used as a high level manager sitting on top of low level routines written in C/Cython. ...

Finally, it has a massive advantage over many other languages: Great documentation and a large community. " -- https://news.ycombinator.com/item?id=7802579

skriticos2 10 hours ago

link

I add that Python is excellent at data representation and transformation. You can easily write some test data and just include it in python, then transform it any way you feel you like to get intelligence out of it. With the standard library you can easily load a CSV file you exported form a data-set and then aggregate and evaluate it with only a few lines of code without much language overhead. Schema-less mixed containers, array operations, dictionaries and sets are incredibly powerful and you can nest them anyway you like. Additionally you can also forget about type limits. Work with numbers as big as you like. That's useful for a lot of things. "

yid 1 hour ago

link

> I add that Python is excellent at data representation and transformation

Indeed! It's a very practical extraction and reporting language.

arjn 36 minutes ago

link

I see what you did there :-)

sergiosgc 6 hours ago

link

Python is also the modern sysadmin's Perl. This is a huge niche, to the point of not being a niche at all. Just take a look at the number of system tools used in a Redhat install that use python.

Coincidentally, Redhat's python tools are one of the major reasons for continuing support of Python 2.

clebio 1 hour ago

link

Also, Ansible: https://github.com/ansible/ansible

edit: markdown... oh, nevermind.

" Nowadays stuff like proper closures, immutability, a good async story, etc, is considered a necessity by discerning hackers. "

Remove the GIL. Or provide a compelling async story. Guido’s PEP 3156 might, or might not be that. Some primitives like Go’s channels would be nice to have.

Make it speedy. Seriously, if Javascript can be made fast, CPython could be made fast too. Or PyPy? could mature enough to replace it (there should be only one). If it takes big bucks or Lars Back to do it, start a Kickstarter — I’ll? contribute. Shame major companies into contributing too. Isn’t Dropbox spending some money on its own LLVM based Python anyway?

Add types. Well, opt-in types. That you can use to speed some code up (a la Cython), or to provide assurances and help type-check (a la Dart). Add type annotations to everything in the standard library.

Shake the standard libraries. Get a team together to go through them and fix long standing annoyances, improve speed and fix bugs. Improve their API, and offer nice simpler interfaces for common stuff (think requests vs urllib) Offer the new improved library alongside the current standard library in a different namespace. Make it easy to switch over (perhaps with some automated tool).

Revamp the REPL. It’s 2014 already. Redo the REPL in a modern way. Add some color. Take notes from iPython. Make it a client/server thing, so IDEs and editors can embed it.

" -- https://medium.com/p/2a7af4788b10

"the author's priorities seem to line up exactly with my own thoughts on Python's shortcomings. Async, speed, types, stdlib, repl." -- https://news.ycombinator.com/item?id=7803432

Powerful. Because it’s so well designed, it’s easier than other languages to transform your ideas into code. Further, Python comes with extensive standard libraries, and has a powerful datatypes such as lists, sets and dictionaries. These really help to organize your data. Namespaces. Matlab supports namespaces for the functions that you write, but the core of Matlab is without namespaces; every function is defined in the global namespace. Python works with modules, which you need to import if you want to use them. (For example from skimage import morphology.) Therefore Python starts up in under a second. Using namespaces gives structure to a program and keeps it clean and clear. In Python everything is an object, so each object has a namespace itself. This is one of the reasons Python is so good at introspection. Introspection. This is what follows from the object oriented nature of Python. Because a program has a clear structure, introspection is easy. Private variables only exist by convention, so you can access any part of the application, including some of Python’s internals. Of course, in good programming practice you would not use private variables of other classes, but it’s great for debugging! String manipulation. This is incredibly easy in Python. What about this line of code which returns a right justified line of 30 characters: "I code in Matlab".replace('Matlab','Python').rjust(30) " -- http://www.pyzo.org/python_vs_matlab.html

on python 2 v.s python 3:

" Nowadays I can't imagine a lot of people discounting Python because of the print statement, unicode support, division rules, or lack of yield from statement. " -- https://news.ycombinator.com/item?id=7802095

http://legacy.python.org/dev/peps/pep-3102/

"nonlocal, asyncio, unicode, yield from. If you ask me, these four are very compelling reasons to switch to Python 3 and they solve very real problems." -- https://news.ycombinator.com/item?id=7801834

" chrishenn 18 hours ago

link

One pain point I've really felt recently with Python is in the deploy step. pip installing dependencies with a requirements.txt file seems to be the recommended way, but it's far from easy. Many dependencies (such as PyTables?) don't install their own dependencies automatically, so you are left with a fragile one-by-one process for getting library code in place.

It was all fine once I got it worked out but it would so much nicer to provide a requirements.txt file and have pip figure out the ordering and dependency stuff. That and being able to install binary packages in a simple way from pip would make me much happier with python (no more waiting 10 minutes for numpy or whatever to compile).

As far as actual language features go however, I still find python a joy to work in.

yaph 18 hours ago

link

If a package doesn't get all its dependencies installed via pip its because of missing information in the package itself. That's neither a flaw of pip or Python but will cause problems for any package manager.

I find the combination of virtual environments and pip very convenient to work with. When I run into trouble with missing dependencies I often find the project on GitHub? and can send a pull request.

Regarding Numpy and the scientific Python stack, check out Anaconda https://store.continuum.io/cshop/anaconda/ it makes managing environments where you need these packages a lot less painful.

andreasvc 8 hours ago

link

Not necessarily. pip cannot resolve all dependencies. For example, if a package specifies both numpy and pandas as a requirement, installation will fail. This is because pandas in turn requires numpy, and pip does not resolve the dependencies in a single step, you need to install numpy first and then go on with pandas.

yaph 7 hours ago

link

In that case, couldn't you just specify Pandas as a dependency and Numpy would automatically be installed?

andreasvc 4 hours ago

link

I think not, I think the problem is that it is a build-time dependency and pip wants to resolve all dependencies before installation begins.

See https://github.com/pypa/pip/issues/1386 (oddly, the issue is closed, while the problem is acknowledged).

Derbasti 12 hours ago

link

I recently tried to deploy a desktop app written in Python. It was a nightmare. I recently taught scientific Python to prospective switchers. The installation step was a nightmare.

We really need pip wheels or conda to become mainstream. Pip alone doesn't cut it on platforms without compilers (Windows / OSX). Standalone installers are fine, but they don't resolve dependencies and they are only available for Windows.

I totally agree that the installation and deployment story should be a top priority. Once done, it would be another very compelling point on Python 3's feature list.

pekk 4 hours ago

link

wheels are pretty mainstream and are getting better fast. Forking the package management ecosystem again, just after we got over the last round of headaches there, is only going to make things worse

thinkpad20 18 hours ago

link

At my company we build with Jenkins, tarball it and deploy from that. I haven't had experience another way but I think it ends up being more efficient and less error-prone than doing actual pip installs during deployment.

dagw 7 hours ago

link

That's basically what the company I used to work for did as well with our large python application. Build using a build script, pulling in dependencies from local build server, test, package up the result and deploy. Pip was used for pulling in libraries to the build machine and never used during deployment.

L_Rahman 17 hours ago

link

One of the reasons I've come to do most of my early stage prototyping in node is that managing npm packages has thus proven much easier than finagling with pip or gem.

Ruby installs are especially difficult to manage. Despite the numerous tutorials out there, I still don't know what, if there even is one, the canonically best way to install ruby and necessary gems is. RVM? Install through Brew? Add path to ~/.bashrc?

I say this as someone who likes using command line so much that I've written Caskfiles to automate my deployment to fresh OS X Machines.

pak 16 hours ago

link

> Despite the numerous tutorials out there, I still don't know what, if there even is one, the canonically best way to install ruby and necessary gems is.

Wait, what? On production, install the exact ruby you need from your favorite package manager. On your dev box, install any rubies you need through rbenv [1]. Put all your gem dependencies into a Gemfile [2]. On either end, bundle install [--deployment] and call it a day.

[1]: https://github.com/sstephenson/rbenv

[2]: http://bundler.io/v1.6/gemfile.html

pmontra 10 hours ago

link

I install rvm on my production boxes as explained at https://rvm.io/ then I put .ruby-version and .ruby-gemset files in the application directory to select the interpreter and the gemset (I might need to have different applications running - especially on staging machines). Finally I use bundle and a Gemfile. It's pretty easy. You got another answer suggesting rbenv which is also fine.

ermintrude 3 hours ago

link

Check out devpi (http://doc.devpi.net/latest/). We build our projects as wheels with pip using devpi as a custom index, i.e.:

BUILD_DIR="/tmp/wheelhouse/$PROJECT" pip wheel --wheel-dir=$BUILD_DIR --find-links=$BUILD_DIR --index=http://$DEVPI_HOST:$DEVPI_PORT/${DEVPI_USER}/${DEVPI_INDEX} --extra-index-url=https://pypi.python.org/simple/ ../$PROJECT

This will try to get packages from your devpi index, but will fall back to pypi if you don't have them.

Then upload this to devpi. It can later be installed by pip with:

pip install --use-wheel --index=http://$DEVPI_HOST:$DEVPI_PORT/${DEVPI_USER}/${DEVPI_INDEX} $PROJECT

This makes deployments super fast because you're deploying pre-built wheels instead of downloading and compiling from pypi. It also gives you resiliance by storing copies of the dependencies you need in devpi, so if they vanish from pypi (or it's unavailable) you can still deploy your software with all the dependencies you developed against.

SoftwareMaven? 16 hours ago

link

There is work going on in the python packaging SIG to make installing better. One of the big additions for pip is the ability to install pre-built wheels for those "hard to build" packages. There is also work happening on a new package metadata standard.

Unfortunately, that's all I've learned from lurking on the mailing list for a couple weeks.

Keats 8 hours ago

link

As several person mentioned already, wheels (http://wheel.readthedocs.org/en/latest/) make deploying much easier (and faster), not needing to get anything from pypi.

marcosdumay 15 hours ago

link

You should avoid using normal requirements.txt files in production. If you want to use pip, it's better to "pip freeze" your test environment, and use that requirements file to specify the production environment.

Otherwise you are just asking for nasty surprises when packages upgrade.

erichurkman 15 hours ago

link

Or just specify versions in your requirements.txt file to begin with.

If you want to keep up to date with security and bug fixes (but aren't yet ready for the next big feature/backwards incompatible release), you can specify the lines as 'package>=1.1,<1.2' to get 1.1.x fix releases.

`pip list --outdated` is helpful, too.

jaymon 14 hours ago

link

Just a note so people don't get confused, while specifying packages with >=1.1,<1.2 seems to be similar to the tilde in npm, in practice it isn't.

Basically, when you use >=1.1,<1.2 it will install the best version that matches at the time of first install, and then that version will never be upgraded because it will always satisfy the requirements. So you don't actually get 1.1.x release updates unless you install them manually.

We do, however, use this syntax in development when testing new versions to make sure any subsequent runnings of pip doesn't obliterate the new versions of modules we are testing.

I'd love official pip support for ~1.1 type declarations.

cjg_ 15 hours ago

link

Long compile times can be fixed with pre-compiled wheels, http://wheel.readthedocs.org/en/latest/ I shaved off 4 minutes of our build time of numpy/pandas with it.

icebraining 8 hours ago

link

There's nothing like a good old deb (or rpm) package. Learn fpm[1] and bundle your dependencies instead of hoping that they get there.

[1] https://github.com/jordansissel/fpm

Alphasite_ 18 hours ago

link

In my experience, you can bundle the dependancies, and then add the path to your search path.

Not the best approach, but it does work.

maaku 18 hours ago

link

`pip freeze`

andreabedini 17 hours ago

link

(dormant) PyTables? developer here, we do have a requirements.txt file but I understand our setup.py needs an update. Please open an issue on github and tell us about your experience and how we can improve it.

chrishenn 2 hours ago

link

I will soon, thanks!

reply "

jerf 17 hours ago

link

"People are leaving Python for Go because people have always left Python for fast compiled languages."

I think the angst about Go comes from the fact that someone who leaves Python for Java may still come back, because you can develop far more quickly in Python than Java. But someone who leaves Python for Go probably isn't coming back... my experience is that it is slightly slower (10-20%, YMMV but we're certainly not talking integer multiples) to cut out a prototype in Go, but that said prototype runs an order of magnitude faster, can potentially be optimized without much work for another order of magnitude (though ultimately this is more a reflection of Python's slowness than Go's speed), and is then much easier to maintain and extend, even on the timescale of a week or two, to say nothing of larger time scales.

A couple of people elsewhere in the comments here assume that Python must still be much easier to develop for than Go. It really isn't anymore; it turns out the combination of garbage collection and structural-typing-esque interfaces pretty much does anything you might have ever wanted Python to do, and a great deal of the rest of the differences turn out to be far less important in practice than in theory.

I first saw the idea in Joel Spolsky's "How Microsoft Lost the API War" [1], under the heading "Automatic Transmissions Win The Day", that the primary innovation of the 1990s was simply garbage collection instead of memory management. (As he is aware the idea is older than 1990, one presumes he means that it became practical and widespread.) The languages that spread this innovation, the "scripting languages" like Perl and Python and PHP and Javascript, changed a lot of things at once, which as any good scientist knows means it was hard to tell what about those changes actually contributed to the enhanced productivity that they certainly did bring. My experience with Go certainly gives me further belief in Joel's thesis... you read a bullet point listing of the Python features and Go features and it seems obvious that Python is a wildly better language than Go, yet... I've learned after all these years and all the languages I've tried out to ask myself, if the features are so important, why don't I miss them? Because in practice, I don't. Rip closures out of Go, and I'd miss that. Rip out the goroutines and I'd miss not having something like generators or something, indeed, the language would nearly be useless to me. But I certainly don't miss metaclasses or decorators or properties. I will cop to missing pervasive protocols for the built-in types, though; I wish I could get at the [] operator for maps, or implement a truly-transparent slice object. But that's about it.

[1]: http://www.joelonsoftware.com/articles/APIWar.html

nostrademons 12 hours ago

link

My experience was Go was about 2-3x slower (development speed) than Python for prototyping. Web programming, though, which is a particular strength of Python and a particular weakness of Go. YMMV, of course.

I actually really do miss list comprehensions and properties and pervasive protocols for built-in types and really concise keyword/literal syntax. I don't miss metaclasses, and I only vaguely miss decorators. (Go has struct annotations and compiler-available-as-a-library, which in some ways are better.) Properties in particular are really useful for the evolvability of code; otherwise you have to overdesign up front to avoid a big rewrite as you switch from straight struct fields to accessors.

I'm actually leaning towards Java 8 + Jython 2.7 as I consider what language to write my startup/personal projects in. Jython 2.7 gives me basically all the language features I actually cared about in Python, and it fixes the Unicode mess that necessitated Python 3. It has no GIL and solid multithreading support. The Java layer gives you all the speed of Go and more. And the interface between them lets you seamlessly use Java within your Jython scripts, so you can easily push code down into the Java layer without having to rewrite your whole product.

jerf 4 hours ago

link

Yes, the Go web prototyping story is a bit weak. If you want to do serious development in it I think it's pretty good, because frankly for years our web frameworks in most other languages have been unbelievably contorted around not having something like goroutines in ways that we've stopped even being able to see because they're so darned pervasive, but if you just want to slam a decent site out (and there's nothing wrong with that) there's no great story there right now.

rwallace 13 hours ago

link

I found to my surprise that these days Java is as productive as anything else. It's not like the early days anymore; modern Java has generics, reflection, now lambda; there would still be more finger typing than I'd ideally like, but the IDEs are more than good enough to make up for that, to the point that a nontrivial program now usually takes fewer keystrokes to write in Java than almost any other language.

Which of course reinforces your main point: nowadays the language differences just aren't that big.

sitkack 12 hours ago

link

I don't know why people are leaving Python for Go when they could be using PyPy? or Shedskin. There are plenty of ways to be fast using Python.

jerf 5 hours ago

link

It's easy to lose track of this in the torrent of "PyPy? Benchmarks Show It Sped Up 2x!!!!" and "Latest Javascript Engine 50% Faster Than The Last One!!!! OMG!! Node!!!!", but in absolute terms, the dynamic language JITs are still quite slow. It's an urban legend that they are anywhere near compiled performance. Except LuaJIT?. "Faster Python" is still slow. Unless you're in a tight loop adding numbers together, but I'm not.

Moreover, a plot of their performance over time rather strongly suggests that they've plateaued. Personally, barring any major JIT advances, I'm considering the book closed on the topic of whether JITs can take a dynamic language and make it as fast as C or C++: No.

(Recall that a Java JIT is another issue; it starts with the static typing of Java, and then JITs from there. It has a much larger array of much more powerful guarantees to work with, that the dynamic languages simply don't. Go will be running at straight-up C speeds (which it does not currently) long before Python will.)

---

SoftwareMaven? 16 hours ago

link

other than proper functional programming, maybe, which isn't about to happen in Python

If we could just have tail call optimization, I think we could make the rest work (well, maybe better lambdas, too).

irahul 8 hours ago

link

> If we could just have tail call optimization, I think we could make the rest work

Manually thunk/trampoline it?

    class trampoline(object):
        def __init__(self, fn):
            self.fn = fn

        def __call__(self, *args, **kwargs):
            ret = self.fn(*args, **kwargs)
            while isinstance(ret, thunk):
                ret = ret()
            return ret

    class thunk(object):
        def __init__(self, fn, *args, **kwargs):
            self.__dict__.update(fn=fn, args=args, kwargs=kwargs)

        def __call__(self):
            if isinstance(self.fn, trampoline):
                return self.fn.fn(*self.args, **self.kwargs)
            else:
                return self.fn(*self.args, **self.kwargs)

    @trampoline
    def fact(n, accum=1):
        if n <= 1:
            return accum
        else:
            return thunk(fact, n-1, n*accum)

    print fact(1000)

    @trampoline
    def is_even(n):
        if n == 0:
            return True
        else:
            return thunk(is_odd, n - 1)

    @trampoline
    def is_odd(n):
        if n == 0:
            return False
        else:
            return thunk(is_even, n - 1)

    print is_even(1000001)

You only have to write thunk/trampoline utility once, and it is similar to how we do it in clojure.

keeperofdakeys 15 hours ago

link

To remove the GIL, you'd need to introduce proper atomicity to all of the standard library, and include concurrency checks for many data structures. And the GIL isn't really the enemy, if you are doing cpu intensive work, you'd use the C interface - which happens to not be affected by the GIL in the first place. This also happens to be how to get "more performance". Unfortunately this means you can't switch to pure JIT, because as PyPY? have found out, you need to rewrite parts of the standard library that are written in C (like the crypto libraries).

annnnd 10 hours ago

link

+1 just for mentioning requests (versus urllib). If someone hasn't tried this library yet, they really should. I replaced tons of libcurl / urllib / ... blocks with simple and legible oneliners. Incredible API!

EDIT: url for those who don't know the library yet: http://docs.python-requests.org/en/latest/ (I am not affiliated, just really impressed)

</offtopic>

thomasahle 1 hour ago

link

It always strikes me, that the things that parts of python that create problems for people trying to optimize it, are all things of relatively small importance. Like __new__, __del__ semantics and changing the stack by inspection. I wish Python3 had just let go of the slow scripting-only parts.

dbpatterson 18 minutes ago

link

The problem is that often while regular users are not directly using those features, they are using libraries that do use them. As a concrete example, people writing webapps with Django probably (often) aren't using some of the crazier class metaprogramming, but they are using Django's models, which use lots of it.

plorkyeran 1 hour ago

link

It's still well over an order of magnitude bigger than Lua. MicroPython? is ~200k non-blank-non-comment lines of code (as reported by cloc), while Lua 5.1 is 12k.

" Micro Python has the following features:

Supports almost full Python 3 syntax, including yield (compiles 99.99% of the Python 3 standard library).
Most scripts use significantly less RAM in Micro Python, and various benchmark programs run faster, compared with CPython.
A minimal ARM build fits in 80k of program space, and with all features enabled it fits in around 200k on Linux.
Micro Python needs only 2k RAM for a basic REPL.
It has 2 modes of AOT (ahead of time) compilation to native machine code, doubling execution speed.
There is an inline assembler for use in time-critical microcontroller applications.
It is written in C99 ANSI C and compiles cleanly under Unix (POSIX), Mac OS X, Windows and certain ARM based microcontrollers.
It supports a growing subset of Python 3 types and operations.
Part of the Python 3 standard library has already been ported to Micro Python, and work is ongoing to port as much as feasible.

More info at:

http://micropython.org/

You can follow the progress and contribute at github:

www.github.com/micropython/micropython www.github.com/micropython/micropython-lib "

ngoldbaum 5 days ago

link

It was just pointed out to me that micropython starts an order of magnitude faster than both python2 and python3:

    $ time ./micropython -c 'print(1)' 
    1
    ./micropython -c 'print(1)'  0.00s user 0.00s system 0% cpu 0.002 total
    
    $ time ./python2 -c 'print(1)'
    1
    python2 -c 'print(1)'  0.01s user 0.00s system 52% cpu 0.019 total

    $ time ./python3 -c 'print(1)'
    1
    python3 -c 'print(1)'  0.03s user 0.00s system 85% cpu 0.035 total

chubot 4 days ago

link

Yeah this is one of my longstanding annoyances with Python... the interpreter is quite slow to start up because every import generates a ton of stat() calls. Moreover, there is a tendency to use a very long PYTHONPATH, which increases the number of stats.

It's basically doing random access I/O (the slowest thing your computer can do) proportional to (large constant factor) * (num imports in program) * (length of PYTHONPATH).

When you can use it, a shebang of

!/usr/bin/python -S

can speed things up substantially.

Perl starts up an order of magnitude faster too (more like 3ms than 30ms). Ruby seems to have the same problem as Python.

pekk 4 days ago

link

What are you doing that you are noticing this routinely and finding it to be a large annoyance?

price 4 days ago

link

Any command-line script in Python that uses a significant codebase or a significant number of libraries. For me the main example is Mercurial -- it takes about 200ms (on my Mac) to run even a trivial command, and that's plenty long enough to get in the way.

It's about 60ms on my other laptop (Linux), and that's just at the threshold of noticeable and annoying.

im3w1l 4 days ago

link

Hmm should be possible to reduce that to num imports + length of pythonpath, no?

sp332 4 days ago

link

For each import statement, it has to check each directory in the path. So (num imports) * (length of path).

---

michaelhoffman 5 days ago

link

> Supports almost full Python 3 syntax, including yield (compiles 99.99% of the Python 3 standard library).

What parts of Python 3 syntax are missing? Which parts of the library don't compile?

maxerickson 5 days ago

link

It's discussed some here:

https://github.com/micropython/micropython/wiki/Differences

But I guess they are still in the process of writing that/filling it out.

The biggest difference listed is probably no Unicode. Another significant point is that lots of library functions are only partially implemented.

carbon12 5 days ago

link

> What parts of Python 3 syntax are missing? Which parts of the library don't compile?

The only things that don't compile properly are certain uses of "super()". super() without arguments is a very strange beast that captures the first argument of the function (interpreting it as the self object), and needs to infer its class.

Other than that, all the Python scripts in the Python 3 standard library will compile.

thomasahle 5 days ago

link

dbpatterson 5 days ago

link

---

plorkyeran 5 days ago

link

It's still well over an order of magnitude bigger than Lua. MicroPython? is ~200k non-blank-non-comment lines of code (as reported by cloc), while Lua 5.1 is 12k.

carbon12 5 days ago

link

Actually, most of that SLOC is in the stmhal/ port, of which ~100k is the ST library.

The true count comes from the py/ directory, for which cloc gives only 25k. And there are a lot of extra bits there, eg the inline assembler, the Thumb and X64 assembler helpers, etc.

EDIT: without the additional assemblers, cloc gives 22k. Remember that Python has ints, floats, complex and bignum, and these are all included in Micro Python. So that 22k SLOC includes complex arithmetic and a 1k bignum implementation.

nly 4 days ago

link

    $ size /usr/bin/micropython 
       text    data     bss     dec     hex filename
     272356    1336    1216  274908   431dc /usr/bin/micropython
    $ size /usr/lib/liblua.so.5.2.3 
       text    data     bss     dec     hex filename
     195101    6408       8  201517   3132d /usr/lib/liblua.so.5.2.3

I think it would really interesting for µPy to be opened up to run as a hosted runtime.

trynumber9 5 days ago

link

And then there is LuaJIT?. As fast as V8, smaller than Micro Python, with a dead simple FFI.

---

 garbage collection is used in such a resource constrained environment?

Also tangent question, what is it about languages like Python and Ruby that make it more amenable to reimplementation than Perl?

Demiurge 5 days ago

link

To take a stab at the second question, I think BNF that fits on one page https://docs.python.org/3.4/reference/grammar.html vs http://perldoc.perl.org/perlfaq7.html#Can-I-get-a-BNF%2fyacc...

This basically means Perl is very complex and its grammar can be self contradicting, such that behavior is undefined. C++ has a similar problem to a lesser extent.

stcredzero 5 days ago

link

When I looked at such things last, Python had about 29 terminals & nonterminals in its grammar. Ruby had 110. (These are numbers I remember from playing with particular parser libraries, so YMMV.) By contrast, a commercial Smalltalk with some syntax extensions had 8. I have no idea about Perl, but I'd guess it's about the same as Ruby.

dragonwriter 5 days ago

link

> To take a stab at the second question, I think BNF that fits on one page

Maybe for python, but not for Ruby. Ruby is not particularly simple to parse (though it may be simpler to parse than Perl, and clearly seems to be simpler to implement -- or perhaps its just that more motivation exists to implement it.)

Demiurge 5 days ago

link

First google result: http://www.cse.buffalo.edu/~regan/cse305/RubyBNF.pdf

I think 2 pages is not bad :) The point is, Perl is just impossible to formally define, it depends on the implementation to make arbitrary choices. This means multiple implementations are much harder, if possible.

dragonwriter 5 days ago

link

> First google result: http://www.cse.buffalo.edu/~regan/cse305/RubyBNF.pdf

Yeah, but its not:

1) One page, or

2) Current (it claims to be for Ruby v1.4), or

3) (apparently, I can't verify this for the version of Ruby it claims to represent) Accurate [1]

[1] http://stackoverflow.com/questions/663027/ruby-grammar

But, yes, Ruby can be parsed independent of being execution, which makes means you can separate the work of a separate implementation into (1) building (or reusing) a parser, and (2) building a system to execute the result of the parsing. Being able to divide the work (and, as a result, to share the first part between different implementations) makes it easier to implement.

carbon12 5 days ago

link

A lot of the internals are significantly different, and would be difficult to back port. Eg, a Micro Python object is a machine word which might be a pointer to a real object, a small integer, or an interned string. By stuffing small integers (and strings) into the unused bit-space of a machine word, Micro Python can do a lot of integer operations without allocating heap RAM. This is in contrast to CPython where all integers are on the heap (although the ones below 257 are cached).

Another example: calling a function in CPython requires a heap allocation of the call frame. In Micro Python this is generally not needed (unless the function call is significantly complicated).

keeperofdakeys 4 days ago

link

Quite simply, this project cut most of the features that big programs depend on, in order to make a smaller, faster core. Things like most of the standard library, meta-programming, and general scalability are included in these cuts. Most people don't realise how much python does underneath for them, which is why cpython is so "bloated" in comparison to this.

Micropython Cython support #658 https://github.com/micropython/micropython/issues/658

https://github.com/micropython/micropython/wiki/Differences

Differences pfalcon edited this page on May 24 · 25 revisions Pages 41

    Board FreeSOC
    Board Apogee V1.0
    Board Arduino Due
    Board Digi X
    Board EA LPC4357
    Board Mikroe mini M4
    Board NetduinoPlus2
    Board Olimex STM32 405STK
    Board Olimex STM32 H407
    Board OpenMV
    Board OpenPilot Revolution
    Board Pixy
    Board Polyhelo
    Board STM32F401 Discovery
    Board STM32F407 Discovery
    Show 26 more pages…

Clone this wiki locally

This page is a proof-of-concept effort to document the differences between CPython3 (considered to be a reference implementation of the Python3 language) and MicroPython?. It classifies differences into 3 categories, each category having a different status regarding the possibility that items belonging to it will change. Differences By Design

MicroPython? is intended for constrained environments, in particular, microcontrollers, which have orders of magnitude less performance and memory than "desktop" systems on which CPython3 runs. This means that MicroPython? must be designed with this constrained environment in mind, leaving out features which simply won't fit, or won't scale, with target systems. "By design" differences are unlikely to change.

MicroPython does not ship with an extensive standard library of modules. It's not possible and does not make sense, to provide the complete CPython3 library. Many modules are not usable or useful in the context of embedded systems, and there is not enough memory to deploy the entire library on small devices. So MicroPython takes the minimalist approach - only core datatypes (plus modules specific to particular hardware) are included with the interpreter, and all the rest is left as 3rd-party dependencies for particular user applications. The micropython-lib project provides the non-monolithic standard library for MicroPython (forum)
Unlike CPython3, which uses reference-counting, MicroPython uses garbage collection as the primary means of memory management.
MicroPython does not implement complete CPython object data model, but only a subset of it. Advanced usages of multiple inheritance, __new__ method may not work. Method resolution order is different (#525). Metaclasses are not supported (at least yet).
By virtue of being "micro", MicroPython implements only subset of functionality and parameters of particular functions and classes. Each specific issue may be treated as "implementation difference" and resolved, but there unlikely will ever be 100% coverage of CPython features.
By virtue of being "micro", MicroPython supports only minimal subset of introspection and reflection features (such as object names, docstrings, etc.). Each specific feature may be treated as "implementation difference" and resolved, but there unlikely will ever be 100% coverage of CPython features.
print() function does not check for recursive data structures, and if fed with such, may run in infinite loop. Note that if this is a concern to you, you can fix this at the Python application level. Do this by writing a function which keeps the history of each object visited, and override the builtin print() with your custom function. Such an implementation may of course use a lot of memory, which is why it's not implemented at the MicroPython level.

Implementation Differences

Some features don't cater for constrained systems, and at the same time are not easy to implement efficiently, or at all. Such features are known as "implementation differences" and some of them are potentially the subject for future development (after corresponding discussion and consideration). Note that many of them would affect the size and performance of any given MicroPython? implementation, so sometimes not implementing a specific feature is identified by MicroPython?'s target usage.

    No unicode support is actually implemented. Python3 calls for strict difference between str and bytes data types (unlike Python2, which has neutral unified data type for strings and binary data, and separates out unicode data type). MicroPython faithfully implements str/bytes separation, but currently, underlying str implementation is the same as bytes. This means strings in MicroPython are not unicode, but 8-bit characters (fully binary-clean).
    Object finalization (__del__() method) is supported for builtin types, but not yet user classes. This is tracked by #245.
    Subclassing of builtin types is partially implemented, but there may be various differences and compatibility issues with CPython. #401

Known Issues

Known issues are essentially bugs, misfeatures, and omissions considered such, and scheduled to be fixed. So, ideally any entry here should be accompanied by bug ticket reference. But note that these known issues will have different priorities, especially within wider development process. So if you are actually affected by some issue, please add details of your case to the ticket (or open it if does not yet exist) to help planning. Submitting patches is even more productive. (Please note that the list of not implemented modules/classes include only those which are considered very important to implement; per the above, MicroPython? does not provide full standard library in general.)

    Some functions don't perform argument checking well enough, which potentially may lead to crash if wrong argument types are passed.
    Some functions use assert() for argument checking, which will lead to crash if wrong argument types are passed/erroneous conditions are faced. This should be replaced with Python exception raising.
    It's not possible to override builtin functions in a general way. So far - no "builtins" module is implemented, so any overrides work within the current module only.
    There's no support for persistent bytecode, and running bytecode (vs running Python source). #222
    print() function doesn't use the Python stream infrastructure, it uses the underlying C printf() function directly. This means if you override sys.stdout, then print() won't be affected. #209
    Some more advanced usages of package/module importing is not yet fully implemented. #298
    Exception handling with generators is not fully implemented. (Some small details may need to be done yet.) #243
    str.format() may miss few advanced/obscure features (but otherwise almost completely implemented). #407, #574
    % string formatting operator may miss more advanced features #403, #574
    struct module should have all functionality, but misses some syntactic sugar (like repetition specifiers in type strings).
    No builtinre module implementation. micropython-lib offers the initial PCRE FFI module for "unix" port #13
    Only beginning of io module and class hierarchy exists so far.
    collections.deque class is not implemented.
    memoryview object not implemented.
    Container slice assignment/deletion is only partially implemented. #509
    Keyword and keyword-only arguments need more work #466, #524.
    Only basic support for __new__ method is available #606, #622.

---

micropython hn https://news.ycombinator.com/item?id=7840566

---

https://news.ycombinator.com/item?id=8008944

tribaal 1 day ago

link

The more frequent and dangerous pitfalls are, in my humble opinion:

Bare except: statements (that catches everything, even Ctrl-C)
Mutables as default function/method arguments
Wildcard imports!

SEJeff 1 day ago

link

Couldn't agree more! One of my all time new python interview questions gets a surprisingly large number of developers.

Given a function like:

    def append_one(l=[]):

        l.append(1)

        return l

What does this return each time?