notes-computer-programming-programmingLanguageDesign-prosAndCons-python

" My own experience with Python, after being a heavy Perl and Ruby user isn't that good. Especially if you are coming from a language like Perl, Python as a language has an interesting trend. At first you feel very happy that things are so simple with it. Then slowly once you learn the language and start getting deeper into it, you try to use it for day to day work. Suddenly you realize that text processing sucks badly, something that is very big problem on Unix based environments. Writing a single regex takes a 10's of lines of exception handling. You try to use it for quick dirty scripting, again the language runs out of steam very soon. Its just too verbose for scripting class.

Now coming to more important reasons. Sacrificing power for readability might be great for newbie's but isn't good experienced programmers. Lack of multiline lambda's, no good support for TOC, GIL, barely there object system(compared to Moose), lack of extensibility, lack of syntax plugins. "

" Consider bash:

    if [ `which foo` == "/usr/bin/foo" ]; then
      do_stuff
    fi

compared to Ruby:

    if `which foo`.strip == "/usr/bin/foo"
      do_stuff
    end

compared to Python:

    import os
    if os.popen("which foo").read().strip() == "/usr/bin/foo":
      do_stuff()

or

  if [ -e '/bin/bash' ];
  then
      do_stuff
  fi

(I don't know Ruby.)

Python:

  import os.path
  if os.path.exists('/bin/bash'):
      do_stuff()

"

" lobster_johnson 7 days ago

... three core similarities/differences:

• Ruby has bashisms like `foo`; Python requires a function call.
• Ruby has a nesting syntax similar to bash with if/end; Python is whitespace-sensitive.
• Ruby has method calls without parentheses, similar to bash commands, whereas Python requires them.

> Although if your goal is to determine if e.g. bash is located in /bin,

Actually, that was not the goal at all. The goal (which, again, is irrelevant to my point) was to determine if the PATH-resolved executable binary "foo" was in a specific location. Rest assured that I would not use "which" in a real-life app. :-)

(Thanks for commands.getoutput(), my Python is rusty.)

reply "

https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx

" What is async I/O? • Do something else while waiting for I/O • It's an old idea (as old as computers) • With lots of approaches – threads, callbacks, events... • I'll come back to this later Why async I/O? • I/O is slow compared to other work – the CPU is not needed to do I/O • Keep a UI responsive – avoid beach ball while loading a url • Want to do several/many I/O things at once – some complex client apps – typical server apps Why not use threads? • (Actually you may if they work for you!) • OS threads are relatively expensive • Max # open sockets >> max # threads • Preemptive scheduling causes races – "solved" with locks Async I/O without threads • select(), poll(), etc. • asyncore :-( • write your own • frameworks, e.g. Twisted, Tornado, zeroMQ • Wrap C libraries, e.g. libevent, libev, libuv • Stackless, gevent, eventlet • (Some overlap)

Downsides • Too many choices • Nobody likes callbacks • APIs not always easy • Standard library doesn't cooperate

Let's standardize the event loop • At the bottom of all of these is an event loop – (that is, all except OS threads) • Event loop multiplexes I/O • Various other features also common Why is the event loop special? • Serializes event handling – handle only one event at a time • There should be only one – otherwise it's not serializing events • Each framework has its own event loop API – even though the functionality has much overlap What functionality is needed? • start, stop running the loop – variant: always running • schedule callback DT in the future (may be 0) – also: repeated timer callback • set callback for file descriptor when ready – variant: call when I/O done Interop • Most frameworks don't interoperate • There's a small cottage industry adapting the event loop from framework X to be usable with framework Y – Tornado now maintains a Twisted adapter – There's also a zeroMQ adapter for Tornado – I hear there's a gevent fork of Tornado – etc. Enter PEP 3156 and Tulip

I know this is madness • Why can't we all just use Tornado? • Let's just import Twisted into the stdlib • Standardizing gevent solves all its problems – no more monkey-patching – greenlets in the language • Or maybe use Stackless Python? • Why reinvent the wheel? – libevent/ev/uv is the industry standard Again: PEP 3156 and Tulip • I like to write clean code from scratch • I also like to learn from others • I really like clean interfaces • PEP 3156 and Tulip satisfy all my cravings What is PEP 3156? What is Tulip? • PEP 3156: – standard event loop interface – slated for Python 3.4 • Tulip: – experimental prototype (currently) – reference implementation (eventually) – additional functionality (maybe) – works with Python 3.3 (always) PEP 3156 is not just an event loop • It's also an interface to change the event loop implementation (to another conforming one) – this is the path to framework interop – (even gevent!) • It also proposes a new way of writing callbacks – (that doesn't actually use callbacks) But first, the event loop • Influenced by both Twisted and Tornado • Reviewed by (some) other stakeholders • The PEP is not in ideal state yet • I am going to sprint Mon-Tue on PEP and Tulip Event loop method groups • starting/stopping the loop • basic callbacks • I/O callbacks • thread interactions • socket I/O operations • higher-level network operations Starting/stopping the event loop • run() # runs until nothing more to do • run_forever() • run_once([timeout]) • run_until_complete(future, [timeout]) • stop() • May change these around a bit Basic callbacks • call_soon(callback, *args) • call_later(delay, callback, *args) • call_repeatedly(interval, callback,

• args) • call_soon_threadsafe(callback, *args) • All return a Handler instance which can be used to cancel the callback I/O callbacks • add_reader(fd, callback, *args) -> Handler • remove_reader(fd) • add_writer(fd, callback, *args) -> Handler • remove_writer(fd) • Not all fd types are always acceptable • fd may be an object with a fileno() method UNIX signals • add_signal_handler(sig, callback,
• args) -> Handler • remove_signal_handler(sig) • Raise RuntimeError? if signals are unsupported Thread interactions • wrap_future(future) -> Future • run_in_executor(executor, callback,
• args) -> Future • Used to run code in another thread – sometimes there is no alternative – e.g. getaddrinfo(), database connections • Threads may use call_soon_threadsafe() Socket I/O operations • sock_recv(sock, nbytes) -> Future • sock_sendall(sock, data) -> Future • sock_accept(sock) -> Future • sock_connect(sock, address) -> Future • Only transports should use these High-level network operations • getaddrinfo(host, port, ...) -> Future

getnameinfo(address, [flags]) -> Future • create_connection(factory, host, port, ...) -> Future • start_serving(factory, host, port, ...) -> Future • Use these in your high-level code Um, Futures? • Like PEP 3148 Futures (new in Python 3.2): – from concurrent.futures import Future – f.set_result(x), f.set_exception(e) – f.result(), f.exception() – f.add_done_callback(func) – wait(fs, [timeout, [flags]]) -> (done, not_done) – as_completed(fs, [timeout]) -> <iterator> • However, adapted for use with coroutines Um, coroutines? • Whoops, let me get back to that later What's a Future? • Abstraction for a value to be produced later – Also known as Promises (check wikipedia) – Per wikipedia, these are explicit futures • API: – result() blocks until result is ready – an exception is a "result" too: will be raised! – exception() blocks ands checks for exceptions – done callbacks called when result/exc is ready Futures and coroutines • Not the concurrent.futures.Future class! • Nor exactly the same API • Where PEP 3148 "blocks", we must use... Drum roll, please PEP 380: yield-from

I cannot possibly do this justice • The best way to think about this is that yield-from is magic that "blocks" your current task but does not block your application • It's almost best to pretend it isn't there when you squint (but things don't work without it) • PS. @coroutine / yield-from are very close to async / await in C# How to think about Futures • Most of the time you can forget they are there • Just pretend that: data = yield from <function_returning_future> is equivalent to: data = <equivalent_blocking_function> ...and keep calm and carry on • Also forget about result(), exception(), and done-callbacks Error handling • Futures can raise exceptions too • Just put a try/except around the yield-from: • try: data = yield from loop.sock_connect(s, h, p) except OSError: <error handling code> Coroutines • Yield-from must be used inside a generator • Use @coroutine decorator to indicate that you're using yield-from to pass Futures • Coroutines are driven by the yield- from • Without yield-from a coroutine doesn't run • What if you want an autonomous task? Tasks • Tasks run as long as the event loop runs • A Task is a coroutine wrapped in a Future • Two ways to create Tasks: – @task decorator (instead of @coroutine) – f = Task(some_coroutine()) • The Task makes sure the coroutine runs "

"

codygman 1 day ago

Unicode support, concurrency/parallelism, and monkey patching were very big reasons for me using Go rather than python.

reply

aba_sababa 1 day ago

The only reason I'm not doing that is because I'm anchored by Python's stats libraries.

reply

visitorx 1 day ago

seconded, I doubt many a sane person would use Python for IO bound concurrency, they should definitely be looking at Rust (in the future slightly) and Erlang (currently the defacto standard for easy-of-use concurrency).

reply "

"

Ixiaus 2 days ago

I used gevent at first for a project that needed Async I/O and it worked really well, but then I switched to Erlang and I realized how poor a choice Python is for such tasks. The language really needs to be designed from the start for it (like Go, Rust, Erlang &c... Haskell wasn't designed from start for it but because of it's functional purity, bolting it on was "natural" but - it isn't so for Python IMHO).

reply

craigyk 2 days ago

Yeah, if you're going to have to do async to be performant than it better be pretty pervasive throughout all the libraries. Bonus points if the language supports syntax to make async easier as well. Node is beating out Python for server stuff not simply because it is "async", but because it is so much FASTER. The speed of V8 vs. CPython is a big part of that. In fact, vanilla JS doesn't have much to make async programming particularly easy: it has verbose function declaration and no yield mechanisms. Even library-level solutions like promises are merely 'OK'.

Still, it is easier to build a fast server that can handle streams in Node than it is in Python. Async Python? I'll just stick to async JS in that case. "

" nashequilibrium 1 day ago

The reason why you getting so many downvotes is because your comment is not just silly but flat wrong. Twisted, Tornado were around before nodejs. There are also async frameworks that make writing async code the same as synchronous code. I like tornado but I am using nodejs for my current app because of the libraries. This is where nodejs really shines, the community and libraries are awesome. Twisted has a lot of libraries but it has so much going on that many developers find it too complex. Tornado is a much simpler async framework to adapt to and allows you to run twisted libraries.

Twisted's inlinecallback's and tornado's gen module gets rid of all the async spaghetti code. This is hard to do with nodejs but I still chose nodejs because the available libraries made my project quicker to develop.

reply "

"

rdtsc 1 day ago

Sorry I didn't express myself correctly (see my reply to akuchling below).

Basically yes, Python had Twisted for years, it had Diesel, Monocle, Tornado, and some other ones. I am aware of those and as you've read my comment you saw that I used Twisted enough to know its ins and outs (5 years).

> There are also async frameworks that make writing async code the same as synchronous code.

Yes there is inlineCallbacks and I used. Node.js also has async (https://github.com/caolan/async). But you don't address the main problem that I raised -- fragmentation of libraries. Python is great because it comes with batteries, and then you can find even more batteries everywhere, _except_ if you use an async framework like Twisted, which, percolates all the way through you API code. Once your socket.recv() returns a Deferred(), that deferred will bubble up all the way to the user interface. So you now you end up searching or recreating a parallel set of libraries.

> Twisted has a lot of libraries but it has so much going on that many developers find it too complex.

It is too complex with too many libraries for those who want to take it up but it is not complex and doesn't have enough libraries if you are in it already -- every library you use has to be Twisted now. That's the danger of inventing a new framework.

Yes it will be standard, but there is already a practical standard -- eventlet and gevent. This is somethin Node.js doesn't have. I will personally take monkey-patching and the danger that my function will context switch inside while doing IO over using Twisted. I saw a practical benefit from it at least. "

"

pjscott 2 days ago

YES. Thank you for saying this. Some of the async code that I maintain uses Twisted and some of it uses Eventlet, and the difference between them is night and day. The code using Eventlet is so much cleaner, so much easier to maintain, and (oddly enough) so much less magical than the Twisted stuff. This was written by the same people, and they're all really good programmers, so the obvious confounding variables are not an issue here. Eventlet and Gevent are just so much better.

Worried about monkey-patching? Then only monkey-patch the parts you need to be asynchronous. Worried about magic that you don't understand? Have a look at the code; the magic is actually pretty straightforward after you've paid a little attention to the man behind the curtain.

If you're interested in async stuff for Python, I urge you to have a look at Eventlet or Gevent. "

"

eurleif 1 day ago

One thing I love about gevent is that you can share code between async and non-async. Most of my project benefits from async IO, but there's one part that needs to use a lot of CPU within a single process. So that part uses multithreaded Jython, the rest uses gevent, the common code is shared, and it all just works.

reply

rdtsc 1 day ago

Exactly, I was surprised how in the whole "ideas" mailing list discussion Guido had and in other forums that is dismissed as "meh" not even mentioned.

Discussions quickly turn theoretical and academic. "But you don't know when your green threads will switch, man, so I'll add yields in there for you". Yes, and then also make sure there is a complete universe of libraries.

Python is awesome not just because it is fun to write little hello world examples in (so is Logo), it is awesome because it is easy to GetShitDone?(TM) quickly. The big part of GetShitDone?(TM) quickly is reusing libraries not rewriting everything from scratch.

Using an exotic database for some reason -- great. Found a Python library to interact with it -- great. Oh but my framework is based on Deferreds and this one was written without Deferreds or this one returns Futures. Sorry, go write your own from scratch.

This has been the story of my life for 5+ years search or re-writing Twisted version of already existing libraries.

Now at least just adopt Twisted and go with it if they are going this route. But now, they are 'standardizing' on something new. I think had they done this in 2007, yeah rock on, that would have made sense. They didn't. What saved and kept Python on the back-end during the past 5 or so years was greenlet (eventlet and gevent). Guido is kicking all those people in nuts and saying, "no", we'll do Twisted now (with some changes).

reply "

"

rdw 1 day ago

As someone who's contributed to Eventlet, I've always felt that the only way that it could get over this scary hurdle (and it is legitimately scary) is for it to be integrated within Python itself. Almost all of the weird problems come from fighting with the baked-in assumptions of the Python runtime. Eventlet does try to alleviate the scariness a little bit by allowing you to import "greened" modules rather than changing the global versions, but that has its own problems.

If it were integrated with Python, there would be no monkeypatching, no special magic, it would be just how things work. That said, I'm not at all surprised that Guido doesn't favor a coroutine-based solution; his opposition to general coroutines is as famous as his opposition to anonymous functions. (to clarify: I don't think the @coroutine decorator creates "real" coroutines, any more than generators were already coroutines) "

"

fzzzy 1 day ago

From a certain perspective it is a rational decision. Because the CPython API relies so heavily on the C stack, either some platform-specific assembly is required to slice up the C stack to implement green threads, or the entire CPython API would have to be redesigned to not keep the Python stack state on the C stack.

Way back in the day [1] the proposal for merging Stackless into mainline Python involved removing Python's stack state from the C stack. However there are complications with calling from C extensions back into Python that ultimately killed this approach.

After this Stackless evolved to be a much less modified fork of the Python codebase with a bit of platform specific assembly that performed "stack slicing". Basically when a coro starts, the contents of the stack pointer register are recorded, and when a coro wishes to switch, the slice of the stack from the recorded stack pointer value to the current stack pointer value is copied off onto the heap. The stack pointer is then adjusted back down to the saved value and another task can run in that same stack space, or a stack slice that was stored on the heap previously can be copied back onto the stack and the stack pointer adjusted so that the task resumes where it left off.

Then around 2005 the Stackless stack slicing assembly was ported into a CPython extension as part of py.lib. This was known as greenlet. Unfortunately all the original codespeak.net py.lib pages are 404 now, but here's a blog post from around that time that talks about it [2].

Finally the relevant parts of greenlet were extracted from py.lib into a standalone greenlet module, and eventlet, gevent, et cetera grew up around this packaging of the Stackless stack slicing code.

So you see, using the Stackless strategy in mainline python would have either required breaking a bunch of existing C extensions and placing limitations on how C extensions could call back into Python, or custom low level stack slicing assembly that has to be maintained for each processor architecture. CPython does not contain any assembly, only portable C, so using greenlet in core would mean that CPython itself would become less portable.

Generators, on the other hand, get around the issue of CPython's dependence on the C stack by unwinding both the C and Python stack on yield. The C and Python stack state is lost, but a program counter state is kept so that the next time the generator is called, execution resumes in the middle of the function instead of the beginning.

There are problems with this approach; the previous stack state is lost, so stack traces have less information in them; the entire call stack must be unwound back up to the main loop instead of a deeply nested call being able to switch without the callers being aware that the switch is happening; and special syntax (yield or yield from) must be explicitly used to call out a switch.

But at least generators don't require breaking changes to the CPython API or non-portable stack slicing assembly. So maybe now you can see why Guido prefers it.

Myself, I decided that the advantages of transparent stack switching and interoperability outweighed the disadvantages of relying on non-portable stack slicing assembly. However Guido just sees things in a different light, and I understand his perspective.

  [1] http://www.python.org/dev/peps/pep-0219/
  [2] http://agiletesting.blogspot.com/2005/07/py-lib-gems-greenlets-and-pyxml.html"

-- https://news.ycombinator.com/item?id=5395385

---

(has this been fixed by http://www.python.org/dev/peps/pep-0342/ ? )

http://stackoverflow.com/questions/346365/why-cant-yield-return-appear-inside-a-try-block-with-a-catch

---

> (1) It's a whole lot easier to debug a problem with gevent than with > anything > which uses yield / Deferreds / asyncore / whatever. With gevent, you get a > standard stack trace. With anything else, the "where did this call come > from" > information is not part of the call chain and thus is either unavailable, > or > will have to be carried around preemptively (with associated overhead). >

mb relevant to http://lambda-the-ultimate.org/node/4592 , see my Filtered1 note.

http://eventlet.net/

http://sdiehl.github.com/gevent-tutorial/

http://blog.gevent.org/2010/02/27/why-gevent/

---

"

• While gevent is not preemptive, context switches are implicit, that means you cannot write code without locks unless you want to do some handwaving. You have to know the entire system to know that a multistep operation that calls functions is actually safe. " -- https://news.ycombinator.com/item?id=3751510

" ...Some of the async code that I maintain uses Twisted and some of it uses Eventlet, and the difference between them is night and day. The code using Eventlet is so much cleaner, so much easier to maintain, and (oddly enough) so much less magical than the Twisted stuff. This was written by the same people, and they're all really good programmers, so the obvious confounding variables are not an issue here. Eventlet and Gevent are just so much better. "

" I am surprised that gevent was dismissed. Ok, there is also eventlet, if someone doesn't like gevent. Monkey patching is scary? Is it really _that_ scary? Most sane and portable IO code probably runs on that today. Why? Because there is no need to create a parallel world of libraries. Write a test does it pass? Does it handle your use case? I'll take not knowing if my green threads switch tasks and add another green thread lock, instead of doubling my code size with yields, callbacks and futures.

Let's talk about Twisted (sorry couldn't resist WAT video reference). I remember for years searching for parallel libraries to parse exotic protocols. Regular Python library is there, but no, can't use that, sorry. Gotta go find or write one that returns Deferreds. You add a single Twisted module in your code, good luck! -- it ripples all the way to the top through your API and you are doomed being locked into the Twisted world forever.

When gevent and eventlet came around it was like a breath of fresh air. This is what sane concurrent IO looks like in Python:

http://eventlet.net/doc/examples.html

My fear is that many will just say fuck it, I'll just use Go/Rust/Erlang for IO bound concurrent problems. "

"

eurleif 2 days ago

One thing I love about gevent is that you can share code between async and non-async. Most of my project benefits from async IO, but there's one part that needs to use a lot of CPU within a single process. So that part uses multithreaded Jython, the rest uses gevent, the common code is shared, and it all just works. "

"

codygman 2 days ago

Unicode support, concurrency/parallelism, and monkey patching were very big reasons for me using Go rather than python.

"

" visitorx 2 days ago

seconded, I doubt many a sane person would use Python for IO bound concurrency, they should definitely be looking at Rust (in the future slightly) and Erlang (currently the defacto standard for easy-of-use concurrency).

reply "

"

>> "@coroutine / yield-from are very close to async / await in C# 5" "

---

" Maybe it's because I had a very similar feeling about Python (indentation matters? Having to pass self explicitly to methods? No ++? No assignment in if, while, etc.? I hated all this things at first, but after understanding the rationale and using then in real work, it works great!). "

---

1] What does this program do?

    print object() > object()

reply

raverbashing 9 hours ago

About your code snippet, it prints a "random" boolean value

You're creating two objects (with random addresses, which affect the __str__ method result, which in turn result in a string comparison that returns False or True)

reply

ptx 6 hours ago

  >>> print(object() > object())
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: unorderable types: object() > object()

Yet another reason to upgrade to Python 3! :)

---

"Friday, March 23, 2012 PyCon?: Keynote: Guido Van Rossum Take the survey at svy.mk/pycon2012survey.

Videos are already being uploaded to pyvideo.org.

Guido is wearing a "Python is for girls" T-shirt.

He showed a Python logo made out of foam on a latte. Yuko Honda really did it.

He had some comments about trolls.

Troll, n:

    A leading question whose main purpose is to provoke an argument that cannot be won.
    A person who asks such questions.

Guido said that Python, Ruby, and Perl are exactly the same language from 10,000 feet. They should be our friends. Lisp is not all that different from Python, although you might not notice it. Ruby is not kicking Python's butt in any particular case.

They added unicode literals back to Python 3.3 just to make compatibility with Python 2.7 easier.

"I should have used a real presentation tool instead of Google Docs." (Doh! He's a Googler!)

More and more projects are being ported to Python 3.

People are actually capable of using one code base for Python 2.7 and Python 3.3. There's a helpful library called "Six".

The Python 3 migration is painful now, but it's going to be worth it.

If PyPy? is faster than cPython, why not abandon cPython? No one is using PyPy? in production. PyPy? is still hard for people writing C extensions. cPython is still a great glue system (for Fortran, C, and random binary code). "PyPy? still has a long way to go. Speed is not everything." PyPy? and cPython will live together in slightly different contexts for years. cPython will remain the reference implementation.

People still criticize dynamic typing, saying it's not safe. "The type checking in compilers is actually really weak." For instance, most of the time, a type checker can't tell you if an int represents inches or centimeters. The only way to create good code is to test, audit, hire smart people, etc. Test! A type checking compiler can only find a narrow set of errors; no one has conquered the problem of writing perfect software.

A large program written in a statically typed language will often have a dynamically typed, interpreted subsystem, done via string manipulation, or something else.

Dynamic typing is not inherently inferior to static typing.

Guido hopes people are "doing interesting things with static analysis."

Guido pronounces GIL as "Jill".

"OS level threads are meant for parallel IO, not for parallel computation." Use separate processes for each core. One team was using Python on a machine with 64,000 cores. "You have to be David Beazley to come up with the counterexample."

Guido has complaints about each of the async libraries, but he doesn't have a solution that he recommends. It's not a problem he has to deal with. gevent is interesting. He can't add stackless to Python because it would make other implementations much more complicated or impossible. He doesn't want to put an event loop in the stdlib because not everyone needs it. Some things are better developed outside of Python. He's not a fan of callback-style approaches; he would prefer the gevent approach. He likes synchronous looking code that under the hood does async, without using threads. All these things deserve more attention.

We might want to make Python 4 the new language to solve the concurrent programming problem. He wants to see lots of alternatives.

Introducting a new language to replace JavaScript?