proj-oot-ootNotes11

potentially interesting search (though maybe not):

https://www.google.com/search?q=cons+concatenate+lift+elementwise&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official

---

BKuhn likes http://vertx.io/manual.html

http://lwn.net/Articles/615834/

---

kdbus

http://lwn.net/Articles/580194/

" Unlike most other kernels, Linux has never had a well-designed IPC mechanism. Windows and Mac OS have this feature; even Android, based on Linux, has one in the form of the "binder" subsystem. Linux, instead, has only had the primitives — sockets, FIFOs, and shared memory — but those have never been knitted together into a reasonable application-level API. Kdbus is an attempt to do that knitting and create something that is at least as good as the mechanisms found on other systems.

Linux does have D-Bus, which he said, is a powerful IPC system; it is the closest thing to a standard in this area as can be found on Linux. Lennart put up an extensive list of advantages to using D-Bus. It provides a nice method-call transaction mechanism (allowing for sending a message and getting a response) and a means for sending "signals" (notifications) to the rest of the system. There is a discovery mechanism to see what else is running on the bus and the introspection facilities needed to learn about [Lennart Poettering] what services are offered. D-Bus includes a mechanism for the enforcement of security policies, a way of starting services when they are first used, type-safe marshaling of data structures, and passing of credentials and file descriptors over the bus. There are bindings for a wide range of languages and network transparency as well.

On the other hand, D-Bus also suffers from a number of limitations. It is well suited to control tasks, but less so for anything that has to carry significant amounts of data. So, for example, D-Bus works well to tell a sound server to change the volume, but one would not want to try to send the actual audio data over the bus. The problem here is the fundamental inefficiencies of the user-space D-Bus implementation; a call-return message requires ten message copies, four message validations, and four context switches — not the way to get good performance. Beyond that, credential passing is limited, there are no timestamps on messages, D-Bus is not available at early boot, connections to security frameworks (e.g. SELinux) must happen in user space, and there are race conditions around the activation of services. D-Bus also suffers from what Lennart described as a "baroque code base" and heavy use of XML.

...

That implementation is kdbus, an in-kernel implementation of D-Bus. This implementation is able to carry large amounts of data; it can be reasonably used for gigabyte-sized message streams. It can perform zero-copy message passing, but even in the worst case, a message and its response are passed with no more than two copy operations, two validations, and two context switches. Full credential information (user ID, process ID, SELinux label, control group information, capabilities, and much more) is passed with each message, and all messages carry timestamps. Kdbus is always available to the system (no need to wait for the D-Bus daemon to be started), Linux security modules can hook into it directly, various race conditions have been fixed, and the API has simplified.

Kdbus is implemented as a character device in the kernel; processes wishing to join the bus open the device, then call mmap() to map a message-passing area into their address space. Messages are assembled in this area then handed to the kernel for transport; it is a simple matter for the kernel to copy the message from one process's mapped area to another process's area. Messages can include timeouts ("method call windows") by which a reply must be received. There is a name registry that is quite similar to the traditional D-Bus registry.

The "memfd" mechanism enables zero-copy message passing in kdbus. A memfd is simply a region of memory with a file descriptor attached to it; it operates similarly to a memory-mapped temporary file, "but also very differently." A memfd can be "sealed," after which the owning process can no longer change its contents. A process wishing to send a message will build it in the memfd area, seal it, then pass it to kdbus for transport. Depending on the size of the message, the relevant pages may just be mapped into the receiving process's address space, avoiding a copy of the data. But the break-even point is larger than one might expect; Lennart said that it works better to simply copy anything that is less than about 512KB. Below that size, the memory-mapping overhead exceeds the savings from not copying the data.

Memfds can be reused at will. A process that needs to repeatedly play the same sound can seal the sample data into a memfd once and send it to the audio server whenever it needs to be played. All told, Lennart said, memfds work a lot like the Android "ashmem" subsystem.

The signal broadcasting mechanism has been rewritten to use Bloom filters to select recipients. A Bloom filter uses a hash to allow the quick elimination of many candidate recipients, making the broadcast mechanism relatively efficient. "

---

should probably review the great video game console and framework APIs, and also Linden Scripting Language, since i want this to be a good programming language for that kind of thing, except for the performance requirements of video games (that is, i want it to be a good scripting language for cyberspace/the metaverse, serving the same role as Linden Scripting Language).

i guess some of the things i should review are:

a lecture by Jessica D. Bayliss notes: " Game specialty languages Second Life scripting language (Second Life) UnrealScript? (Unreal Tournament) NWN script (Neverwinter Nights) Customizing other languages Lua (World of Warcraft) Python (Civ IV) C/C++ (Half Life II) "

also, OpenSim? uses C#: http://opensimulator.org/wiki/Scripting_Languages and "OSL" which is a guess a descendent of LSL?: http://opensimulator.org/wiki/Scripting_Documentation http://opensimulator.org/wiki/OSSL http://opensimulator.org/wiki/OSSL_Proposals

---

incidentally, should look at what makes me like Grape, the Ruby API framework

---

rust example from https://medium.com/@adamhjk/rust-and-go-e18d511fbd95 :

let r = regex!(r"refs/heads/(.+)"); let caps_result = r.captures(head_ref.as_slice()); let branch: String = match caps_result { Some(cap) => { String::from_str(cap.at(1)) }, None => { panic!("You must be somewhere on refs/heads, and you aren't") } };

in oot, would like to have the 'match' construct be extensible, so it could do not only vanilla destructuring, but also regexes, and whatever else the user adds.

this probably goes well with my idea that destructuring be done via some interface/protocol for an object to say if it should be considered to have been constructed by Cons or by Null or whatever; the same interface could allow an object to say, yes it matches regex R, and here are the captures.

---

https://code.google.com/p/american-fuzzy-lop/

---

to do wrappers, need a way to say "pass on all arguments to this function except for these: .."

---

in python,

f(blah, *varArgs, kwArgs)

is special function syntax, eg

a = (0, *[1,2,3], 4) is illegal

but we could just do it more generally

--- from https://news.ycombinator.com/item?id=8394381

michaelsbradley 38 days ago

link

I'm fairly certainly that he has in mind the vision put forward in Prof. David Harel's late-career manifesto Can Programming be Liberated, Period? (2008)[1]

Five years earlier, Prof. Harel wrote a book, Come, Let's Play[2], in which he explored the role that temporal logic[3] could have in the building of reactive software systems (e.g. desktop GUI applications, among many others). If you visit Harel's full list of publications[4] and do an in-page search for the terms "LSC", "behavioral" and "scenario", you can follow the links to a number of papers published since the early 2000s, related to the application of temporal logic in the process of software design.

It's possible that my connection of Granger's "eve" to Harel's work is completely mistaken; and if so, I apologize for jumping the gun.

[1] http://www.wisdom.weizmann.ac.il/~harel/papers/LiberatingPro...

[2] http://www.wisdom.weizmann.ac.il/~playbook/

[3] http://en.wikipedia.org/wiki/Temporal_logic

[4] http://www.wisdom.weizmann.ac.il/~harel/papers.html


seanmcdirmid 38 days ago

link

Prof. Rodney Brooks has always had a better more pragmatic take on this, coming from the robotics field no less:

http://en.m.wikipedia.org/wiki/Subsumption_architecture

http://en.m.wikipedia.org/wiki/Behavior-based_robotics

Kodu was based on this model, and it worked very well.


jamii 38 days ago

link

We did read the manifesto and I'm fairly convinced that Harel's vision of interactive specification and refinement is the the long-term future of software. I'm not sure how to get there though, short of general AI. Eve aims to reduce the amount of specification required and can make sensible guesses about implementation, but it is not able to 'fill in the blanks' for partial specifications and we won't be focusing on that any time soon.


daralthus 38 days ago

link

Thinking of programming as a dialogue with the computer is a really strong metaphor that I gladly recognize as something that inspired you [1][2].

Short: You provide the goals and the constraints, then through a back and forth dialogue with the computer you describe and clarify the problem (and also your understanding). You keep on negotiating and providing more constraints until it simply finds a possible solution (the problem domain is now computable) or the best fitting solution now satisfies your needs (in time).

The hardest parts are probably: 1. Collecting, developing the constraint solvers. Should we imagine that collection more as platform of user contributed (programmed/trained) packages? 2. Developing the dialouge that is not just textual but visual (in both ways) (notepad vs IDE vs Theater(?!) don't forget macros/acting out!)). Showing is always faster then writing and gives one better cues on the closures/framework of the actual discussion.

[1] http://scattered-thoughts.net/blog/2014/07/21/imperative-thi... [2] https://gist.github.com/jamii/8957881c8eaa4035f4ae


mtrimpe 38 days ago

link

Seems very closely related to what I've been talking about / working on lately. Cool!

https://speakerdeck.com/mtrimpe/graphel-the-meaning-of-an-im...

---

zmanian 37 days ago

link

For someone to become a effective creator of programs today, they need to learn

Outside of academic environments you need to learn all of these thing simultaneously. If we can collapse away a few of these things with minimal loss of productivity we make programming much accessible.


jamesrcole 37 days ago

link

> Knowing some programming language isn't it, its a mode of thinking and analysis.

You seem to be implying that we currently have tools that provide a suitably natural and frictionless medium for such thinking and analysis. But what if our current languages/tools are more like Roman numerals and there's still the computing version of Arabic numerals out there to be found?

---

intro to database stuff:

https://medium.com/@jeeyoungk/why-i-love-databases-1d4cc433685f

---

discussion on SQL replacements:

https://news.ycombinator.com/item?id=8579940

---

in atr, there are different ways to get historical data depending on if you want it from the database or from the broker, whether you want a lot of it at once or just a little, etc. It would be clearer if, at one level of description, you just said 'get', and then at another level, specified all that junk; i think that's just a library/API design error of mine in designing atr, but perhaps it's something relevant to language design. The obvious way to fix it at the API level would be to make a generic 'get_historical' wrapper that then consulted current conditions and some configuration data to decide how to get it, but that requires specifying the policy for making this decision, which does not seem like it is relevant to the 'higher' level of description. Perhaps this relaates to Alan Key's separation of policy from tactics.

another way of putting that is that we may want various 'levels' of detail in descriptions.

at some level you may even want to just say "get the historical data on SPY" without specifying how much data to get, which columns to get, how to get it and from where, what the error handling policy (including timeouts) is, and in what form (data representation} to get the data.

---

"

Inferring algebraic effects

Logical methods in computer science just published Matija Pretnar's latest take on algebraic effects and handlers:

    We present a complete polymorphic effect inference algorithm for an ML-style language with handlers of not only exceptions, but of any other algebraic effect such as input & output, mutable references and many others. Our main aim is to offer the programmer a useful insight into the effectful behaviour of programs. Handlers help here by cutting down possible effects and the resulting lengthy output that often plagues precise effect systems. Additionally, we present a set of methods that further simplify the displayed types, some even by deliberately hiding inferred information from the programmer. 

Pretnar and Bauer's Eff has made previous appearances here on LtU?. Apart from the new fangled polymorphic effect system, this paper also contains an Eff tutorial. By Ohad Kammar at 2014-09-27 23:16

Functional Implementation Paradigms Semantics Theory 8 comments other blogs 6856 reads

"

Facebook releases "Flow", a statically typed JavaScript? variant

    The goal of Flow is to find errors in JavaScript code with little programmer effort. Flow relies heavily on type inference to find type errors even when the program has not been annotated - it precisely tracks the types of variables as they flow through the program.
    At the same time, Flow is a gradual type system. Any parts of your program that are dynamic in nature can easily bypass the type checker, so you can mix statically typed code with dynamic code.
    Flow also supports a highly expressive type language. Flow types can express much more fine-grained distinctions than traditional type systems. For example, Flow helps you catch errors involving null, unlike most type systems. 

Read more here. Here's the announcement from Facebook. By James Iry at 2014-11-18 19:19

Why do we need modules at all?
General 6 comments other blogs 8062 reads

Post by Joe Armstrong of Erlang fame. Leader:

    Why do we need modules at all? This is a brain-dump-stream-of-consciousness-thing. I've been thinking about this for a while. I'm proposing a slightly different way of programming here. The basic idea is:
        do away with modules
        all functions have unique distinct names
        all functions have (lots of) meta data
        all functions go into a global (searchable) Key-value database
        we need letrec
        contribution to open source can be as simple as contributing a single function
        there are no "open source projects" - only "the open source Key-Value database of all functions"
        Content is peer reviewed 
    Why does Erlang have modules? There's a good an bad side to modules. Good: Provides a unit of compilation, a unit of code distribution. unit of code replacement. Bad: It's very difficult to decide which module to put an individual function in. Break encapsulation (see later). 

By Sean McDirmid? at 2014-11-05 23:56

Conservation laws for free!
LtU? Forum 223 comments other blogs 16164 reads

In this year's POPL, Bob Atkey made a splash by showing how to get from parametricity to conservation laws, via Noether's theorem:

    Invariance is of paramount importance in programming languages and in physics. In programming languages, John Reynolds’ theory of relational parametricity demonstrates that parametric polymorphic programs are invariant under change of data representation, a property that yields “free” theorems about programs just from their types. In physics, Emmy Noether showed that if the action of a physical system is invariant under change of coordinates, then the physical system has a conserved quantity: a quantity that remains constant for all time. Knowledge of conserved quantities can reveal deep properties of physical systems. For example, the conservation of energy, which by Noether’s theorem is a consequence of a system’s invariance under time-shifting.
    In this paper, we link Reynolds’ relational parametricity with Noether’s theorem for deriving conserved quantities. We propose an extension of System Fω with new kinds, types and term constants for writing programs that describe classical mechanical systems in terms of their Lagrangians. We show, by constructing a relationally parametric model of our extension of Fω, that relational parametricity is enough to satisfy the hypotheses of Noether’s theorem, and so to derive conserved quantities for free, directly from the polymorphic types of Lagrangians expressed in our system. 

By Ohad Kammar at 2014-10-28 07:52

Category Theory Fun Functional Lambda Calculus Scientific Programming Semantics Theory Type Theory 5 comments other blogs 5715 reads

Breaking the Complexity Barrier of Pure Functional Programs with Impure Data Structures

Breaking the Complexity Barrier of Pure Functional Programs with Impure Data Structures by Pieter Wuille and Tom Schrijvers:

    Pure functional programming language offer many advantages over impure languages. Unfortunately, the absence of destructive update, imposes a complexity barrier. In imperative languages, there are algorithms and data structures with better complexity. We present our project for combining existing program transformation techniques to transform inefficient pure data structures into impure ones with better complexity. As a consequence, the programmer is not exposed to the impurity and retains the advantages of purity.

This paper is along the same lines a question I asked a couple of years ago. The idea here is to allow programming using immutable interfaces, and then automatically transform it into a more efficient mutable equivalent. By naasking at 2014-09-22 14:10

Functional Theory 1 comment other blogs 5522 reads

An operational and axiomatic semantics for non-determinism and sequence points in C

In a recent LtU? discussion, naasking comments that "I always thought languages that don't specify evaluation order should classify possibly effectful expressions that assume an evaluation order to be errors". Recent work on the C language has provided reasonable formal tools to reason about evaluation order for C, which has very complex evaluation-order rules.

An operational and axiomatic semantics for non-determinism and sequence points in C Robbert Krebbers 2014

    The C11 standard of the C programming language does not specify the execution order of expressions. Besides, to make more effective optimizations possible (e.g. delaying of side-effects and interleav- ing), it gives compilers in certain cases the freedom to use even more behaviors than just those of all execution orders.
    Widely used C compilers actually exploit this freedom given by the C standard for optimizations, so it should be taken seriously in formal verification. This paper presents an operational and ax- iomatic semantics (based on separation logic) for non-determinism and sequence points in C. We prove soundness of our axiomatic se- mantics with respect to our operational semantics. This proof has been fully formalized using the Coq proof assistant.

One aspect of this work that I find particularly interesting is that it provides a program (separation) logic: there is a set of inference rules for a judgment of the form Δ;J;R⊢{P}s{Q}, where s is a C statement and P,Q are logical pre,post-conditions such that if it holds, then the statement s has no undefined behavior related to expression evaluation order. This opens the door to practical verification that existing C program are safe in a very strong way (this is all validated in the Coq theorem prover). By gasche at 2014-09-14 09:36

Lambda Calculus 2 comments other blogs 6356 reads

Luca Cardelli Festschrift

Earlier this week Microsoft Research Cambridge organised a Festschrift for Luca Cardelli. The preface from the book:

    Luca Cardelli has made exceptional contributions to the world of programming
    languages and beyond. Throughout his career, he has re-invented himself every
    decade or so, while continuing to make true innovations. His achievements span
    many areas: software; language design, including experimental languages;
    programming language foundations; and the interaction of programming languages
    and biology. These achievements form the basis of his lasting scientific leadership
    and his wide impact.
    ...
    Luca is always asking "what is new", and is always looking to
    the future. Therefore, we have asked authors to produce short pieces that would
    indicate where they are today and where they are going. Some of the resulting
    pieces are short scientific papers, or abridged versions of longer papers; others are
    less technical, with thoughts on the past and ideas for the future. We hope that
    they will all interest Luca. 

Hopefully the videos will be posted soon. By Ohad Kammar at 2014-09-12 10:10

Category Theory Lambda Calculus Misc Books Semantics Theory Type Theory 4 comments other blogs 4170 reads

Re-thinking Prolog

A recent paper by Oleg Kiselyov and Yukiyoshi Kameyama at the university of Tsukuba discusses weaknesses and areas for improvement to Prolog.

    Quite many computations and models are mostly deterministic. Implementing them in Prolog with any acceptable performance requires the extensive use of problematic features such as cut. Purity is also compromised when interfacing with mainstream language libraries, which are deterministic and cannot run backwards. Divergence is the constant threat, forcing the Prolog programmers to forsake the declarative specification and program directly against the search strategy. All in all, Classical Prolog is the exquisite square peg in the world with mostly round holes 
    The strong points of Prolog can be brought into an ordinary functional programming language. Using OCaml as a representative, we implement lazy guessing as a library, with which we reproduce classical Prolog examples. Furthermore, we demonstrate parser combinators that use committed choice (maximal munch) and can still be run forwards and backwards. They cannot be written in Classical Prolog. Logic variables, unification, and its WAM compilation strategy naturally emerge as a "mere optimization" of the Herbrand universe enumeration. 

The paper mentions the strength of the approach used by miniKanren (which embeds logic programming with fairer search strategy than normal Prolog into Scheme) and Hansei (which embeds probability based nondeterminism into Ocaml using delimited continuations to allow direct-style expression of monadic code).

After motivating some choices by studying the prototypical example of running append backwards they cover running parsers with "maximal munch" rule backwards - something that cannot be (declaratively) expressed in prolog.

A very interesting paper on logic programming! It also thanks Tom Schrijvers of CHR fame at the end. By Alex 1 at 2014-09-11 14:31

LtU? Forum 22 comments other blogs 9869 reads

Scala woes?

A fork in the back? See discussion over at HN. People in the know are encouraged to shed light on the situation. By Ehud Lamm at 2014-09-06 12:12

Scala 48 comments other blogs 16498 reads

Stream Processing with a Spreadsheet

ECOOP 2014 paper (distinguished) by Vaziri et. al, abstract:

    Continuous data streams are ubiquitous and represent such a high volume of data that they cannot be stored to disk, yet it is often crucial for them to be analyzed in real-time. Stream processing is a programming paradigm that processes these immediately, and enables continuous analytics. Our objective is to make it easier for analysts, with little programming experience, to develop continuous analytics applications directly. We propose enhancing a spreadsheet, a pervasive tool, to obtain a programming platform for stream processing. We present the design and implementation of an enhanced spreadsheet that enables visualizing live streams, live programming to compute new streams, and exporting computations to be run on a server where they can be shared with other users, and persisted beyond the life of the spreadsheet. We formalize our core language, and present case studies that cover a range of stream processing applications.

By Sean McDirmid? at 2014-08-13 04:10

Safely Composable Type-Specific Languages
LtU? Forum login or register to post comments other blogs 17330 reads

Cyrus Omar, Darya Kurilova, Ligia Nistor, Benjamin Chung, Alex Potanin, and Jonathan Aldrich, "Safely Composable Type-Specific Languages", ECOOP14.

    Programming languages often include specialized syntax for common datatypes (e.g. lists) and some also build in support for specific specialized datatypes (e.g. regular expressions), but user-defined types must use general-purpose syntax. Frustration with this causes developers to use strings, rather than structured data, with alarming frequency, leading to correctness, performance, security, and usability issues. Allowing library providers to modularly extend a language with new syntax could help address these issues. Unfortunately, prior mechanisms either limit expressiveness or are not safely composable: individually unambiguous extensions can still cause ambiguities when used together. We introduce type-specific languages (TSLs): logic associated with a type that determines how the bodies of generic literals, able to contain arbitrary syntax, are parsed and elaborated, hygienically. The TSL for a type is invoked only when a literal appears where a term of that type is expected, guaranteeing non-interference. We give evidence supporting the applicability of this approach and formally specify it with a bidirectionally typed elaboration semantics for the Wyvern programming language.

By Allan McInnes? at 2014-08-11 06:27

DSL Implementation Software Engineering 3 comments other blogs 13946 reads

Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU)

We are having another PLATEAU workshop at SPLASH 2014. We have a new category for "Hypotheses Papers" and thought this would be particularly appealing to the LTU community.

http://2014.splashcon.org/track/plateau2014

Programming languages exist to enable programmers to develop software effectively. But how efficiently programmers can write software depends on the usability of the languages and tools that they develop with. The aim of this workshop is to discuss methods, metrics and techniques for evaluating the usability of languages and language tools. The supposed benefits of such languages and tools cover a large space, including making programs easier to read, write, and maintain; allowing programmers to write more flexible and powerful programs; and restricting programs to make them more safe and secure.

PLATEAU gathers the intersection of researchers in the programming language, programming tool, and human-computer interaction communities to share their research and discuss the future of evaluation and usability of programming languages and tools.

Some particular areas of interest are:

Submission Details

PLATEAU encourages submissions of three types of papers:

Research and position papers: We encourage papers that describe work-in-progress or recently completed work based on the themes and goals of the workshop or related topics, report on experiences gained, question accepted wisdom, raise challenging open problems, or propose speculative new approaches. We will accept two types of papers: research papers up to 8 pages in length; and position papers up to 2 pages in length.

Hypotheses papers: Hypotheses papers explicitly identify beliefs of the research community or software industry about how a programming language, programming language feature, or programming language tool affects programming practice. Hypotheses can be collected from mailing lists, blog posts, paper introductions, developer forums, or interviews. Papers should clearly document the source(s) of each hypothesis and discuss the importance, use, and relevance of the hypotheses on research or practice. Papers may also, but are not required to, review evidence for or against the hypotheses identified. Hypotheses papers can be up to 4 pages in length.

Papers will be published in the ACM Digital Library at the authors’ discretion.

Important Dates

Workshop paper submission due - 1 August, 2014 Notification to authors - 22 August, 2014 Early registration deadline - 19 September, 2014

Keynote

Josh Bloch, former Chief Java Architect at Google and Distinguished Engineer at Sun Microsystems. By craiganslow at 2014-07-17 22:19

LtU? Forum login or register to post comments other blogs 6362 reads

---

should have a more compact way to say the "row['time']" in the following (i guess "row.item" ?)

    def appendRow(self, item, duration, time = dtm.utcnow(), row):
        """
        note: mutates 'row'
        """
        #print 'trying to append'
        cf = self._getColumnFamily(duration, 'main')
        row['item'] = unicode(item)
        row['time'] = datetime64(time).astype(int)
        cf.insert(self._makeKey(item, time), row)

---

much of the following should be able to be replaced by a macro

    def handleHistoricalTrade(self, name, price=None, time=None, volume=None, venue=None, external_id_domain=None, external_id=None, buyer=None, seller=None, buyer_or_seller_initiated=None):
        row = {}
        if price is not None:
            row['price'] = price
        if volume is not None:
            row['volume'] = volume
        if venue is not None:
            row['venue'] = venue
        if external_id_domain is not None:
            row['external_id_domain'] = external_id_domain
        if external_id is not None:
            row['external_id'] = external_id
        if buyer is not None:
            row['buyer'] = buyer
        if seller is not None:
            row['seller'] = seller
        if buyer_or_seller_initiated is not None:
            row['buyer_or_seller_initiated'] = buyer_or_seller_initiated        

---

this way that KeyboardInterrput? always has to be excepted from catch-alls is annoying. Find a way to shorten this (again, maybe just a macro?):

        try:
            #print (name, field, duration, price, time)
            self.historyStorage.append(name, field, duration, price, time)
            
        except KeyboardInterrupt:
            raise
        except:
            self.log.warn('Exception when appending to CassandraTimeSeries', exc_info=True)

---

rust's config language:

https://github.com/toml-lang/toml

this guy likes it: https://news.ycombinator.com/item?id=8645475

---

http://www.cs.unm.edu/~fastos/05meeting/PLAN9NOTDEADYET.pdf

---

when interacting with another system, statefulness is common in 'minor' ways. For example, in various APIs, even when just sending unauthenticated GETs for public information, there is sometimes an injunction not to query the API too many times per minute, necessitating a stateful throttle in the client-side implementation. Even more common is the requirement to login first. I say this is 'minor' because the core operation being performed, a GET, is not by itself conceptually stateful.

Then there is the matter of caching. You could make caching transparent, but sometimes you need fine-grained control over it. Eg if the server is down, do you want the client-side API to hide this by returning the last value for a query? And blocking: if you don't want a cached value, and the server is down, or if you have hit your requests per minute limit and are throttled, do you want the query to block until it gets an answer, or return an exception?

---

it's verbose to have to write pycassa.index twice in an expression like:

        pycassa.index.create_index_expression('time', begin_time, pycassa.index.GTE) 

GTE should just be a keyword (.GTE)

---

oot should have a slightly more concise and more readable way to write this sort of thing from Python (a bunch of simple @properties, and composition, and a bunch of argument checking):

class SingleContractSingleCurrencyBroker?: def __init__(self, broker, contract, currency): """contract is a string, not a Contract object """ self.broker = broker self.contract = contract self.baseCurrency = currency

        self.broker.subscribePrice(self.contract)
    def cancelAllOrders(self, **kw):
        """note/todo: right now this just calls broker.cancelAllOrders, but
        in theory it should only cancel the orders on self.contract
        """
        self.broker.cancelAllOrders(**kw)
    def placeOrder(self, **kw):
        self.broker.placeOrder(self.contract, **kw)
    @property
    def position(self):
        return self.broker.positions.current[self.contract]
    @property
    def price(self):
        return self.lastOrClose(self.contract)
    @property
    def contractValue(self):
        return self.price * self.position
    @property
    def netWorth(self):
        currencyAmount = self.broker.positions.current[self.baseCurrency]
        currencyValue = currencyAmount
        return currencyValue + self.contractValue
    @property
    def positionAsFraction(self):
        return self.contractValue / self.netWorth
    def fractionToContractUnits(self, fraction):
        return fraction*self.netWorth / self.price
  1. todo def orderSum(self): pass
    @property
    def openOrders(self):
        return self.broker.orders.openOrders[self.contract]
   class ComposedCrazyLimitOrderTester?: def __init__(self, broker, crazyLowPrice, crazyHighPrice): self.broker = broker self.crazyLowPrice = crazyLowPrice self.crazyHighPrice = crazyHighPrice
    def placeOrder(self, **kw):
        if ('orderType' in kw) and (kw['orderType'] not in ['MKT', 'LMT']):
            raise InvalidOrder('SingleContractSingleCurrencyBrokerCrazyLimitOrderTester cant do orders of type ' + kw['orderType'] + (': %s' % kw))
        kw.update({'orderType' : 'LMT'})
        if 'quantity' not in kw:
            raise InvalidOrder('SingleContractSingleCurrencyBrokerCrazyLimitOrderTester requires a quantity to be specified in the order: %s' % kw)
        if kw['quantity'] >= 0:
            kw.update({'limit' : self.crazyLowPrice})
        else:
            kw.update({'limit' : self.crazyHighPrice})
            
        self.broker.placeOrder(**kw)
    def __getattr__(self, attr):
        return getattr(self.broker, attr)

in the above example, there were a bunch of things like contractValue and netWorth which are just read-only functional computations on object fields. Imo these should be expressed, not as methods, but rather as functions on inputs, plus a way to implicitly take the relevant inputs from the object fields. E.g. instead of saying:

    def contractValue(self):
        return self.price * self.position

you should first create a free-standing function

def contractValue(price, position): return price * position

and then map it onto the object somehow:

    def contractValue(self): contractValue(self.price, self.position)
    but there should be some syntactic sugar to make this more concise. Mb just:

def contractValue(price=self.price, position=self.position): return price * position

(note no leading 'self.', b/c the object instance is unused other than the specified fields)

(in Python i think mb self.price would only be evaluated the first time, but we want it to be evaluated every time)

---

https://github.com/emberjs/rfcs/pull/15

summary of interesting changes (mostly quoted):

You would expect to be able type something like:

<a href="[[image:url?]]">Click here</a>

But instead, in today's Ember, you have to learn about and use the bind-attr helper:

<a {{bind-attr href=url}}>Click here</a>

...

<my-video src=[[image:movie.url?]]></my-video>

<!-- equivalent to -->

{{my-video src=movie.url}}

...

toread (read the rest)

---

look at all this verbiage (from atr), shouldn't this sort of thing be shorter?

...

class SubscribeUniverseMixin?(): #def __init__(self, universe, autoSubscribeBarType='TRADES', autoSubscribeBarDurations=[timedelta(0)], shouldCancelAllUponStartup=True, shouldGetConIds=False): def __init__(self, universe, autoSubscribeBarType='TRADES', autoSubscribeBarDurations=[timedelta(0)], shouldGetConIds=False): self.universe = universe self.autoSubscribeBarType = autoSubscribeBarType self.autoSubscribeBarDurations = autoSubscribeBarDurations self._shouldGetConIds = shouldGetConIds

    def bindBroker(self, myBroker):
        myBroker.log.debug('SubscribeUniverseMixin: subscribing...')
        for duration in self.autoSubscribeBarDurations:
            self.b.subscribePrices(self.universe, bar_duration=duration, bar_type=self.autoSubscribeBarType)
        [myBroker.waitForNewSubscriptionLatency(name) for name in self.universe]
        myBroker.log.debug('SubscribeUniverseMixin: ...done subscribing')
    def idle(self):
        #for duration in self.autoSubscribeBarDurations:
            #self.b.subscribePrices(self.universe, bar_duration=duration, bar_type=self.autoSubscribeBarType)
        #self.b.subscribePrices(self.universe)
        if self._shouldGetConIds:
            for name in self.universe:
                self.b.getConid(name)
        

class LivenessConsumerMixin?(): def bindLivenessProvider(self, livenessProvider): self.livenessProvider = livenessProvider

    def isAlive(self):
        if not hasattr(self, 'livenessProvider'):
            return True
        return livenessProviderIsAlive(self.livenessProvider)

class LivenessProviderMixin?(LivenessConsumerMixin?): def __init__(self): self.dead = Event() self.bindLivenessProvider(self)

class AStrategy1(BindBrokerMixin?, DailyShutdownIntervalMixin?, LivenessProviderMixin?, SubscribeUniverseMixin?, ShutdownMixin?, PeriodicCallbackMixin?, IdleInitMixin?, CurrentUniverseMixin?): def __init__(self, universe, shouldCancelAllUponShutdown=True, daily_shutdown_intervals = None, shouldGetConIds = False, currentUniverseChangedCallbacks = None, periodic_callbacks = None, periodic_callbacks_periods = None, daily_interval_callback_intervals=None, daily_interval_callbacks=None, autoSubscribeBarType='TRADES', autoSubscribeBarDurations=[timedelta(0)], shouldCancelAllUponStartup=True, kwds): BindBrokerMixin?.__init__(self, shouldCancelAllUponStartup = shouldCancelAllUponStartup) LivenessProviderMixin?.__init__(self) SubscribeUniverseMixin?.__init__(self, universe=universe, autoSubscribeBarType=autoSubscribeBarType, autoSubscribeBarDurations=autoSubscribeBarDurations) ShutdownMixin?.__init__(self, shouldCancelAllUponShutdown = shouldCancelAllUponShutdown) PeriodicCallbackMixin?.__init__(self, periodic_callbacks = periodic_callbacks , periodic_callbacks_periods = periodic_callbacks_periods) DailyIntervalCallbackMixin?.__init__(self, daily_interval_callback_intervals=daily_interval_callback_intervals, daily_interval_callbacks=daily_interval_callbacks) DailyShutdownIntervalMixin?.__init__(self, daily_shutdown_intervals=daily_shutdown_intervals) CurrentUniverseMixin?.__init__(self, currentUniverseChangedCallbacks = currentUniverseChangedCallbacks)

    def bindBroker(self, broker):
        BindBrokerMixin.bindBroker(self, broker)
        SubscribeUniverseMixin.bindBroker(self, broker)
        self.post_subscribe_pre_idle_init_bindBroker_hook()
        IdleInitMixin.bindBroker(self, broker)
    def post_subscribe_pre_idle_init_bindBroker_hook(self):
        pass
    def idle_main(self):
        pass
    def idle(self):
        if not self.isAlive():
            return
        
        IdleInitMixin.idle(self)
        PeriodicCallbackMixin.idle(self)
    def idle_init_and_main(self):
        # todo: try resubscribing to pick up dropped names
        # next line is untested
        SubscribeUniverseMixin.idle(self)
        DailyIntervalCallbackMixin.idle(self)
        CurrentUniverseMixin.idle(self)

---

how to make the above shorter:

1) a lot of the verbiage is copying init args into object instance fields, eg def __init__(self, universe, autoSubscribeBarType='TRADES', autoSubscribeBarDurations=[timedelta(0)], shouldGetConIds=False): self.universe = universe

2) a lot of the verbiage is passing init args to various mixin inits, eg

    def __init__(self, universe, shouldCancelAllUponShutdown=True, daily_shutdown_intervals = None, shouldGetConIds = False, currentUniverseChangedCallbacks = None, periodic_callbacks = None, periodic_callbacks_periods = None, daily_interval_callback_intervals=None, daily_interval_callbacks=None, autoSubscribeBarType='TRADES', autoSubscribeBarDurations=[timedelta(0)], shouldCancelAllUponStartup=True, **kwds):
        BindBrokerMixin.__init__(self, shouldCancelAllUponStartup = shouldCancelAllUponStartup)
        LivenessProviderMixin.__init__(self)
        SubscribeUniverseMixin.__init__(self, universe=universe, autoSubscribeBarType=autoSubscribeBarType, autoSubscribeBarDurations=autoSubscribeBarDurations)

3) a lot of the verbiage is calling mixin routines (hook callbacks) of the same name for various methods, eg

    def bindBroker(self, broker):
        BindBrokerMixin.bindBroker(self, broker)
        SubscribeUniverseMixin.bindBroker(self, broker)
        self.post_subscribe_pre_idle_init_bindBroker_hook()
        IdleInitMixin.bindBroker(self, broker)

4) a lot of the verbiage is the bindX methods, eg

    def bindLivenessProvider(self, livenessProvider):
        self.livenessProvider = livenessProvider 

---

it's annoying how using the Python interface framework that atr uses, that superclasses can't/shouldn't declare 'implements', only subclasses.

Also, the global broadcast/subscribe thing there is annoying, it should be local variables all the way

---

Python's slice notation has the small flaw that

X[:] is not identical to X

---

having local mutable variables allows you to do things like:

    dataRows = getData()
    if shouldFastForward:
        dataRows = fastForwardToNow(dataRows, now, price, 0)
  instead of
    dataRows = getData()
    if shouldFastForward:
        dataRows2 = fastForwardToNow(dataRows, now, price, 0)
    else:
        dataRows2 = dataRows

---

in Python, using the command line to dynamically explore structure is key. Eg you have some complicated algorithm encapsulated in an object x, you sort of remember that it has a replacable subalgorithm but you forgot the name, so in the REPL just do x.<tab>, ah there it is, 'x.subalgorithm', now you can look at its properties too, and you can test out its parts, let's call x.subalgorithm.behaviorA(arg1) and see what happens

---

kinda confusing that

In [36]: a = array([[1,2,3], [4,5,6]])

In [37]: a Out[37]: array([[1, 2, 3], [4, 5, 6]])

In [38]: b = a

In [39]: c = a[:,:2]

In [40]: concatenate([b,c], axis=1)


ValueError? Traceback (most recent call last) /home/bshanks/<ipython-input-40-a97486711826> in <module>()


> 1 concatenate([b,c], axis=1)

ValueError?: all the input arrays must have same number of dimensions

it's because b = a is different from c = a[:,:2]; in the former, a single integer is in the second element of the index, whereas in the latter, a whole list is in the second element of the index:

In [26]: a = array([[1,2,3], [4,5,6]])

In [27]: a Out[27]: array([[1, 2, 3], [4, 5, 6]])

In [28]: b = a

In [29]: b Out[29]: array([1, 4])

In [30]: c = a[:,:2]

In [31]: c Out[31]: array([[1, 2], [4, 5]])

In [32]: concatenate([b,c], axis=1)


ValueError? Traceback (most recent call last) /home/bshanks/<ipython-input-32-a97486711826> in <module>()


> 1 concatenate([b,c], axis=1)

ValueError?: all the input arrays must have same number of dimensions

In [33]: b = a[:,[0]]

In [34]: b Out[34]: array([[1], [4]])

In [35]: concatenate([b,c], axis=1) Out[35]: array([[1, 1, 2], [4, 4, 5]])

---

want it to be convenient to call a non-side-effectful function with zero arguments on an object as if it were an attribute:

obj.f instead of obj.f()

(in Python, this is accomplished via the @property decorator)

---

in Python, it sucks that when you spawn a different Process which dies with an error, that you don't see the exception trace (you can if you wrap the process body with a try..except and then manually print out the trace in the catch.. this should be the default imo)

---

a thing i run into a lot in Python:

so you have some object attribute, obj.x

obj.x holds a list.

you modify the list or an item in the list: obj.x[3] = 5

but now you want to put a hook on reads and writes to x. To hook reads, you try making x a @property and hook the getter; to hook writes, you try making x a property and hooking the setter. But when someone reads and writes from the list or from elements of the list, your hooks are not engaged.

So, for example, say that the list is actually maintained on a server somewhere, and reading obj.x fetches the current list from the server, and writing to obj.x posts your modified list to the server. But now someone does: obj.x[3] = 2

this changes the local version of x but not the version on the server.

You could have the convention "whenever you mutate something in x, you have to do 'obj.x = new_x' afterwards". But this adds complexity and may be forgotten.

You could hook the object stored in x, in this case the list, to intercept reads/writes to it. But then you have to insert a hook into the list object. And what if the list elements are themselves objects? You have to recursively insert a hook into these.

There should be language support for such recursive hook insertion. Or, another way to look at it, the language could characterize x[3] as part of @obj if it was gotten via @obj.x[3], and apply the hooks applied to @obj to it. x[3] is something like just a view on obj, rather than a value type (indeed, shall we mark it as @x[3]?). This might have something to do with Milewski's idea that sharing data with locks must be done by an ownership hierarchy, and/or with rust's ownership typing.

eg

self.current[name][field] = price

should be equivalent to

x = self.current # self.current getter invoked y = x[name] # self.current[name] getter invoked y[field] = price # self.current[name][field] setter invoked x[name] = y # self.current[name] setter invoked self.current = x # self.current setter invoked

---

how does Python implement

class C(object): def __init__(self): self._x = None

    @property
    def x(self):
        """I'm the 'x' property."""
        return self._x
    @x.setter
    def x(self, value):
        self._x = value
    @x.deleter
    def x(self):
        del self._x

? (see https://docs.python.org/2/library/functions.html#property )

---

in Python, often you have cause to do something like:

def rest_of_program(..): ... actual_function() ...

def actual_function(): stuff ... wrapper_function(main_body_of_actual_function) ... stuff

def main_body_of_actual_function(): ...

def wrapper_function(): ...

this is slightly harder to read and more annoying to write than if you could just apply 'wrapper function' inline, eg:

def rest_of_program(..): ... actual_function() ...

def actual_function(): stuff ... WRAP_WITH_BEGIN(wrapper_function) main_body_of_actual_function WRAP_WITH_END(wrapper_function) ... stuff

def wrapper_function(): ...

macros could do this, but this may be one of those special cases that can go lower on our 'metaprogramming hierarchy' than macros


it's good that in Python, you can do:

"

self.errorHandler = nullErrorHandler()

def nullErrorHandler(): raise

try: ... except: self.errorHandler = nullErrorHandler "

as a default, and then override f with something that uses sys.exc_info() to get the error (rather than only being able to bind the error with syntax; or rather than not being able to say 'raise' outside of a try/except; would be even better if you could do self.errorHandler = lambda: raise

---

Python's "global" variables are misleading:

http://stackoverflow.com/questions/1977362/how-to-create-module-wide-variables-in-python

" First, the only global variables Python really has are module-scoped variables. You cannot make a variable that is truly global; all you can do is make a variable in a particular scope. (If you make a variable inside the Python interpreter, and then import other modules, your variable is in the outermost scope and thus global within your Python session.) "

---

" One of the "DJB philosophy" key points is to question the interfaces. You have a task to do; you have existing interfaces. What do you do?

    Most people don't even think about it and use the existing interfaces, even if it amounts to cramming a square peg into a round hole. This is why buffer overflows exist. This is why people use abominations such as gets(), which is still in the Single Unix Specification as of version 4, in freaking June 2011. This is why the System V initialization scheme is still prevalent in Linux distributions, despite being one of the slowest and most unreliable of all initialization schemes. This is why people still use the atrocious "libresolv" DNS client library.
    An alternative way of thinking is to ask yourself: "Is the interface I have adequate for the task at hand?"
        If yes: perfect, use that interface.
        If no: then do not use that interface, duh. Design a better one and use it: so the complexity will be split and the code will be easier to maintain.

Interfaces should be questioned right down to the libc. You cannot build strong software on flakey foundations. And from a system and network programmer's point of view, one thing is clear: most standard libc interfaces suck. There is no buffered asynchronous I/O. There is no timed I/O. There is no heap management helper. Even simple system calls are not guaranteed to succeed!

That is where skalibs comes from. skalibs results from questioning the libc interfaces, and providing replacements or additions where the existing interfaces do not make it easy to write reliable, secure and efficient software. It is inspired by DJB's work. It is not a shrine or anything of the kind. http://www.skarnet.org/software/skalibs/djblegacy.html " --

---

https://news.ycombinator.com/item?id=6087076

	Poll: What do you use for Unix process management/monitoring?
	135 points by gosuri 513 days ago | flag | comments
	I'm setting up production infrastructure for an new project and would love to know what you use for process management
	
	monit 123 points

supervisord 199 points

daemonize 5 points

runit 44 points

perp 3 points

DJB's daemontools 53 points

systemd 88 points

god 26 points

upstart 131 points

sehrope 513 days ago

link

Clickable:

monit: http://mmonit.com/monit

supervisord: http://supervisord.org

daemonize: http://bmc.github.com/daemonize

runit: http://smarden.org/runit/

perp: http://b0llix.net/perp/

DJB's daemontools: http://cr.yp.to/daemontools.html

systemd: http://www.freedesktop.org/wiki/Software/systemd

god: http://godrb.com

upstart: http://upstart.ubuntu.com


moonboots 513 days ago

link

I use runit in production for http://typing.io. I appreciate runit's strong unix philosophy (shell scripts instead of dsls). However, I'm starting to experiment with systemd because of features like properly tracking and killing services [1]. This feature would be useful with a task like upgrading an nginx binary without dropping connections [2]. This isn't possible with runit (and most process monitors) because nginx double forks, breaking its supervision tree.

[1] http://0pointer.de/blog/projects/systemd-for-admins-4.html

[2] http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_...

otterley 513 days ago

link

Some points of note:

1. daemontools and runit are practically identical. I do prefer runit somewhat, as svlogd has a few more features than multilog (syslog forwarding, more timestamp options), and sv has a few more options than svc (it can issue more signals to the supervised process).

2. Among the criteria I look for in a process manager are: (1) the ability to issue any signal to a process (not just the limited set of TERM and HUP), and (2) the ability to run some kind of test as a predicate for restarting or reloading a service. The latter is especially useful to help avoid automating yourself into an outage. As far as I'm aware, none of the above process supervisors can do that, so I tend to eschew them in favor of initscripts and prefer server implementations that are reliable enough not to need supervision.


moonboots 513 days ago

link

Systemd provides a hook (ExecStartPre? [1]) for running test commands before starting a service, e.g. checking nginx configuration before starting. However, I also prefer the flexibility of shell scripts over something like systemd's INI service definition format.

[1] http://www.freedesktop.org/software/systemd/man/systemd.serv...

http://stackoverflow.com/questions/12156434/what-is-the-advantage-of-using-supervisord-over-monit

up vote 15 down vote

If you want to additionally monitor resources you should settle for monit. In addition to just checking whether a process is running (availability), monit can also perform some checks of resource usage (performance, capacity usage), load levels and even basic security checks (md5sum of a bianry file, config file, etc). It has a rule-based config which is quite easy to comprehend. Also there is a lot of ready to use configs: http://mmonit.com/wiki/Monit/ConfigurationExamples

Monit requires processes to create PID files, which can be a flaw, because if a process does not create pid file you have to create some wrappers around. See http://mmonit.com/wiki/Monit/FAQ#pidfile

Supervisord on the other hand is more bound to a process, it spawns it by itself. It cannot make any resource based checks as monit. It has a nice CLI servicectl and a web GUI though. share

editflag

edited Oct 24 '12 at 20:40 Piotr Zierhoffer 2,481834

answered Oct 24 '12 at 9:55 Dārayavahuš tdi 1,064729

Creating such a wrapper is not really a problem - if you are running a monitoring software, then usually you have some control over your file system. And it only takes to create a trivial script. +1 for good explanation. – Piotr Zierhoffer Oct 24 '12 at 21:41 3

@xavier disagree, a wrapper script is yet anothe SPOF and not every deamon can be wrapped deterministically, think of some java things for example – Dārayavahuš tdi Oct 25 '12 at 7:28 2

@Dārayavahuštdi, you have a valid point, but with supervisord it’s the other way around: some programs like to daemonize, while supervisord requires everything to stay in the foreground. Writing a wrapper for monit seems much more straight forward, though. supervisord.org/subprocess.html#nondaemonizing-of-subprocesses mmonit.com/wiki/Monit/FAQ#pidfile – Amir Mar 19 '13 at 10:45

@Dārayavahuštdi in the case of docker containers and keeping them up, docker wants something running in the foreground. In these use cases you're better off with supervisord – Jujhar Singh Aug 12 at 7:35


annoyingly hard to (non)portably set the process name of an interpreted process. But we should do it.

http://stackoverflow.com/questions/564695/is-there-a-way-to-change-effective-process-name-in-python

---