![[Home]](http://www.bayleshanks.com/cartoonbayle.png)
rdtsc 12 hours ago [-]
> Implement backpressure throughout your system.
This is a subtle one and gets lost among CAP theorem talk, but it is important.
Just recently had to struggle with adding backpressure. Did some things as just sending a message (a "gen_server:cast" in Erlang). The sender shoved all 300K+ messages in the mailbox of receiver which could not process them in time because of a bottleneck and bug. Receiver's memory usage went to 120GB when it eventually restarted.
The simple solution was just to switch a gen_server:cast with a gen_server:call i.e. make the receiver acknowledge, and the sender wait for acknowledge before sending another message. (As an aside that's the nice benefit of Erlang/Elixir there, it was really a 2 line change. If it was some kind of serialization library + rpc socket layer, it would have been a lot more code to write).
...
sitkack 9 hours ago [-]
This isn't backpressure, this is forcing synchronicity by using a queue depth of 1. For low throughput applications this is fine, but the latency will kill you on the long run and vastly underutilize available resources.
Backpressure should be proportional to the amount of work you want to prevent. And that backpressure should propagate up the event stream to limit the whole capacity of the subgraph. The slowest chain governs the speed of the whole system so that it can retain functionality instead of falling over. Which relates to other properties of the system
reply
rdtsc 9 hours ago [-]
You can think of synchronicity as the most basic form of backpressure. The goal is to avoid flooding the target when source sends at a too high of a rate.
But that was a quick fix. Eventually I implemented something else. Notice that in my case there was a also a bug, so it wasn't just due to normal hardware limits.
...
---
maybe just clone the libuv api, eg: http://nikhilm.github.io/uvbook/filesystem.html
i think it's cross-platform across windows, GNU/Linux, MacOS?, Android
---
maybe mix the libuv api with a plan9 style thingee to unify the different file-like object APIs
---
some stream util functions:
https://github.com/htoooth/mississippi2
---
two utils libraries that i randomly found, dunno if any of them are any good (the first two have a bunch of github stars though):
http://pocoproject.org/ and http://pocoproject.org/docs/00100-GuidedTour.html ( https://github.com/pocoproject/poco )
https://github.com/waruqi/tbox https://github.com/Amaury/Ylib
and of course C++ Boost
---
"C++ template library of lock-free and fine-grained algorithms": http://libcds.sourceforge.net/
---
"the most useful data structures, such as strings, lists, trees, associative arrays, vectors, matrices, and sets, " -- Evolving a language in and for the real world: C++ 1991-2006
---
"Java's Calendar class is the poster child for APIs which are just miserable to use. You have to use all of their weird C like constants for access, months are freaking zero based, but weekdays are one based!" -- http://fantom.org/doc/docIntro/WhyFantom
---
" Some of this is normal cruft setting in, but much of it is a general philosophy in both Java and .NET API design. Both platforms tend toward APIs using a proliferation of small classes that are over abstracted and under powered. Fantom follows a very different philosophy - we believe in a very few, but powerful classes. A good example is the java.io package which contains over 60 classes and interfaces. To do anything useful requires three of four classes, and if you forget to use buffered streams, then performance goes to hell. And even with all of these classes, it is still a lot of work to do basic things like parse a file into lines of text. Fantom collapses most of the java.io functionality into four classes: File, Buf, InStream?, and OutStream?. The IO streams classes are buffered by default, support both binary and text, and have lots of conveniences built right in. "
"
Fantom was designed from the ground up to support functions as first class objects. "
---
Fantom claims to be "quite obsessive about providing all the key featuresrequired for a standard library, but with much less surfacearea than the APIs found in Java or .NET."
" Fantom provides a set of APIs which abstract away the Java and .NET APIs. We actually consider this one of Fantom's primary benefits, because it gives us a chance to develop a suite of system APIs that are elegant and easy to use compared to the Java and .NET counter parts...we are obsessed with making the Fantom APIs beautiful"
"Closures are a key feature of the language, and all the APIs are written to use functions and closures where appropriate.
---
libbsd ?
---
https://en.wikipedia.org/wiki/Newlib ?
---
'standard libraries' in https://en.wikipedia.org/wiki/Template:C_programming_language
---
https://glyph.twistedmatrix.com/2016/08/attrs.html
---
http://www.haskellforall.com/2016/07/list-transformer-beginner-friendly-listt.html
---
"
thinkpad20 34 days ago
| parent [-] | on: A founder's perspective on 4 years with Haskell |
> The heavy and unnecessary usage of symbols, like, >>, 1=, >=>, <=<, and so on just adds cognitive overload and gives no inherent benefit but most of the Haskell library code is littered with it
This is an oft-repeated criticism of Haskell which I frankly find has no basis. Firstly, symbols are preferred for these kinds of functions precisely because of their ubiquity. Once you're familiar with them (which you almost certainly will quickly become, for the common ones like `>>=`, and `< * >`) they make the code easier to read, not harder, in general. Few would complain about having to write "<" to compare two numbers in most languages, rather than, say, "lessThan", because it's all over the place, and having a symbol for it makes it easier for a developer to read. The same is true for many common combinators in Haskell. There is no "cult-like insistence" on them; it's simply the case that many Haskell developers find them more pleasant to write than named functions. Honestly, the comparison to brainfuck is completely unfounded, and the references to cults and religious fetishes are IMO unnecessarily derisive.
Furthermore, I'd add that what makes the symbols seem hard is not the fact that they're symbols, but that the ideas that they're representing are challenging when coming from the world of imperative programming. Using a named function like "bind" rather than ">>=" or "apply" rather than "< * >" ..."
---
---
---
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/asplos2011-drawbridge.pdf page 4
or
https://github.com/oscarlab/graphene/wiki/PAL-Host-ABI
---
Golang's bufio.NewReader?() and bufio.NewReader?().ReadString? and time.Now()
compare:
golang:
reader := bufio.NewReader?(os.Stdin) reader.ReadString?('\n')
C: int c; while((c = getc(stdin)) != EOF) { if( c == '\n' ) { gettimeofday((struct timeval *)timestamp, NULL); break; }
---
Golang's channels:
golang channel := make(chan time.Time)
compare to C's:
pthread_create(&listener, NULL, listen, &entered)
and golang:
... := <- channel
compared to C:
pthread_join(listener, NULL)
note also that pthread_join(listener, NULL) and pthread_join(listener, NULL) return an error status value that must be checked, whereas the Golang channel ops don't (i don't understand how that could be, though; surely it's possible to have 'too many channels'?). The Golang stuff is more concise.
---
" In fact, the synchronization classes in java.util.concurrent (locks, semaphores, countdown-latches and more) do not rely on OS-provided constructs, but are implemented directly in Java using CPU concurrency primitives like CAS combined with the ability to park and unpark fibers. Therefore, all we had to do to adapt these classes to work for fibers as well as threads was to replace all references to Thread with Strand, and every call to LockSupport?.park/unpark with calls to Strand.park/unpark. Quasar contains a few such ports from java.util.concurrent (the package’s source code is in the public domain) that work on strands and maintain the exact same API, so no code changes are required by clients of these classes. "
---
" With Java 8, we’ve been introduced to the new Stream API that allows applying aggregate operations like Filter, Sort or Map on streams of data. Another thing Streams allow are parallel operations on multicore machines when applying .parallelStream() – Splitting the work between threads using the Fork/Join framework introduced in Java 7. An evolution from the Java 6 java.util.concurrency library, where we met the ExecutorService? which creates and handles our worker thread pools.
Fork/Join is also built on top of the ExecuterService?, the main difference from a traditional thread pool is how they distribute the work between threads and thereby multicore machine support. With a simple ExecuterService? you’re in full control of the workload distribution between worker threads, determining the size of each task for the threads to handle. With Fork/Join on the other hand, there’s a work-stealing algorithm in place that abstracts workload handling between threads. "
---
---
http://www.1024cores.net/home/technologies
" The first thing to say is that there are two main categories of libraries and technologies. The first category is targeted at parallel computations (HPC), that is, it will help you to parallelize computations (like matrix multiplication, sorting, etc). OpenMP?, Cilk++, TBB, QuickThread?, Ct, PPL, .NET TPL, Java Fork/Join fall into this category.
The second category is targeted at general concurrency, that is, it will help to implement things like multi-threaded servers, rich client application, games, etc. TBB, QuickThread?, just::thread, PPL, AAL, AppCore? fall into this category. .NET and Java standard libraries also include some things that may be of help (like concurrent containers, synchronization primitives, thread pools, etc).
As you may notice, some libraries are mentioned in both categories. For example, Intel TBB contains some primitives that help with parallel computations (parallel algorithms and task scheduler), and some things that help with general concurrency (threads, synchronization primitives, concurrent containers, atomics).
Efficient and scalable parallel computations is a rather difficult field, it's easy to create a parallel program that executes slower than original single-threaded version (and the more cores you add the slower it executes). So, if you are not doing something extraordinary, you may consider using internally parallelized libraries like Intel MKL, Intel IPP, AMD ACML. They are not only internally parallelized, they are also highly optimized and will be updates for future architectures. So you only need to call a function for matrix multiplication, FFT, video stream decoding, image processing or whatever you need.
Subpages (6): Concurrency Kit (C) FastFlow? (C++) Intel TBB Just::Thread (C++) OpenMP? QuickThread? (C++/Fortran) "
---
http://www.1024cores.net/home/technologies/just-thread
---
in Python, you can pack together a bunch of reused optional arguments into a namedtuple and pass that along instead, but then it's cumbersome to construct one of these while inheriting the defaults, eg:
disassemble_raw_bytecode_file(infile,outfile, dis_opts=dis_defs._replace(allow_unknown_opcodes=args.allow_unknown_opcodes))
it would be better to have a namedtuple with defaults.
---
http://stackoverflow.com/questions/29290359/existence-of-mutable-named-tuple-in-python
recordclass simplenamespace
---
also, commonjs
---
on python testing libs:
" At first I thought I would discover that one was far superior to all the rest. But I don’t think that’s the case.
I do think pytest is by far the coolest and most advanced. However, unittest keeps improving, and is no slouch. And although nose isn’t really in development as far as I can tell, lots of people still use it successfull " -- https://leanpub.com/pythontesting/read
---
js functional programming frameworks and libraries:
"React, Cycle.js, Redux, MobX?, RxJS? or Ramda"
---
https://github.com/dotnet/standard/tree/master/docs/netstandard-20#composition
https://github.com/dotnet/standard/tree/master/netstandard/ref
---
" Some avenues for quality improvement:
Provide stable, extensible test/bench frameworks.
Provide more push-button CI setup, e.g. have cargo new set up Travis/Appveyor.
Restart the API guidelines project.
Use badges on crates.io to signal various quality metrics.
Perform API reviews on important crates.Some avenues for discoverability improvement:
Adding categories to crates.io, making it possible to browse lists like "crates for parsing".
More sophisticated ranking and/or curation.A number of ideas along these lines were discussed in the Rust Platform thread. " -- [1]
---
Haskell98's 16 basic typeclasses:
https://www.haskell.org/onlinereport/basic.html#standard-classes
---
Elm's package approval process:
https://github.com/elm-lang/package.elm-lang.org http://wayback.archive.org/web/20161109235120/https://github.com/elm-lang/package.elm-lang.org
---
fixed-size tuple size limits
http://reasonablypolymorphic.com/blog/elm-is-wrong#fn1 opines that a size limit of 7 (Elm) is too small, but 63 (Haskell) is okay
---
https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/ argues that Python's curio is a better framework than Python's (stdlib) asyncio. I already took notes on this essay elsewhere.
---
mi100hael 10 days ago [-]
After using Akka-HTTP, I never want to write a HTTP service with anything else.
reply
---
mstijak 1 hour ago [-]
I find wrong that most frameworks do not include any component library. Even if there is a vibrant community, you end up with 30 dependencies which do not play well together.
I'm the author of a commercial framework for BIG applications [1]. Once you put everything on the table you can see which features should go into the framework and you end up with things like localization, date/number formatting, form validation, keyboard navigation, layouts, tooltips, routing, modals, etc. Once you have all that, you can build a coherent widget/charting library on top of it.
[1]: http://cx.codaxy.com/v/master/docs/intro/feature-list
reply
---
open, read, write, connect, sendto, recvfrom, execve
-- https://drawings.jvns.ca/syscalls/
---
basic functionality, library functions, and syscalls needed for a shell:
https://brennan.io/2015/01/16/write-a-shell-in-c/ https://news.ycombinator.com/item?id=13112589
also contains a list of some key system calls and shell builtins.
some of its contents:
and https://news.ycombinator.com/item?id=13112721 mentions:
---
" Intel Cilk Plus allows C/C++ operations to be applied to multiple array elements in parallel, and also provides a set of built-in functions that can be used to perform vectorized shifts, rotates, and reductions. Similar functionality exists in Fortran 90; Cilk Plus differs in that it never allocates temporary arrays, so memory usage is easier to predict. Elemental functionsEdit
In Cilk Plus, an elemental function is a regular function which can be invoked either on scalar arguments or on array elements in parallel. They are similar to the kernel functions of OpenCL?. "
---
from Python:
PEP 519: Adding a file system path protocol
File system paths have historically been represented as str or bytes objects. This has led to people who write code which operate on file system paths to assume that such objects are only one of those two types (an int representing a file descriptor does not count as that is not a file path). Unfortunately that assumption prevents alternative object representations of file system paths like pathlib from working with pre-existing code, including Python’s standard library.
---
some Python datetime efforts:
sametmax 16 hours ago [-]
I wish Kenneth would have contributed to an existed project for once.
Arrow and pendulum (my current favorite) have a very decent API. The later one is especially well tested for the numerous corner cases of date handling, which I doubt Kenneth got right on first try.
For request, a full rewrite made sense because urllib sucked so much and we had no good alternatives. But for date time, alternative exists and they are good. Do not split the open source effort, join forces!
reply
robochat 13 hours ago [-]
Right, there's arrow and pendulum and also http://delorean.readthedocs.io/en/latest/index.html plus the Datetime options offered by pandas and the numpy.datetime64 type. It gets to be pretty confusing particularly when you need to start transforming between the different types.
reply
BoppreH?