![[Home]](http://www.bayleshanks.com/cartoonbayle.png)
" 1. Learn Key Syscalls
You should know what the following 12 key system calls do, which you'll see regularly in strace output. Test your knowledge! How many of these do you know? Mouse-over for answers. syscall what it does read write open close fork exec connect accept stat ioctl mmap brk
Click here to reveal all. Each syscall has a man page, so if you are at the command line, it should only take a few seconds to jog your memory.
There are variants of these syscalls, so you may see "execve" for "exec", and "pread" as well as "read". There should be man pages for these, too.
"
---
" Category of Syscall Syscalls Seen in Critical Sections Frequency in Critsecs: MySQL? Firefox Time gettimeofday , clock gettime 3.91% 70.18%
Filesystem read* , write* , open , close , lseek , access , dup , mkdir , ftruncate , fsync , writev , pread* , pwrite* , stat , fstat , fcntl , getdents , getcwd , fdatasync , mmap* , munmap* , mprotect* 53.79% 28.75%
Process Memory brk , mmap* , munmap* , mprotect* 31.03% 0.32%
Process Maintenance waitpid , clone , sched setscheduler , sched get priority max , sched get priority min , rt sigaction , rt sigprocmask , tgkill 8.97% 0.32%
Communication ioctl , socket , pipe , read* , write* , pread* , pwrite* 2.07% 0.40%
System Info sysinfo , uname 0.23% 0.03%
Table 1. Syscalls Seen in Critical Sections Six different categories of syscall were seen in critical sections in MySQL? and Firefox; their dynamic distributions in the critical sections of each workload are shown at right. Communications syscalls, which are most difficult to make transaction-safe, are uncommon. Syscalls marked with asterisks belong to several categories "
---
" vectors. The process of transforming filtered system calls into the histogram representation is depicted in Python algorithm 5.1.
Histogram Raw Normalised recvfrom 25284 0.2968720 write 12396 0.1455476 epoll_pwait 11581 0.1359783 futex 9995 0.1173562 getuid 6209 0.0729029 ioctl 5244 0.0615724 sendto 4512 0.0529776 read 4108 0.0482340 mprotect 1509 0.0177179 pread64 691 0.0081133 mmap 613 0.0071975 newfstatat 418 0.0049079 munmap 392 0.0046026 fstat 366 0.0042973 openat 237 0.0027827
Table 5.2: Histogram; excerpt of 15 first lines. Constructed from a 100 second benign sample of BBC News ( "
---
" Priority value Syscalls
0 socket , bind , getsockname , getsockopt , setsockopt
1 poll , select , getpeername
2 accept , recv , recvfrom , recvmsg , read
3 connect , send , sendto , sendmsg , write , writev , sendfile
4 close , shutdown , listen
Table 1: Simulation priority of common syscalls: the lower the priority value, the higher the priority. "
---
" TABLE IV. OVERVIEW OF ACTIONS PERFORMED BY CUSTOM SHARED LIBRARIES IN NATIVE CODE .
Writing log messages Performing memory management system calls, such as mmap and mprotect Reading files in the application directory Calling JNI functions Performing general multiprocess and multithread related system calls, such as fork , clone , setpriority , and futex Reading common files, such as system libraries, font files, and “/dev/random” Performing other operations on files or file descriptors, such as lseek , dup , and readlink Performing operations to read information about the system, such as uname , getrlimit , and reading special files (e.g., “/proc/cpuinfo” and “/sys/devices/system/cpu/possible”) Performing system calls to read information about the process or the user, such as getuid32 , getppid , and gettid Performing system calls related to signal handling Performing cacheflush or set_tls system calls or per- forming nanosleep system call Reading files under “/proc/self/” or “/proc/ < PID > /”, where PID is the process’ pid Creating directories "
" TABLE V. T OP FIVE MOST COMMON ACTIONS PERFORMED BY APPS IN NATIVE CODE , THROUGH SHARED LIBRARIES (SL) AND CUSTOM BINARIES (CB). F OR THE INTERESTED READER , WE REPORT THE FULL VERSION OF THIS TABLE IN [1]. SL CB Description 3,261 72 ioctl system call 1,929 39 Write file in the app’s directory 1,814 35 Operations on sockets 1,594 5 Create network socket 1,242 144 Terminate process or thread group "
" TABLE IX. G ROUPS OF JNI CALLS USED FROM NATIVE CODE . Apps Description 94,543 Get class or method identifier and class reference 71,470 Get or destroy JavaVM?, and Get JNIEnv 53,219 Manipulation of String objects 49,321 Register native method 45,773 Manipulate object reference 41,892 Thread manipulation 35,231 Call Java method 19,372 Manipulate arrays 18,601 Manipulate exceptions 14,330 Create object instance 6,918 Modify field of an object 2,203 Manipulate direct buffers 47 Memory allocation 37 Enter or exit monitor
TABLE X. T OP 10 GROUPS OF J AVA METHODS FROM THE A NDROID FRAMEWORK CALLED FROM NATIVE CODE . Apps Description 7,423 Get path to the Android package associated with the context of the caller 6,896 Get class name 5,499 Manipulate data structures 4,082 Methods related to cryptography 3,817 Manipulate native types 3,769 Read system information 3,018 Audio related methods 2,070 Read app information 1,192 String manipulation and encoding 575 Input/output related methods 483 Reflection "
" TABLE XII. T OP 10 MOST USED STANDARD LIBRARIES . Apps Name Description 24,942 libjnigraphics.so Manipulate Java bitmap objects 2,646 libOpenSLES.so Audio input and output 2,645 libwilhelm.so Multimedia output and audio input 349 libpixelflinger.so Graphics rendering 347 libGLES android.so Graphics rendering 183 libGLESv1 enc.so Encoder for GLES 1.1 commands 183 gralloc.goldfish.so Memory allocation for graphics 182 libOpenglSystemCommon.so Common functions used by OpenGL? 182 libGLESv2 enc.so Encoder for GLES 2.0 commands 181 lib renderControl enc.so Encoder for rendering control commands
TABLE XIII. T OP 10 MOST USED CUSTOM LIBRARIES . Apps Name Description 19,158 libopenal.so Rendering audio 17,343 libCore.so Used by Adobe AIR 16,450 libmain.so Common name 13,556 libstlport shared.so C++ standard libraries 11,486 libcorona.so Part of the Corona SDK, a devel- opment platform for mobile apps 11,480 libalmixer.so Audio API of the Corona SDK 11,458 libmpg123.so Audio library 11,090 libmono.so Mono library, used to run .NET on Android 10,857 liblua.so Lua interpreter 10,408 libjnlua5.1.so Lua interpreter "
" TABLE XIV. A LLOWED SYSTEM CALLS AUTOMATICALLY GENERATED USING A THRESHOLD OF 99% APPS UNAFFECTED BY THE POLICY . accept(*,*,*) access( < SYS-PATH > , F OK) access( < SYS-PATH > ,R OK) access( < SYS-PATH > , W OK) access( < SYS-PATH > ,X OK) access( < USER-PATH > , F OK) access( < USER-PATH > ,R OK) access( < USER-PATH > , R OK j W OK j X OK) bind BINDER(android.os.IServiceManager?. CHECK SERVICE TRANSACTION) brk cacheflush(*,*,0,*,*) cacheflush(*,*,0,0,*) chdir chmod( < USER-PATH > ,*) clone(child stack=*,flags=CLONE VM j CLONE FS j CLONE FILES j CLONE SIGHAND j CLONE THREAD j CLONE SYSVSEM) connect(*, f sa family=AF UNIX, path=@”jdwp-control” g ,*) connect(*, f sa family=AF INET,*,* g ,*) connect(*, f sa family=AF UNIX, path=@”android:debuggerd” g ,*) connect(*, f sa family=AF UNIX, path= < SYS-PATH > g ,*) dup dup2 epoll create(*) epoll ctl(*,*,*,*) epoll wait execve exit( < NEG INT > ) exit(0) exit group( < POS INT > ) exit group(0) fcntl64( < NON STD FD > ,F DUPFD,*) fcntl64( < NON STD FD > ,F GETFD) fcntl64(*,F GETFL) fcntl64( < NON STD FD > ,F SETFD,*) fcntl64( < NON STD FD > ,F SETFL,*) fcntl64( < NON STD FD > ,F SETLK,*) fdatasync(*) fork fstat64 fsync(*) ftruncate(*,*) futex getcwd getegid32 geteuid32 getgid32 getpeername getpgid(0) getpid getppid getpriority(PRIO PROCESS,*) getrlimit(RLIMIT DATA,*) getrlimit(RLIMIT NOFILE,*) getrlimit(RLIMIT STACK,*) getrusage(RUSAGE CHILDREN,*) getrusage(RUSAGE SELF,*) getsockname getsockopt(*,SOL SOCKET,SO ERROR,*,*) getsockopt(*,SOL SOCKET, SO PEERCRED,*,*) getsockopt(*,SOL SOCKET,SO RCVBUF,*,*) gettid getuid32 ioctl( < ASHMEM-DEV > ,*,*) ioctl(*,FIONBIO,*) ioctl( < LOG-DEV > ,*,*) ioctl(*,SIOCGIFADDR,*) ioctl(*,SIOCGIFBRDADDR,*) ioctl(*,SIOCGIFCONF,*) ioctl(*,SIOCGIFFLAGS,*) ioctl(*,SIOCGIFHWADDR,*) ioctl(*,SIOCGIFINDEX,*) ioctl(*,SIOCGIFNETMASK,*) ioctl( < STD IN/OUT/ERR > , SNDCTL TMR TIMEBASE or TCGETS, *) ioctl(*,SNDCTL TMR TIMEBASE or TCGETS,*) ioctl( < URANDOM-DEV > , SNDCTL TMR TIMEBASE or TCGETS,*) listen lseek(*,*,SEEK CUR) lseek(*,*,SEEK END) lseek(*,*,SEEK SET) lstat64 madvise(*,*,MADV DONTNEED) madvise(*,*,MADV NORMAL) madvise(*,*,MADV RANDOM) mkdir( < SYS-PATH > ,*) mkdir( < USER-PATH > ,*) mmap2 mprotect mremap(*,*,*,MREMAP MAYMOVE) munmap nanosleep open( < SYS-PATH > ,*,*) open( < SYS-PATH > ,*) open( < USER-PATH > ,*,*) open( < USER-PATH > ,*) pipe poll prctl(PR GET NAME,*,0,0,0) prctl(PR SET NAME,*,*,*,*) prctl(PR SET NAME,*,*,*,0) prctl(PR SET NAME,*,0,0,0) ptrace(PTRACE TRACEME,*,0,0) readlink( < USER-PATH > ,*,*) recvfrom recvmsg rename( < USER-PATH > , < USER-PATH > ) rmdir( < USER-PATH > ) rt sigprocmask(SIG BLOCK,*,*,*) rt sigprocmask(SIG SETMASK,*,*,*) rt sigreturn(*) rt sigtimedwait([QUITUSR1], NULL, NULL, 8) sched getparam sched getscheduler sched yield select sendmsg sendto setitimer(ITIMER REAL,*,*) setpriority(PRIO PROCESS,*, < POS INT > ) setpriority(PRIO PROCESS,*,0) setrlimit(RLIMIT NOFILE,*) setsockopt(*,SOL IP,*,*,*) setsockopt(*,SOL SOCKET,*,*,*) set tls(*,*,*,*,*) set tls(*,*,0,*,*) sigaction sigprocmask(SIG BLOCK,*,*) sigprocmask(SIG SETMASK,*,*) sigprocmask(SIG UNBLOCK,*,*) sigreturn sigsuspend([]) socket(PF INET,SOCK DGRAM, IPPROTO ICMP) socket(PF INET,SOCK DGRAM, IPPROTO IP) socket(PF INET,SOCK DGRAM, IPPROTO UDP) socket(PF INET,SOCK STREAM, IPPROTO IP) socket(PF INET,SOCK STREAM, IPPROTO TCP) socket(PF NETLINK,SOCK RAW, 0) socket(PF UNIX, SOCK STREAM, 0) stat64 statfs64( < SYS-PATH > ,*) statfs64( < USER-PATH > ,*) symlink( < USER-PATH > , < USER-PATH > ) tgkill(*,*,SIGTRAP) umask uname unlink( < USER-PATH > ) utimes vfork wait4 "
" TABLE XVII. L IST OF ALLOWED METHODS (J AVA METHODS CALLED FROM NATIVE CODE ) AUTOMATICALLY GENERATED FOR ALLOWING A MINIMUM OF 97%, 98% AND 99% OF APPS THAT REACHED NATIVE CODE . Allowed apps (%) Method 97 java.lang.Integer.doubleValue 97 android.content.ContextWrapper?.getPackageName 97 java.lang.String.getBytes 98 java.lang.Double.doubleValue 98 android.content.ContextWrapper?.getClassLoader 98 android.content.ContextWrapper?.getFilesDir 98 java.io.File.getPath 98 android.content.ContextWrapper?.getExternalFilesDir 98 android.view.WindowManagerImpl?.getDefaultDisplay 98 java.lang.String.toLowerCase 98 android.app.Activity.getWindowManager 98 java.util.ArrayList?.add 98 android.view.Display.getMetrics 98 android.app.Activity.getWindow 98 android.view.View.getWindowVisibleDisplayFrame 98 java.util.Calendar.getInstance 98 android.view.View.getDrawingRect 99 java.util.Calendar.get 99 android.os.Bundle.getByteArray 99 android.content.ContextWrapper?.getPackageManager 99 android.content.res.AssetManager?$AssetInputStream?.read 99 java.lang.Long.doubleValue 99 java.lang.ClassLoader?.loadClass 99 android.app.ApplicationPackageManager?.getPackageInfo 99 android.content.res.AssetManager?$AssetInputStream?.close 99 java.lang.Float.doubleValue 99 java.lang.Class.getClassLoader "
---
dazzawazza 1 day ago [-]
I've coded desktop apps (mostly editors and tools for games) for 25 years across X (Motif toolkit), Amiga, TOS, DOS, MFC, Cocoa, Win32, WxWidgets?, Fox Toolkit and Qt. Qt has by far been the best, most rewarding, most empowering experience. It's a great library.
Now Cocoa would win but I prefer C/C++ to Objective-C, it's close though.
Good work people.
reply
hannofcart 22 hours ago [-]
Glad to see the update. Most people associate Qt with GUIs which is unfortunate.
I see that when people think of Qt, they think of WxWidgets?, Cocoa or MFC as alternatives. No, I submit to you that Qt framework is a more elegant, easier to use alternative to Boost as well.
This is not to say that QtQuick? or QtWidgets? aren't solid. However, the success of these two modules ends up occluding the others which to me are the real gems from the QtFramework?: QtCore? and QtNetwork?.
QtCore? provides a solid event loop, the most easy to use implementation of the observer pattern via its signal-slot mechanism, robust threading utilities and a bunch of other utilities that make writing apps in C++ an absolute breeze.
QtNetwork? for a series of networking utilities that are elegantly simple.
If I were to write a command line app or a database or server, I'd reach for Qt in a jiffy.
Qt is not just for GUIs!
reply
Jasper_ 20 hours ago [-]
I really wish QtNetwork? was better. I recently went and had to replace QtNetwork? with our own event loop because it is extremely unreliable on Windows. The only mention of this is a footnote in the API docs:
http://doc.qt.io/qt-5/qabstractsocket.html#waitForBytesWritten
reply
justinclift 17 hours ago [-]
Yeah. It also turns out that just having a Qt application running on a computer can tank network latency for all applications on the computer:
https://bugreports.qt.io/browse/QTBUG-40332
There is a workaround, but developers have to know about it first. And it's such an unexpected and weird behaviour that 99% of developers don't.
This was our fix (1 line of code), in case that helps anyone:
https://github.com/sqlitebrowser/sqlitebrowser/commit/73946400c32d1f7cfcd4672ab0ab3f563eb84f4e
reply
nly 17 hours ago [-]
Yup, i work on software with a Qt GUI and it took us ages to figure out why our daemon (no Qt dependencies) would experience latency when the GUI was open.
For Windows users we now put this var in to global environment via our installer, and probably break other software in the process.
reply
--- HN discussion of: Comparison of C/Posix standard library implementations for Linux (etalabs.net)
---
geofft 22 hours ago [-]
ncurses-base and ncurses-bin are marked "Essential" in Debian (from which Ubuntu derives, and automatically imports packages), which means that any package is permitted to depend on things in those packages without declaring a dependency - including in pre-installation scripts. This image is defined as being large enough that any package in the Ubuntu archive can be correctly installed.
https://wiki.debian.org/BusterPriorityRequalification
the previous page contains a complete list of packages in 'minbase' (those with either Essential: yes or has Priority: required, and dependencies), as well as a list of those files within that list that can be uninstalled without error. You can also grep for 'this is an essential package' in this page to find the essential packages
i think this page has a list of Debian essential packages:
https://github.com/SailorCrane/my-essential-env-packages
base-files
base-passwd
bash
bsdutils
coreutils
dash
debianutils
diffutils
dpkg
e2fsprogs
findutils
grep
gzip
hostname
init
libc-bin
login
mount
ncurses-base
ncurses-bin
perl-base
sed
tar
util-linux---
also compare Debian/Ubuntu minimal installs to Alpine:
fotcorn 23 hours ago [-]
I did a short comparision between this and alpine by installing nginx inside the image (I used nginx-light inside ubuntu):
CONTAINER ID IMAGE SIZE 493450e7bc12 alpine 1.37MB (virtual 5.52MB) 62b1db90500c ubuntu:bionic 6MB (virtual 87MB)
I deleted the cache files from apt after installing (/var/lib/apt/lists*)
Looks quite nice, but it seems Ubuntu packages are much bigger than Alpine package, e.g. Postgres is 159 MB in Ubuntu Bionic and only 32 MB in Alpine (including dependencies). Do the Ubuntu packages have more feature than the equivalent Alpine packages?
reply
" Alpine Linux is built around musl libc and busybox. This makes it smaller and more resource efficient than traditional GNU/Linux distributions. A container requires no more than 8 MB and a minimal installation to disk requires around 130 MB of storage. "
alpine-standard x86_64 apks directory: 8.1M:
alpine-standard 3.7.0 x86_64/apks/x86_64$ ls -1 alpine-base-3.7.0-r0.apk alpine-baselayout-3.0.5-r2.apk alpine-conf-3.7.0-r0.apk alpine-keys-2.1-r1.apk alpine-mirrors-3.5.6-r0.apk APKINDEX.tar.gz apk-tools-2.8.1-r1.apk bonding-2.6-r3.apk bridge-1.5-r3.apk busybox-1.27.2-r6.apk busybox-initscripts-3.1-r2.apk busybox-suid-1.27.2-r6.apk chrony-3.2-r1.apk dbus-libs-1.10.24-r0.apk e2fsprogs-1.43.7-r0.apk e2fsprogs-libs-1.43.7-r0.apk kbd-bkeymaps-2.0.4-r2.apk libblkid-2.31-r0.apk libcap-2.25-r1.apk libcom_err-1.43.7-r0.apk libc-utils-0.7.1-r0.apk libnl3-3.2.28-r1.apk libressl-2.6.3-r0.apk libressl2.6-libcrypto-2.6.3-r0.apk libressl2.6-libssl-2.6.3-r0.apk libressl2.6-libtls-2.6.3-r0.apk libusb-1.0.21-r0.apk libuuid-2.31-r0.apk musl-1.1.18-r2.apk musl-utils-1.1.18-r2.apk network-extras-1.2-r0.apk openrc-0.24.1-r4.apk openssh-7.5_p1-r6.apk openssh-client-7.5_p1-r6.apk openssh-keygen-7.5_p1-r6.apk openssh-server-7.5_p1-r6.apk openssh-server-common-7.5_p1-r6.apk openssh-sftp-server-7.5_p1-r6.apk pcsc-lite-libs-1.8.22-r0.apk ppp-atm-2.4.7-r5.apk ppp-chat-2.4.7-r5.apk ppp-daemon-2.4.7-r5.apk ppp-l2tp-2.4.7-r5.apk ppp-minconn-2.4.7-r5.apk ppp-passprompt-2.4.7-r5.apk ppp-passwordfd-2.4.7-r5.apk ppp-pppoe-2.4.7-r5.apk ppp-radius-2.4.7-r5.apk ppp-winbind-2.4.7-r5.apk scanelf-1.2.2-r1.apk tzdata-2017c-r0.apk usb-modeswitch-2.5.1-r0.apk vlan-2.0-r0.apk wireless-tools-30_pre9-r0.apk wpa_supplicant-2.6-r8.apk zlib-1.2.11-r1.apk
complete contents of Alpine's "mini root filesystem" release (6.4M):
./srv ./mnt ./media ./media/usb ./media/floppy ./media/cdrom ./bin ./bin/sleep ./bin/getopt ./bin/mount ./bin/netstat ./bin/kbd_mode ./bin/ionice ./bin/mkdir ./bin/base64 ./bin/cp ./bin/ping6 ./bin/sed ./bin/ed ./bin/run-parts ./bin/date ./bin/stty ./bin/fatattr ./bin/tar ./bin/setpriv ./bin/fgrep ./bin/bbconfig ./bin/sh ./bin/chmod ./bin/mktemp ./bin/mpstat ./bin/ipcalc ./bin/mknod ./bin/pwd ./bin/more ./bin/usleep ./bin/ping ./bin/makemime ./bin/fsync ./bin/setserial ./bin/ln ./bin/rev ./bin/busybox ./bin/hostname ./bin/umount ./bin/cat ./bin/zcat ./bin/linux64 ./bin/mv ./bin/false ./bin/chown ./bin/chgrp ./bin/dd ./bin/dumpkmap ./bin/nice ./bin/stat ./bin/touch ./bin/rm ./bin/watch ./bin/lzop ./bin/gzip ./bin/df ./bin/rmdir ./bin/iostat ./bin/kill ./bin/fdflush ./bin/link ./bin/login ./bin/true ./bin/pidof ./bin/grep ./bin/printenv ./bin/linux32 ./bin/mountpoint ./bin/conspy ./bin/egrep ./bin/echo ./bin/sync ./bin/ash ./bin/ls ./bin/dmesg ./bin/uname ./bin/pipe_progress ./bin/dnsdomainname ./bin/su ./bin/reformime ./bin/ps ./bin/gunzip ./root ./sys ./home ./var ./var/lock ./var/lock/subsys ./var/local ./var/spool ./var/spool/cron ./var/spool/cron/crontabs ./var/lib ./var/lib/misc ./var/lib/apk ./var/lib/udhcpd ./var/empty ./var/run ./var/tmp ./var/opt ./var/log ./var/cache ./var/cache/misc ./var/cache/apk ./lib ./lib/ld-musl-x86_64.so.1 ./lib/libc.musl-x86_64.so.1 ./lib/libz.so.1 ./lib/apk ./lib/apk/db ./lib/apk/db/lock ./lib/apk/db/installed ./lib/apk/db/triggers ./lib/apk/db/scripts.tar ./lib/libssl.so.44 ./lib/libz.so.1.2.11 ./lib/mdev ./lib/firmware ./lib/libssl.so.44.0.1 ./lib/libcrypto.so.42.0.0 ./lib/libcrypto.so.42 ./proc ./run ./alpine-minirootfs-3.7.0-x86_64.tar.gz ./sbin ./sbin/init ./sbin/hdparm ./sbin/hwclock ./sbin/ifdown ./sbin/modprobe ./sbin/getty ./sbin/tunctl ./sbin/raidautorun ./sbin/watchdog ./sbin/fsck ./sbin/arp ./sbin/loadkmap ./sbin/route ./sbin/halt ./sbin/mkmntdirs ./sbin/swapon ./sbin/reboot ./sbin/udhcpc ./sbin/iprule ./sbin/iplink ./sbin/blkid ./sbin/fdisk ./sbin/klogd ./sbin/apk ./sbin/ldconfig ./sbin/logread ./sbin/inotifyd ./sbin/ifconfig ./sbin/nologin ./sbin/rmmod ./sbin/poweroff ./sbin/fstrim ./sbin/switch_root ./sbin/mdev ./sbin/adjtimex ./sbin/setconsole ./sbin/modinfo ./sbin/ifenslave ./sbin/acpid ./sbin/lsmod ./sbin/losetup ./sbin/iptunnel ./sbin/syslogd ./sbin/mkdosfs ./sbin/findfs ./sbin/slattach ./sbin/ip ./sbin/swapoff ./sbin/iproute ./sbin/sysctl ./sbin/ipneigh ./sbin/insmod ./sbin/blockdev ./sbin/fbsplash ./sbin/nameif ./sbin/depmod ./sbin/ifup ./sbin/ipaddr ./sbin/vconfig ./sbin/mkswap ./sbin/mkfs.vfat ./tmp ./usr ./usr/bin ./usr/bin/install ./usr/bin/free ./usr/bin/killall ./usr/bin/traceroute ./usr/bin/expand ./usr/bin/bzcat ./usr/bin/pkill ./usr/bin/vlock ./usr/bin/uptime ./usr/bin/mkfifo ./usr/bin/less ./usr/bin/unshare ./usr/bin/beep ./usr/bin/mkpasswd ./usr/bin/hostid ./usr/bin/id ./usr/bin/shred ./usr/bin/comm ./usr/bin/dirname ./usr/bin/crontab ./usr/bin/xargs ./usr/bin/wget ./usr/bin/ttysize ./usr/bin/iconv ./usr/bin/lsof ./usr/bin/cmp ./usr/bin/pmap ./usr/bin/printf ./usr/bin/mesg ./usr/bin/traceroute6 ./usr/bin/cryptpw ./usr/bin/cpio ./usr/bin/dos2unix ./usr/bin/cksum ./usr/bin/hexdump ./usr/bin/pstree ./usr/bin/ldd ./usr/bin/logger ./usr/bin/nc ./usr/bin/tee ./usr/bin/lzopcat ./usr/bin/factor ./usr/bin/head ./usr/bin/setkeycodes ./usr/bin/groups ./usr/bin/patch ./usr/bin/volname ./usr/bin/realpath ./usr/bin/nohup ./usr/bin/udhcpc6 ./usr/bin/xzcat ./usr/bin/time ./usr/bin/[ ./usr/bin/sha1sum ./usr/bin/hd ./usr/bin/split ./usr/bin/od ./usr/bin/unexpand ./usr/bin/renice ./usr/bin/ssl_client ./usr/bin/tr ./usr/bin/resize ./usr/bin/passwd ./usr/bin/unxz ./usr/bin/getconf ./usr/bin/xxd ./usr/bin/test ./usr/bin/awk ./usr/bin/reset ./usr/bin/readlink ./usr/bin/eject ./usr/bin/fold ./usr/bin/microcom ./usr/bin/uniq ./usr/bin/ipcs ./usr/bin/diff ./usr/bin/sha256sum ./usr/bin/seq ./usr/bin/getent ./usr/bin/nl ./usr/bin/unlzop ./usr/bin/whoami ./usr/bin/flock ./usr/bin/nproc ./usr/bin/du ./usr/bin/unzip ./usr/bin/blkdiscard ./usr/bin/uuencode ./usr/bin/sort ./usr/bin/wc ./usr/bin/dumpleases ./usr/bin/lzma ./usr/bin/expr ./usr/bin/cut ./usr/bin/find ./usr/bin/basename ./usr/bin/paste ./usr/bin/truncate ./usr/bin/fallocate ./usr/bin/clear ./usr/bin/unix2dos ./usr/bin/dc ./usr/bin/openvt ./usr/bin/lsusb ./usr/bin/tac ./usr/bin/ipcrm ./usr/bin/lzcat ./usr/bin/unlink ./usr/bin/top ./usr/bin/timeout ./usr/bin/shuf ./usr/bin/scanelf ./usr/bin/tail ./usr/bin/whois ./usr/bin/pscan ./usr/bin/nsenter ./usr/bin/chvt ./usr/bin/showkey ./usr/bin/env ./usr/bin/deallocvt ./usr/bin/sha512sum ./usr/bin/sha3sum ./usr/bin/pwdx ./usr/bin/which ./usr/bin/nslookup ./usr/bin/uudecode ./usr/bin/sum ./usr/bin/setsid ./usr/bin/[[ ./usr/bin/yes ./usr/bin/smemcap ./usr/bin/unlzma ./usr/bin/pgrep ./usr/bin/md5sum ./usr/bin/strings ./usr/bin/bunzip2 ./usr/bin/vi ./usr/bin/bzip2 ./usr/bin/nmeter ./usr/bin/cal ./usr/bin/tty ./usr/bin/fuser ./usr/local ./usr/local/bin ./usr/local/lib ./usr/local/share ./usr/lib ./usr/lib/libssl.so.44 ./usr/lib/libssl.so.44.0.1 ./usr/lib/libcrypto.so.42.0.0 ./usr/lib/libcrypto.so.42 ./usr/share ./usr/share/misc ./usr/share/apk ./usr/share/apk/keys ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub ./usr/share/apk/keys/x86_64 ./usr/share/apk/keys/x86_64/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub ./usr/share/apk/keys/x86_64/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub ./usr/share/apk/keys/aarch64 ./usr/share/apk/keys/aarch64/alpine-devel@lists.alpinelinux.org-58199dcc.rsa.pub ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub ./usr/share/apk/keys/s390x ./usr/share/apk/keys/s390x/alpine-devel@lists.alpinelinux.org-58e4f17d.rsa.pub ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub ./usr/share/apk/keys/armhf ./usr/share/apk/keys/armhf/alpine-devel@lists.alpinelinux.org-524d27bb.rsa.pub ./usr/share/apk/keys/ppc64le ./usr/share/apk/keys/ppc64le/alpine-devel@lists.alpinelinux.org-58cbb476.rsa.pub ./usr/share/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub ./usr/share/apk/keys/x86 ./usr/share/apk/keys/x86/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub ./usr/share/apk/keys/x86/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub ./usr/share/man ./usr/sbin ./usr/sbin/nbd-client ./usr/sbin/addgroup ./usr/sbin/killall5 ./usr/sbin/rdev ./usr/sbin/readahead ./usr/sbin/add-shell ./usr/sbin/delgroup ./usr/sbin/chroot ./usr/sbin/nandwrite ./usr/sbin/remove-shell ./usr/sbin/rfkill ./usr/sbin/lspci ./usr/sbin/deluser ./usr/sbin/arping ./usr/sbin/partprobe ./usr/sbin/readprofile ./usr/sbin/fdformat ./usr/sbin/ether-wake ./usr/sbin/nanddump ./usr/sbin/chpasswd ./usr/sbin/sendmail ./usr/sbin/ntpd ./usr/sbin/fbset ./usr/sbin/adduser ./usr/sbin/brctl ./usr/sbin/setfont ./usr/sbin/crond ./usr/sbin/rdate ./usr/sbin/loadfont ./usr/sbin/powertop ./usr/sbin/setlogcons ./etc ./etc/udhcpd.conf ./etc/periodic ./etc/periodic/daily ./etc/periodic/weekly ./etc/periodic/15min ./etc/periodic/monthly ./etc/periodic/hourly ./etc/fstab ./etc/init.d ./etc/mtab ./etc/alpine-release ./etc/apk ./etc/apk/keys ./etc/apk/keys/alpine-devel@lists.alpinelinux.org-5261cecb.rsa.pub ./etc/apk/keys/alpine-devel@lists.alpinelinux.org-5243ef4b.rsa.pub ./etc/apk/keys/alpine-devel@lists.alpinelinux.org-4a6a0840.rsa.pub ./etc/apk/repositories ./etc/apk/protected_paths.d ./etc/apk/arch ./etc/apk/world ./etc/sysctl.d ./etc/sysctl.d/00-alpine.conf ./etc/hostname ./etc/shadow ./etc/passwd ./etc/inittab ./etc/os-release ./etc/hosts ./etc/profile.d ./etc/profile.d/color_prompt ./etc/crontabs ./etc/crontabs/root ./etc/ssl ./etc/ssl/x509v3.cnf ./etc/ssl/certs ./etc/ssl/openssl.cnf ./etc/ssl/cert.pem ./etc/motd ./etc/conf.d ./etc/TZ ./etc/modprobe.d ./etc/modprobe.d/aliases.conf ./etc/modprobe.d/i386.conf ./etc/modprobe.d/blacklist.conf ./etc/modprobe.d/kms.conf ./etc/services ./etc/protocols ./etc/profile ./etc/shells ./etc/securetty ./etc/logrotate.d ./etc/logrotate.d/acpid ./etc/network ./etc/network/if-pre-down.d ./etc/network/if-post-up.d ./etc/network/if-pre-up.d ./etc/network/if-up.d ./etc/network/if-up.d/dad ./etc/network/if-down.d ./etc/network/if-post-down.d ./etc/opt ./etc/issue ./etc/modules ./etc/sysctl.conf ./etc/modules-load.d ./etc/group ./dev ./dev/pts ./dev/shm
---
an OCaml 'std'lib 'candidate':
https://github.com/janestreet/base
---
another guy's list of 34 Haskell extensions that e always uses:
ApplicativeDo
BangPatterns
ConstraintKinds
DataKinds
DefaultSignatures
DeriveFoldable
DeriveFunctor
DeriveGeneric
DeriveLift
DeriveTraversable
DerivingStrategies
EmptyCase
ExistentialQuantification
FlexibleContexts
FlexibleInstances
FunctionalDependencies
GADTs
GeneralizedNewtypeDeriving
InstanceSigs
KindSignatures
LambdaCase
MultiParamTypeClasses
MultiWayIf
NamedFieldPuns
OverloadedStrings
PatternSynonyms
RankNTypes
ScopedTypeVariables
StandaloneDeriving
TupleSections
TypeApplications
TypeFamilies
TypeFamilyDependencies
TypeOperatorsin the blog post he goes into detail (a little bit) about why he uses each one
e also mentions that he likes:
and sometimes uses the {-# OVERLAPPING #-}, {-# OVERLAPPABLE #-}, and {-# INCOHERENT #-} pragmas to allow Overlapping instances and incoherent instances on a one-off basis.
---
" For other things, like web applications, it’s more likely that a majority of the program logic is going to be side-effectful by its nature—it may involve making HTTP requests to other services, interacting with a database, and writing to logfiles.
Figuring out how to structure these effects in a type-safe, decoupled, composable way can be tricky, especially since Haskell has so many different solutions. I could not bring myself to choose just one, but I did choose two: the so-called “mtl style” and freer monads. "
e gives a code example of 'mtl style' in the post. But then e goes on to say that e prefers freer monads. Then e says:
"
There are a few different implementations of freer monads, but I, sadly, was not satisfied with any of them, so I decided to contribute to the problem by creating yet another one. My implementation is called freer-simple, and it includes a streamlined API with more documentation than any other freer implementation. Writing the above mtl style example using freer-simple is more straightforward:
data FileSystem? r where ReadFile? :: FilePath? -> FileSystem? String WriteFile? :: FilePath? -> String -> FileSystem? ()
readFile :: Member FileSystem? r => FilePath? -> Eff r String readFile a = send $ ReadFile? a
writeFile :: Member FileSystem? r => FilePath? -> String -> Eff r () writeFile a b = send $ WriteFile? a b
runFileSystemIO :: LastMember? IO r => Eff (FileSystem? ': r) ~> Eff r runFileSystemIO = interpretM $ \case ReadFile? a -> Prelude.readFile a WriteFile? a b -> Prelude.writeFile a b
runFileSystemInMemory :: [(FilePath?, String)] -> Eff (FileSystem? ': effs) ~> Eff effs runFileSystemInMemory initVfs = runState initVfs . fsToState where fsToState :: Eff (FileSystem? ': effs) ~> Eff (State [(FilePath?, String)] ': effs) fsToState = reinterpret $ case ReadFile? path -> get >>= \vfs -> case lookup path vfs of Just contents -> pure contents Nothing -> error ("readFile: no such file " ++ path) WriteFile? path contents -> modify $ \vfs -> (path, contents) : delete (path, contents) vfs
(It could be simplified further with a little bit of Template Haskell to generate the readFile and writeFile function definitions, but I haven’t gotten around to writing that.) "
---
" ... to lens or not to lens ...
There’s no getting around it: lens is a behemoth of a library. For a long time, I wrote Haskell without it, and honestly, it worked out alright. I just wasn’t doing a whole lot of work that involved complicated, deeply-nested data structures, and I didn’t feel the need to bring in a library with such a reputation for having impenetrable operators and an almost equally impenetrable learning curve.
But, after some time, I decided I wanted to take the plunge. So I braced myself for the worst, pulled out my notebook, and started writing some code. To my surprise… it wasn’t that hard. It made sense. Sure, I still don’t know how it works on the inside, and I never did learn the majority of the exports in Control.Lens.Operators, but I had no need to. Lenses were useful in the way I had expected them to be, and so were prisms. One thing led to another, and before long, I understood the relationship between the various optics, the most notable additions to my toolkit being folds and traversals. Sure, the type errors were completely opaque much of the time, but I was able to piece things together with ample type annotations and time spent staring at ill-typed expressions. Before long, I had developed an intuition for lens.
After using it for a while, I retrospected on whether or not I liked it, and honestly, I still can’t decide. Some lensy expressions were straightforward to read and were a pleasant simplification, like this one:
paramSpecs ^.. folded._Required
Others were less obviously improvements, such as this beauty:
M.fromList $ paramSpecs ^.. folded._Optional.filtered (has $ _2._UsePreviousValue?)
But operator soup aside, there was something deeper about lens that bothered me, and I just wasn’t sure what.
...
a 2014 blog post entitled Lens is unidiomatic Haskell, which includes a point that I think is spot-on:
Usually, types in Haskell are rigid. This leads to a distinctive style of composing programs: look at the types and see what fits where. This is impossible with lens, which takes overloading to the level mainstream Haskell probably hasn’t seen before.
...
The way lens structures its types deliberately introduces a sort of subtyping relationship—for example, all lenses are traversals and all traversals are folds, but not vice versa—and indeed, knowing this subtyping relationship is essential to working with the library and understanding how to use it. It is helpfully documented with a large diagram on the lens package overview page, and that diagram was most definitely an invaluable resource for me when I was learning how to use the library.
On the surface, this isn’t unreasonable. Subtyping is an enormously useful concept! The only reason Haskell dispenses with it entirely is because it makes type inference notoriously difficult. The subtyping relation between optics is one of the things that makes them so useful, since it allows you to easily compose a lens with a prism and get a traversal out. Unfortunately, the downside of all this is that Haskell does not truly have subtyping, so all of lens’s “types” really must be type aliases for types of roughly the same shape, namely functions. This makes type errors completely baffling, since the errors do not mention the aliases, only the fully-expanded types (which are often rather complicated, and their meaning is not especially clear without knowing how lens works under the hood).
...
Still, while it is not the first library I reach for on every new Haskell project, manipulating nested data without lens is just too unpleasant after tasting the nectar, so I can’t advise against it in good faith.
Sadly, this means I’m a bit wishy-washy when it comes to using lens, but I do have at least one recommendation: if you decide to use lens, it’s better to go all-in. Don’t generate lenses for just a handful of datatypes, do it for all of them. You can definitely stick to a subset of the lens library’s features, but don’t apply it in some functions but not others. Having too many different, equally valid ways of doing things leads to confusion and inconsistency, and inconsistency minimizes code reuse and leads to duplication and spaghetti. Commit to using lens, or don’t use it at all. "
---
" Mitigating the string problem
Finally, Haskell has a problem with strings. Namely, String is a type alias for [Char], a lazy, singly linked list of characters, which is an awful representation of text. Fortunately, the answer to this problem is simple: ban String in your programs.
Use Text everywhere. I don’t really care if you pick strict Text or lazy Text, but pick one and stick to it. Don’t ever use String, and especially don’t ever, ever, ever use ByteString? to represent text! There are enormously few legitimate cases for using ByteString? in a program that is not explicitly about reading or writing raw data, and even at that level, ByteString? should only be used at program boundaries. In that sense, I treat ByteString? much the same way I treat IO: push it to the boundaries of your program.
One of Haskell’s core tenets is making illegal states unrepresentable
...
But wait, it gets worse! Data.Text.Encoding exports a function called decodeUtf8, which has type ByteString? -> Text. What an incredible function with a captivating type! Whatever could it possibly do? Again, this function’s type is basically Any -> Text, which is remarkable in the power it gives us. Let’s try it out, shall we?
ghci> decodeUtf8 "\xc3\x28" "* Exception: Cannot decode byte '\x28': Data.Text.Internal.Encoding.decodeUtf8: Invalid UTF-8 stream
Oh. Well, that’s a disappointment.
Haskell’s string problem goes deeper than String versus Text; it seems to have wound its way around the collective consciousness of the Haskell community and made it temporarily forget that it cares about types and totality. This isn’t that hard, I swear! I can only express complete befuddlement at how many of these APIs are just completely worthless.
Fortunately, there is a way out, and that way out is text-conversions. It is the first Haskell library I ever wrote. It provides type safe, total conversions between Text and various other types, and it is encoding aware. It provides appropriately-typed base–16 and base–64 conversion functions, and is guaranteed to never raise any exceptions. Use it, and apply the Haskell philosophy to your strings, just as you already do for everything else in your program.
"
---
[6]:
" Unix command-line interface programs and shell builtins File system
cat chmod chown chgrp cksum cmp cp dd du df file fuser ln ls mkdir mv pax pwd rm rmdir split tee touch type umask
Processes
at bg crontab fg kill nice ps time
User environment
env exit logname mesg talk tput uname who write
Text processing
awk basename comm csplit cut diff dirname ed ex fold head iconv join m4 more nl paste printf sed sort strings tail tr uniq vi wc xargs
Shell builtins
alias cd echo test unset wait
Searching
find grep
Documentation
man
Software development
ar ctags lex make nm strip yacc
Miscellaneous
bc cal expr lp od sleep true and false
"
---
'data science in Go' cheat sheet:
https://www.cheatography.com/chewxy/cheat-sheets/data-science-in-go-a/
---
R tidyverse: " the ggplot2, tibble, tidyr, readr, purrr, and dplyr packages. These are considered to be the core of the tidyverse because you’ll use them in almost every analysis. " [7]
---
the random fn should return a real number in the range [.0, 1.0); note that the set is half-open on the right, that assists in using this to choose a random integer in a certain range) (b/c you want to just multiply by the length of the range and then take floor(), but if 1 was included then you'd have to plan for the improbable event that you got exactly 1 back, which would then be out of your range)
---
note: i recommend ignoring this, see below.
https://github.com/abseil/abseil-cpp https://abseil.io/ https://opensource.googleblog.com/2017/09/introducing-abseil-new-common-libraries.html https://abseil.io/docs/cpp/guides/synchronization
" base Abseil Fundamentals The base library contains initialization code and other code which all other Abseil code depends on. Code within base may not depend on any other code (other than the C++ standard library). algorithm The algorithm library contains additions to the C++ <algorithm> library and container-based versions of such algorithms. container The container library contains additional STL-style containers. debugging The debugging library contains code useful for enabling leak checks. Future updates will add stacktrace and symbolization utilities. memory The memory library contains C++11-compatible versions of std::make_unique() and related memory management facilities. meta The meta library contains C++11-compatible versions of type checks available within C++14 and C++17 versions of the C++ <type_traits> library. numeric The numeric library contains C++11-compatible 128-bit integers. strings The strings library contains a variety of strings routines and utilities, including a C++11-compatible version of the C++17 std::string_view type. synchronization The synchronization library contains concurrency primitives (Abseil's absl::Mutex class, an alternative to std::mutex) and a variety of synchronization abstractions. time The time library contains abstractions for computing with absolute points in time, durations of time, and formatting and parsing time within time zones. types The types library contains non-container utility types, like a C++11-compatible version of the C++17 std::optional type. "
i skimmed through this quickly and it was mostly either stuff that was too specific to C++ to be understood at a glance, or rather advanced optimizationy stuff that we probably don't need (eg 128-bit uints, eg small arrays that are allocated inline). It claims to be well-documented inline but i found it a little hard to read, because in at least one of the directories namely 'base' there is no 'base.h' so it's unclear which of the many files in there should be read -- but otoh i get the impression that 'base' may be meant to be only internal anyhow?
They say there is a Python version coming out though, presumably that would be easier to quickly survey. Also, i was intrigued by the annotations for thread-safety, etc, found in the 'base' folder.
all in all, i recommend ignoring this; for our purposes (finding a very small set of super-essential library functions for the stdlib of a new language, and finding the best way to do them), the ratio of time-to-grok to usefulness is probably low.
---
in Python, it would be better if argparse allowed boolean arguments via just 'type=bool' instead of via 'nargs='?', const=True, default=False, type=bool', eg it should be:
cmdLineParser.add_argument("--simulate", action="store", type=bool, dest="simulate", help="don't actually do anything")
---
