Wednesday, November 13, 2019

Measures of developer productivity

The most used metrics are

  1. lines of code (LOC) per unit time (Delorey, Knutson, & Chun, 2007; Maxwell, Van Wassenhove, & Dutta, Oct 1996) 
  2. Function points per unit time (Delorey et al., 2007; Maxwell & Forselius, 2000).
  3. Number of diffs landed per unit time (Facebook, Uber)

Tuesday, November 5, 2019

Monday, October 14, 2019

Golang tools

Dependency management tools such as Godep, Sltr, etc.
Go’s templating language
Go’s code generation tools, such as Stringer
Popular Go web frameworks, such as Revel
Router packages, such as Gorilla Mux
Godoc comments

Tuesday, October 8, 2019

Software Engineering Blog posts

Crypto Notes

https://wanguolin.github.io/assets/cryptography_and_network_security.pdf
GF(2^n) and it's use in AES: https://engineering.purdue.edu/kak/compsec/NewLectures/Lecture7.pdf (and uploading the pdf to my google drive, in case it goes missing in the future from the Purdue link)

Kubernetes Notes


  1. Getting Started: https://kubernetes.io/
  2. Cloud Native Computing Foundation:  https://www.cncf.io/
  3. Course to learn Kubernetes: https://courses.edx.org/courses/course-v1:LinuxFoundationX+LFS158x+2T2019/course/. As a first step install the software called Minikube ( see Instructions here: https://kubernetes.io/docs/setup/learning-environment/minikube/), as a part of which you will need to install kubectl 
Install go 1.12 (google for it), then clone kubernetes source code from git. We will start navigating the code base using the kubectl command, use the following to build kubectl e.g. on my mac
$ make WHAT='cmd/kubectl'
+++ [1014 17:55:59] Building go targets for darwin/amd64:
    cmd/kubectl 
which shows up in the output directory as :
./_output/local/go/bin/kubectl
./_output/local/bin/darwin/amd64/kubectl 
To look at the source code ; go to the directory cmd; you will see the following files that have an entrypoint main defined, which points out all the different things that the kubernetes product is built out of.

  • clicheck/check_cli_conventions.go:func main() {
  • cloud-controller-manager/controller-manager.go:func main() {
  • gendocs/gen_kubectl_docs.go:func main() {
  • genkubedocs/gen_kube_docs.go:func main() {
  • genman/gen_kube_man.go:func main() {
  • genswaggertypedocs/swagger_type_docs.go:func main() {
  • genyaml/gen_kubectl_yaml.go:func main() {
  • hyperkube/main.go:func main() {
  • importverifier/importverifier.go:func main() {
  • kube-apiserver/apiserver.go:func main() {
  • kube-controller-manager/controller-manager.go:func main() {
  • kube-proxy/proxy.go:func main() {
  • kube-scheduler/scheduler.go:func main() {
  • kubeadm/kubeadm.go:func main() {
  • kubectl/kubectl.go:func main() {
  • kubelet/kubelet.go:func main() {
  • kubemark/hollow-node.go:func main() {
  • linkcheck/links.go:func main() {
  • preferredimports/preferredimports.go:func main() {
  • verifydependencies/verifydependencies.go:func main() {

     

Sunday, September 22, 2019

Tcp timeouts

https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/amp/

Tuesday, September 17, 2019

Thread local storage

This is a good pictorial representation of how linux stores thread local storage, and how it can be accessed using the key used in pthread_setspecific (each thread uses the key to find out its local thread local storage block) e.g. http://weng-blog.com/2016/07/Linux-tls/

how to become a good programmer

Embedded C++

RTTI, dynamic memory allocation and exceptions are among the most hotly debated subjects in embedded circles. (ref: http://forums.codeguru.com/showthread.php?539611-I-have-7-days-to-prepare-for-an-Embedded-C-Interview-Any-tips-and-good-links)

RTTI is used by dynamic_cast to figure out if a base ptr can be reinterpreted as a derived class ptr . look at https://blog.feabhas.com/2013/09/casting-what-could-possibly-go-wrong/

Also look at this https://cs.nyu.edu/courses/fall16/CSCI-UA.0470-001/slides/MemoryLayoutMultipleInheritance.pdf for description about virtual inheritance and memory layouts therein.

Saturday, September 14, 2019

ipfs

Aim is to replace HTTP
https://ipfs.io/

Further, when reading about IPFs i came across Merkle trees, which are incidentally also used in Git to reduce the time for finding out what has changed between 2 branches.

This further took me to this paper : https://people.csail.mit.edu/silvio/Selected%20Scientific%20Papers/Zero%20Knowledge/Zero-Knowledge_Sets.pdf

Sunday, September 8, 2019

Thrift

Containers from scratch

Things to explore further:https://ericchiang.github.io/post/containers-from-scratch/
(1) chroot, and it does not have private namespaces
(1) creating "namespaces" with unshare
(2) entering namspace with nsenter
(3) network namesapces can be shared e.g. across containers.
(4) cgroup directories can be created in /sys/fs/cgroups, and then appropriate values configured. cgroups is a way for the kernel to have "controlled isolation"

Saturday, September 7, 2019

Why BGP is a better iGP

https://archive.nanog.org/meetings/nanog55/presentations/Monday/Lapukhov.pdf and the RFC is https://tools.ietf.org/html/draft-ietf-rtgwg-bgp-routing-large-dc-01

Note that normally, when BGP RIBs are exchanged by two routers, if both sit in the same AS, then while their RIB will show the path, it will be marked with an "i", and that path will not be advertised to the outside world. If we instead give an AS to each rack and then use BGP it can be still made to work because each ASN is considered a private ASN. 

Distributed Systems: Compute Infra.

Tuesday, September 3, 2019

Compute Platform comparison

I liked this comparison of platforms of (a) Virutalized compute resources vs. CaaS (taken from https://thenewstack.io/container-orchestration-scheduling-herding-computational-cattle/). In general modern cloud can be divided into
 Infrastructure as a Service (IaaS), Containers as a Service (CaaS), and Platform as a Service (PaaS).



Sunday, August 25, 2019

Messaging patterns in ZeroMQ

As outlined here http://www.aosabook.org/en/zeromq.html, one can look upon all messaging to fall in these categories
  1. Publish/Subscribe
  2. Synchronous Request/Reply
  3. Asynchronous Request/Reply
  4. Push/Pull
  5. Parallelised pipeline 
Refer to https://blog.scottlogic.com/2015/03/20/ZeroMQ-Quick-Intro.html for how this is used . The idea is to have "smart endpoint, dumb network", unlike Kafka which is a "dumb endpoint, smart network" model. The clients essentially connect to the server, and the server communicates to the clients based on the above messaging modes. Note that the clients and server can both be on the same machine (IPC) or even different threads in a process talking via an "endpoint" etc. 

Sunday, August 18, 2019

Twitter Infra

https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale.html

Periscope infra:

(from https://qr.ae/TWrUa5 ) and a video https://www.youtube.com/watch?v=xjC3ZKYG74g
  1. Wowza Media Systems for streaming
  2. PubNub for the chatroom
  3. Circle CI and Travis CI
  4. Fabric
  5. Iron.io
  6. Algoria for search and indexing
  7. Slack

Tuesday, August 13, 2019

Scaling globally

There are 3 types of scalability issues that need to be addressed to scale to a global scale . Those are
  1. Network scalability & service discovery. 
  2. Compute scalability & virtualization
  3. Storage scalability
One will see that organizations that offer cloud as a service use all three of these scalabilities.

Network Scalability :
(A) Load Balancers
Refer to this blog which points out how modern L4/L7 load balancers work. 
https://blog.envoyproxy.io/introduction-to-modern-network-load-balancing-and-proxying-a57f6ff80236 . I have also seen L3 load balancers used via DNS (e.g. UltraDNS sitebacker pools).
In summary I have seen load balancers of the following types:
  • Proxy based load balancers
  1. L3 load balancing: DNS based load balancing via pools (round-robin) or via mapping changes (Akamai), or via Anycast (See this for how BGP makes this happen: https://www.imperva.com/blog/how-anycast-works/
  2. L4 load balancing via HAProxy (SSL termination via NGINX)
  3. L7 load balancing via HAProxy and a sidecar like Muttley (Uber) , which is essentially based on Healthchecks, Traffic controller rules, and Zookeeper nodes that are maintained at a /zone/service/ level , and updated when a particular service is deployed to a machine.
  • Client side load balancers:
  1. GRPC based load balancing is an example of client-based load balancing.  Refer to https://github.com/grpc/grpc/blob/master/doc/load-balancing.md. (I believe this could be done using something like a muttley sidecar too) 
(B) Service discovery :
When a service is deployed on a machine, it needs to be discoverable. This can be done in the following ways
  1. DNS based service discovery such as Mesos-DNS 
  2. DNS based service discovery using SRV records (See this https://docs.citrix.com/en-us/citrix-adc/13/dns/service-discovery-using-dns-srv-records.html
  3. Zookeeper based service discovery

Storage Scalability:
Refer to http://www.cloudbus.org/reports/DistributedStorageTaxonomy.pdf for a taxonomy of Distributed Storage Systems (DSS)
In summary, Distributed storage can be looked at from different perspectives. If we look at it from the point of view of "functionality" there is the following categorization:

  1. Archival: Provide persistent nonvolatile storage. Achieving reliability, even in the event of failure, supersedes all other objectives and data replication is a key instrument in achieving this
  2. General purpose Filesystem: Persistent nonvolatile  POSIX compliant filesystem e.g. NFS, CODA, xFS, 
  3. Publish/Share: More volatile, think peer-peer
  4. Performance: Operate in parallel over a fast network, typically will stripe data e.g. Zebra, 
  5. Federation middleware: Bring together various filesystems over a single API
  6. Custom: GFS (combination of many of the things above
Example
  1. DHT : Store the keys associated with a node in that node's DNS records (e.g. TXT record) and the node info is obtained via SRV record for that node (refer to : https://labs.spotify.com/2013/02/25/in-praise-of-boring-technology/)
Compute Scalability

There are 4 main categories of cluster workloads (ref: https://eng.uber.com/peloton/)

  1. Stateless jobs
  2. Stateful jobs
  3. Batch jobs
  4. Daemon jobs
The idea is to have these jobs scheduled diversely to a cluster. This is done using the tools such as Borg, YARN (slowly moving to Spark in the industry), Mesos and Kubernetes.

Thursday, May 2, 2019

Turing completeness using Mov

Apparently, turing completeness can be created using a single x86 instruction which is the MOV command:

https://drive.google.com/open?id=1cbnCSdBmkjEGxoScn2VtcR7SiC8hc45x 

Sunday, March 24, 2019

Type Systems in Computer programs

https://en.wikipedia.org/wiki/Type_system has good survey of the landscape. What I was looking for was duck typing (as happens in Golang)

Friday, March 15, 2019

An O(ND) Difference Algorithm and Its Variations

https://neil.fraser.name/writing/diff/myers.pdf  which is considered the best general purpose diff algorithm.  See this: https://github.com/google/diff-match-patch 

Friday, December 14, 2018

The unwritten laws of engineering

Came across this somewhat popular book of engineering - i may not fully agree with it, but it is definitely good advice on what it means to be a professional engineer. (http://rotorlab.tamu.edu/me489/README/2010%20ASME%20Unwritten_Laws_of_Enginering.pdf) 

Tuesday, November 20, 2018

Semantic versioning

https://semver.org/ A proposal to create some formalization around what the X.Y.Z terms mean in a version number and thus, make it clear as to which APIs are backwards compatible and which are not (APIs take in version number as an argument) 

Friday, October 5, 2018

Arguments against OOP

https://caseymuratori.com/blog_0015 "The fallacy of “object-oriented programming” is exactly that: that code is at all “object-oriented”. It isn’t. Code is procedurally oriented, and the “objects” are simply constructs that arise that allow procedures to be reused. So if you just let that happen instead of trying to force everything to work backwards, programming becomes immensely more pleasant."

Thursday, September 27, 2018

SRE role

HAProxy internals

http://www.haproxy.org/

Things to follow up on:
(1) How is zero copy done? Using the "splice" system call on Linux
(2) MRU memory allocator
(3) Accepting multiple accepts at the same time across different processing listening to different ports of course.
(4) Tree based storage e.g.making heavy use of the Elastic Binary Tree http://wtarreau.blogspot.com/2011/12/elastic-binary-trees-ebtree.html

Wednesday, April 4, 2018

Blog on machine learning

Came across what some people say as one of the best places to learn about machine learning. Making a note: https://colah.github.io/ 

Sunday, March 4, 2018

Mobile Real-time video segmentation

Came across this blog from google about how they have managed to find out the background in a video. https://research.googleblog.com/2018/03/mobile-real-time-video-segmentation.html  is the the link to the google research paper about this and https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf  is the paper they have built upon. I have to look into this soon. Adding as a follow up. 

Monday, January 22, 2018

AI Magazine

https://www.aaai.org/ojs/index.php/aimagazine/index 

What got me interested in this is : https://www.quora.com/What-is-the-biggest-unresolved-problem-for-AI, which is the quest for a general purpose intelligence i.e. building systems that can help us come up with the next "theory of relativity", or think like a human etc. Today's AI is mostly focused on classification 

Thursday, January 18, 2018

The set-theoretic multiverse

I came across this somewhat intriguing idea of a set theoretic multiverse. I want to try to find out the links between a set theoretical multiverse and logics. As i look at it, I see logic as a theory based on the rules of set theory to propagate labels such as truth/false. What I am curious about is whether if there are more than 1 foundational set theory - does that lead to many logics .

http://lumiere.ens.fr/~dbonnay/files/talks/hamkins.pdf 

Monday, May 22, 2017

Doxatic logic & types of reasoners

https://en.wikipedia.org/wiki/Doxastic_logic 
Something i read long time back when i read Smullyan's books, but making a note because i love it. His types of reasoners is beautiful :)


Types of reasoners[edit]

To demonstrate the properties of sets of beliefs, Raymond Smullyan defines the following types of reasoners:
  • Accurate reasoner:[1][2][3][4] An accurate reasoner never believes any false proposition. (modal axiom T)
  • Inaccurate reasoner:[1][2][3][4] An inaccurate reasoner believes at least one false proposition.
  • Conceited reasoner:[1][4] A conceited reasoner believes his or her beliefs are never inaccurate.
A conceited reasoner with rationality of at least type 1 (see below) will necessarily lapse into inaccuracy.
  • Consistent reasoner:[1][2][3][4] A consistent reasoner never simultaneously believes a proposition and its negation. (modal axiom D)
  • Normal reasoner:[1][2][3][4] A normal reasoner is one who, while believing  also believes he or she believes p (modal axiom 4).
  • Peculiar reasoner:[1][4] A peculiar reasoner believes proposition p while also believing he or she does not believe  Although a peculiar reasoner may seem like a strange psychological phenomenon (see Moore's paradox), a peculiar reasoner is necessarily inaccurate but not necessarily inconsistent.
  • Regular reasoner:[1][2][3][4] A regular reasoner is one who, while believing , also believes .
  • Reflexive reasoner:[1][4] A reflexive reasoner is one for whom every proposition  has some proposition  such that the reasoner believes .
If a reflexive reasoner of type 4 [see below] believes , he or she will believe p. This is a parallelism of Löb's theorem for reasoners.
  • Unstable reasoner:[1][4] An unstable reasoner is one who believes that he or she believes some proposition, but in fact does not believe it. This is just as strange a psychological phenomenon as peculiarity; however, an unstable reasoner is not necessarily inconsistent.
  • Stable reasoner:[1][4] A stable reasoner is not unstable. That is, for every  if he or she believes  then he or she believes  Note that stability is the converse of normality. We will say that a reasoner believes he or she is stable if for every proposition  he or she believes  (believing: "If I should ever believe that I believe  then I really will believe ").
  • Modest reasoner:[1][4] A modest reasoner is one for whom every believed proposition  only if he or she believes . A modest reasoner never believes  unless he or she believes . Any reflexive reasoner of type 4 is modest. (Löb's Theorem)
  • Queer reasoner:[4] A queer reasoner is of type G and believes he or she is inconsistent—but is wrong in this belief.
  • Timid reasoner:[4] A timid reasoner does not believe  [is "afraid to" believe ] if he or she believes 

Curry Howard Correspondence

Came across this which is a link between computer programs and logic, in the sense that using computer programs to "prove" things is legit. My interpretation of this is that it shows that constructs we use in computers languages can solve the same problems we solve using logic --

https://www.quora.com/Why-is-the-Curry-Howard-isomorphism-interesting

Not really a big deal, but something to make a note of, for future reference.


Wednesday, April 19, 2017

Thursday, February 23, 2017

Wednesday, January 4, 2017

Software Products someone needs to develop


  1. Given a program, create a flowchart out of it.
    • https://code2flow.com/  is one software which already does this. There may many approaches to building the UI, scaling the backend etc.
  2. Software idea generator. Create a list of software ideas e.g. Object Oriented, Operating system, Distributed, Data Store, Functional, Classification, Image Recognition, Ad Placement, Billing, etc. The point is that one first needs to come up with a design for this. 
  3. Automatically convert from one programming language to another -- cross compiler. Difficult problem to solve in general

Friday, October 7, 2016

Python Global interpreter lock !

http://www.dabeaz.com/python/UnderstandingGIL.pdf for reference. Amazed. I was falsely under the impression that my python pthreads were speeding things up :S 

Monday, September 19, 2016

Coding challenges

http://eudyptula-challenge.org/ Some hello-world like challenges for linux kernel
http://cryptopals.com/ similar challenges for cryptography

Thursday, May 5, 2016

On Software Architecture

Going to make a list of common software architecture patterns, and how does one go about choosing one over another. Here are a few links about this: I think some of these things are overlapping e.g. one can create a layered model using an event driven model for parts of the layer.

http://techbeacon.com/top-5-software-architecture-patterns-how-make-right-choice, where they are listing the models as

  1. Layered/n-tier model
  2. Event driven model
  3. Microkernel/Plug-in Architecture
  4. Microservices Architecture
  5. Space based architecture/Cloud Architecture
Also came across this interesting book about software architecture paradigms. I believe this would be very useful when it comes to understanding and designing software systems 


https://manohars.files.wordpress.com/2009/11/97-things-every-software-architect-should-know.pdf

Wednesday, April 6, 2016

How big is the Linux Kernel

http://superuser.com/questions/370586/how-can-a-linux-kernel-be-so-small



Early Linux distributions such as Tom's Root-Boot, MuLinux, Diet-router, (the now defunct) LOAFand many others now departed, all fitted a working Linux system on to one or two 1.44 MB diskettes.

The Linux kernel has grown but don't forget it is modular, kernel modules are loaded as needed. Thus it is still possible to produce a Linux distribution with a very small footprint.
See: Kernel Size Tuning Guide - eLinux.org

If you read Linux_tiny.pdf you will see this
historic 0.99pl15 kernel: Slackware 1.1.2, 1994 301K
Fedora Core 2 1.2M
SuSE 9.1 1.5M
2.6.5-tiny1 test config: IDE, ext2, TCP, NIC 363K

Thursday, March 31, 2016

Variation to a theme: Subtraction

Thoughts on async programming and Coroutines vs Generators vs Continuations

Here is my understanding of these related ideas.

Here is a model for async programming , where the + is the event loop. I think that one could use such kind of diagrams to model a async call model e.g. as one finds in Javascript, or as one constructs using libevent.



On a slightly different topic about continuations, co-routines, generators and functions, i look upon them as the following. This is still work in progress and I will update as my understanding improves. Something to begin with as a note:



Following is a quick note about how generators exist in python:
Generators in Python:

I have worked with generators in the context of RtmpLite

Generators are used to write non-blocking I/O in synchronous style without all the nesting etc that comes with asynchronous programming.  (ref: http://calculist.org/blog/2011/12/14/why-coroutines-wont-work-on-the-web/) . As shown below, the sleep(1000) will invoke a timer on the event loop and the control will yield back to the caller. Eventually, when the timer event triggers it will be added to the event queue and be executed starting off from where it left.

1
2
3
4
5
Task(function() {
    console.log('wait... ' + new Date);
    yield sleep(1000);
    console.log('ok... ' + new Date);
}).run();