Shorts - week 3, 2022

Post with links to what I am reading:

1. A very good post on different aspects of system architecture:

https://lethain.com/introduction-to-architecting-systems-for-scale/

2. About market making and mean-reversion:

https://www.cis.upenn.edu/~mkearns/papers/marketmaking.pdf

3. Why we use log returns in finance?

https://www.youtube.com/watch?v=PtoUlt3V0CI

ZeroMQ Messaging Patterns

In software architecture, a messaging pattern is an architectural pattern that describes how two different parts of an application or different systems connect and communicate with each other (source: Wikipedia).

ZeroMQ supports many messaging patterns. This document lists down the most often used, with samples shown in C# CLRZMQ4 library.

1. Fire and Forget (or Push and Pull)

This is a one-way pattern, the sender sends a message to the queue and it doesn’t care about any response from the message handler. Zero MQ enforces this pattern, i.e. if the receiver sends a response, it throws an error.

It’s important not to think in terms of client/server semantics here because the component that binds to a port can either PULL or PUSH based on the socket type, and one that connects can also do either. Producer/consumer terminology suits better.

For instance, multiple senders can connect and start pushing messages on a queue (e.g. heartbeat) or say a component like a work producer that pushes work to a queue, and these get pulled by multiple workers to load balance the work (essentially multiple producers and a single consumer, or a single producer and multiple consumers).

Sender:

            var sender = new ZSocket(ZSocketType.PUSH);
            sender.Connect("tcp://127.0.0.1:5678");
            for (int i = 0; i < 5; i++)
            {
                sender.Send(new ZFrame(i));
            }        

Receiver:

            ZSocket receiver = new ZSocket(ZSocketType.PULL);
            receiver.Bind("tcp://*:5678");
            while (true)
            {
                using (ZFrame request = receiver.ReceiveFrame())
                {
                    var message = request.ReadInt32();
                    //process message
                }
            }

2. Pub-Sub

This is the multicast pattern. Subscriber registers with a queue; publisher sends the message; once queue receives the message it’s forwarded to all the subscribers. This is a data distribution pattern, involving a one-way transfer from a source to multiple consumers of data.

For example:

Publisher:

            var publisher = new ZSocket(ZSocketType.PUB);
            publisher.Bind("tcp://*:5679");

            int i = 0;
            while(true)
            {
                publisher.Send(new ZFrame(i));
                i++;
            }

Subscriber

            var subscriber = new ZSocket(ZSocketType.SUB);
            subscriber.Connect("tcp://127.0.0.1:5679");
            subscriber.SubscribeAll();
            while (true)
            {
                var msg = subscriber.ReceiveFrame();
            }


3. Request-Reply

This is a pattern to implement a service when we need to call a function remotely. Zero MQ enforces that the requestor and responder orchestrate their actions. For e.g. requestor cannot call “send” two times in a row. Once a call is made to the server, the client needs to wait for the reply.

Requestor (client)

            var requester = new ZSocket(ZSocketType.REQ);
            requester.Connect("tcp://127.0.0.1:5675");
            for (int i = 0; i < 5; i++)
            {
                //send request
                requester.Send(new ZFrame(i));
                //process server's reply
                var reply = requester.ReceiveFrame();
            }

Responder (server)

            var responder = new ZSocket(ZSocketType.REP);
            responder.Bind("tcp://*:5675");
            while (true)
            {
                using (ZFrame request = responder.ReceiveFrame())
                {
                    //wait for request
                    var message = request.ReadInt32();
                    //send response
                    responder.Send(new ZFrame(message++));
                }
            }

C# Reactive Extensions - Buffer and Window

I was going through Buffer and Window in RX, thought a few examples would help clear the differences.

First create a buffer of even numbers, each buffer having 5 items:

(we can ignore Timestamp() for now, that will be used later when we see how buffering and windowing work with streams with items coming in over time)

int bufferId = 0;

var bufferedNumberStream = Observable.Range(0, 100)
         .Where(v => v % 2 == 0)
         .Buffer(5)
         .Timestamp()
         .Subscribe(ts =>
         {
               bufferId++;
               Console.WriteLine("Buffer ID::" + bufferId);
               ts.Value.ToList().ForEach(x => Console.WriteLine(x));
         });

Output: Buffers created with 5 items in each buffer

Odd numbers stream with window with 5 items in each:

int windowId = 0;

var windowedNumberStream = Observable.Range(0, 100)
    .Where(v =>; v % 2 == 1)
    .Timestamp()
    .Window(5, 5)
    .Subscribe(w =>;
    {
         windowId++;
         Console.WriteLine("Window ID::" + windowId);
         w.Subscribe(ts =>;
            {
                Console.WriteLine("Value::" + ts.Value);
             });
    });

The line .Window(5,5) means the window will have count of 5 items, and the next window will be created after skipping these 5 items. (count, skip) or (range, slide).

Output:

Difference 1:

The primary difference in buffer and window is that a buffer gives you an IList of items, in our example IList, and window gives you and IObservable that you need to again subscribe to see the values inside the window.

So .Window() gives back IObservable>, the windows are observable and the items in every single window are also an IObservable.

Difference 2:

A new buffer starts as soon as there are n items are in it. However a window needs two things, the items to be in it, and how far to slide the window. In a window it's possible that you process an item in the stream again, but that's not possible in buffer.

Moving average is excellent example of window.

-----------------------------------------------------
In the next post, let's see how buffer and window behave when we create them based on time stamp and not based on number of items in them.

Visitor Pattern

In the last post (please check that out before reading this, as it shows the problem we are trying to solve here) we saw how C# uses single-dispatch to select the methods to call. The argument type is determined at compile time and that is used to select the method out of many.

Here is an implementation of visitor pattern to resolve that problem:

void Main()
{
 Employee employee = new Employee();
 Employee manager = new Manager();

 IVisitor visitor = new SalaryAlgoPerformanceVisitor();
 
 employee.Accept(visitor);
 manager.Accept(visitor);
 
 Console.WriteLine(visitor.Output);
}

interface IVisitor
{
 string Output {get;}
 void Visit(Employee e);
 void Visit(Manager e);
}

class SalaryAlgoPerformanceVisitor : IVisitor
{
 public string Output { get; private set;}
 public void Visit(Employee e)
 {
  Output += $"Calculating salary of {nameof(Employee)} based on {nameof(SalaryAlgoPerformanceVisitor)}\n";
 }
 public void Visit(Manager e)
 {
  Output += $"Calculating salary of {nameof(Manager)} based on {nameof(SalaryAlgoPerformanceVisitor)}\n";
 }
}

class Employee
{
 public virtual string GetEmployeeType()
 {
  return nameof(Employee);
 }
 
 public virtual void Accept(IVisitor visitor)
 {
  visitor.Visit(this);
 }
}

class Manager : Employee
{
 public override string GetEmployeeType()
 {
  return nameof(Manager);
 }
 
 public override void Accept(IVisitor visitor)
 {
  visitor.Visit(this);
 }
}

Output:

Calculating salary of Employee based on SalaryAlgoPerformanceVisitor
Calculating salary of Manager based on SalaryAlgoPerformanceVisitor

You will notice that Accept method is resolved using the run time type of Employee object - this is the first dispatch. Then in the Accept method we use (this) so that appropriate method is called on the visitor - this is the second dispatch.

Since Accept is inside the class, we use the type of (this) to call the visitor.

Double Dispatch in C#

First few tests to understand the basics:

What do you think about the GetEmployeeType() method that will be called - will it be on Employee class or Manager class?

void Main()
{
 Employee e = new Manager();
 Console.WriteLine(e.GetEmployeeType()); 
}

class Employee
{
 public virtual string GetEmployeeType() {
  return nameof(Employee);
 }
}

class Manager : Employee
{
 public override string GetEmployeeType() {
  return nameof(Manager);
 }
}

Output: Manager

Since Employee object 'e' points to Manager, it will call the GetEmployeeType of Manager class.

Another one:

void Main()
{
 Employee employee = new Manager();
 Console.WriteLine(CalculateSalary(employee));
}

string CalculateSalary(Employee e)
{
 return $"{nameof(CalculateSalary)} of {nameof(Employee)}";
}

string CalculateSalary(Manager e)
{
 return $"{nameof(CalculateSalary)} of {nameof(Manager)}";
}

class Employee ...

class Manager : Employee ...

Output: CalculateSalary of Employee

Since the object is of type Employee the CalculateSalary accepting Employee will be called.

Taking this further, let's move the CalculateSalary to a class.

void Main()
{
 Employee employee = new Employee();
 Employee manager = new Manager();

 SalaryAlgo algo = new SalaryAlgoBasedOnPerformance();
 Console.WriteLine(algo.CalculateSalary(employee));
 Console.WriteLine(algo.CalculateSalary(manager));
}

class SalaryAlgo
{
 public virtual string CalculateSalary(Employee e)
 {
  return $"{nameof(CalculateSalary)} of {nameof(Employee)} based on {nameof(SalaryAlgo)}";
 }

 public virtual string CalculateSalary(Manager e)
 {
  return $"{nameof(CalculateSalary)} of {nameof(Manager)} based on {nameof(SalaryAlgo)}";
 }
}

class SalaryAlgoBasedOnPerformance : SalaryAlgo
{
 public override string CalculateSalary(Employee e)
 {
  return $"{nameof(CalculateSalary)} of {nameof(Employee)} based on {nameof(SalaryAlgoBasedOnPerformance)}";
 }

 public override string CalculateSalary(Manager e)
 {
  return $"{nameof(CalculateSalary)} of {nameof(Manager)} based on {nameof(SalaryAlgoBasedOnPerformance)}";
 }
}

class Employee ...

class Manager : Employee ...

Output:

CalculateSalary of Employee based on SalaryAlgoBasedOnPerformance
CalculateSalary of Employee based on SalaryAlgoBasedOnPerformance

As you can see, CalculateSalary(Manager) is not at all called. This is because the method to call is decided at compile time and the declared type is used to identify the method not the runtime type.

A quick way to fix this is to use "dynamic" keyword:

 Console.WriteLine(algo.CalculateSalary((dynamic)manager));

Output:

CalculateSalary of Employee based on SalaryAlgoBasedOnPerformance
CalculateSalary of Manager based on SalaryAlgoBasedOnPerformance

Another way to resolve this is to use Visitor pattern.

Helpful links:

https://blogs.msdn.microsoft.com/curth/2008/11/15/c-dynamic-and-multiple-dispatch/

EDIT: Post that implements Visitor pattern to resolve this problem.

Python: Certificate of Completion

Python:

I purchased Kindle book: "A smarter way to learn Python" and completed all its exercises.
Finally I got the certificate from the author: ye!

Python: Functions - Part 1

Quick post Python functions:


Define a function and call it:

def add(first_num, second_num):
    sum = first_num + second_num
    print(sum)

>> add(1, 2)
3

Two types of arguments: positional and keyword:
Positional - defined by order of parameters in the method
Keyword - use the parameter name when calling the function

You can forget what order to use when all of your arguments are keyword arguments:

add(second_num = 3, first_num = 1)

But when mixing keyword arguments with positional you need to take care that all keyword arguments are at the end:

add(second_num = 3, 3)
SyntaxError: positional argument follows keyword argument

In the call below, the first argument of value "1" is taken in as parameter "first_num" based on it's position, so defining it again as keyword Python will give error:

>>> add(1, first_num = 3)
Traceback (most recent call last):
  File "<pyshell#9>", line 1, in <module>
    add(1, first_num = 3)
TypeError: add() got multiple values for argument 'first_num'

>>> add(2, second_num = 3)
5

Default value:
Set a default value for a parameter in the method definition by using "=" sign.
In this example, the second_num parameter defaults to value of 5.
You can skip passing in second argument and the function will run fine.

>>> def add(first_num, second_num = 5):
 sum = first_num + second_num
 print(sum)

>>> add(3)
8

If you don't want to use default value of 5, then call the function the usual way.

>>> add(4, 5)
9

If you have many default parameters, you can use keyword to assign a different value to one of them and skip the rest. For example:

>>> def calc(a, b, c = 10, d = 20, e = 30):
 print(a + b + c + d + e)

# skip all default parameters
>>> calc(1, 2)
63

#when not using default val of 'e'
>>> calc(2, 3, e = 45)
80

Next post: Working with unknown number of parameters

Onwards and Upwards

I have concluded that I a need a skill-upgrade. Although my day to day work requires C# and bit of Java, this is not going to be enough.

Javascript is no longer something that I can ignore, neither is Python. In fact JS has come a long way to be used to build a variety of applications - great UIs to services.

And Python, nothing more can be said about it. This language has become such that every programmer should know at least how to read Python code.

So this year, embarking on journey to learn Python and JS along with Angular/TypeScript - at least to the level of intermediate programmer.

Of course along with this delving deeper into core areas such as agile patterns and SOLID principles.

Python 101 - Part 1

PEP	Python Enhancements Proposal (guidelines) PEP 8: Style Guide For Python Code PEP 20: Zen of Python
Indentation	Used to mark blocks, curly braces no longer needed Recommendation is to use 4 spaces for indentation, be consistent and not to mix spaces with tabs
Modules	- Use modules available in Python e.g.: import math After importing use the module name to call functions: math.sqrt(5) math.factorial(5) from math import factorial -> factorial(5) # avoid using math from math import factorial as fac -> fac(5) #shorter name
Help (in REPL cmd)	- Get help on a module help(math) - Get help on a function help(math.sqrt) help(math.factorial)
int, float	int(10), int(10.2), int(“10”) float(10), float(10.2), float(“10.2”)
Nan,inf	float(“nan”) # not a number float(“inf”) # infinity
None	Represents absence of value a = None a Is None # returns True
Boolean	0: falsy Non-zero (inc. negative): truthy bool(0) bool(0.0): # False bool(-1) # True For Lists If list is empty: bool(list) returns False bool([]) # False bool([1,2,3]) # True For strings: bool(“”) # False bool(“a”) # True bool(“      “) # all white spaces, True if “ihavesomething”:  # this is True     print(“this is True”)
Raw strings	path = r’C:\Users\Documents\Course’

Passed CFA Level 2

Now on to CFA Level 2 - June 3rd 2017.

I really love this period of preparing for something worthwhile - which requires your full dedication, focus and attention. It's almost as if you become one with your goal.

Edit [26-July-2017]:
And I cleared it !

Passed CFA Level 1

Happy to share that I cleared CFA Level 1.

Level 1 is supposedly the easiest one out of all level, but it does require time and effort. Thanks to my loving family for bearing with me :)