Category: Cloud

decentralized and distributed systems

Pointers for distributed vs decentralised:

Some bookmarks to read if you are a distributed systems researcher [1]

  • A decentralized system is a subset of a distributed system.
  • The primary difference is how/where the “decision” is made and how the information is shared throughout the control nodes in the system.
  • Decentralized means that there is no single point where the decision is made.
  • Every node makes a decision for it’s own behaviour and the resulting system behaviour is the aggregate response.
  • A key characteristic of decentral systems is that typically no single node will have complete system information.
  • “Distributed means that the processing is shared across multiple nodes, but the decisions may still be centralized and use complete system knowledge” says coinbase in their blog post
  • A scenario to think of:
    • Some control algorithms I’ve seen for multiple quad-copter control are purely distributed in that a central over-seer gives optimization problems to each copter to solve then return the solution.
    • The over-seer will then issue commands based on the aggregate result.
  • Here’s a philosophical question:
    • if the over-seer is “voted” into office by the nodes, is it decentralized or centralized?
    • I’d have to say decentralized, but it is arguable says MaRi Eagar
    • then I tend to ask what is distributed systems?

Keywords that matter

when you start the debate (here central systems are included): [2]

  • Points of Failure / Maintenance
  • Fault Tolerance / Stability
  • Scalability / Max Population
  • Ease of development / Creation
  • Evolution / Diversity

References:

  1. What is the difference between decentralized and distributed systems?
  2. Centralized vs Decentralized vs Distributed

I have an app idea, which cloud provider to choose?

There are millions of cloud providers out there, may be not, but there are enough to confuse you with their features and creative billing terms.

Ofcouse, its traditional VPS style and not using Docker to pack your app.

Now, the two cloud provider I choose to choose between are

Why this two?

I am not paid any one of those cloud providers, that further more vaguer for the reasoning. Being said that, to see from the perspective of highly scalable apps there are two philosophies.

  • I build my app using a framework from my cloud provider and my service provider will make it easier to deploy and scale up or down on demand
  • I setup everything and orchestrate the deployment process using chef, puppet or similar application and write my own logic for processing the on demand scaling

yes, ordered in the same order as the service provider. I will put my two cents of my philosophy in software development and this totally depends on the type of developer.

one follows the guidelines laid out by a framework to strictly develop using those or to use libraries to build ones own app philosophy which dictates its own framework as you build it up.

The fight is between libraries and frameworks, when you know what you are building and the final line is the light at the end of the tunnel, its probably much easier to find the suitable framework and nail it! important thing here would be that everything would have been already thought out for you by the developers. Whereas when you are building something that improvises over time, I prefer developing using libraries which gives a greater scope for solving problems.

Libraries are my way of doing. Choose generic libraries build out the app and improvise over time. frameworks might die out but standard libraries usually stay.

After those two cents, comes the drool factor, Does someone have a command line fetish in your team, stop reading here and starting using digitalocean.

So, whats the framework am I getting myself when i use AWS would be the next question! AWS has an amazing product line and most of them are amazing. I love how they built simple-storage-service s3, relational-database-service RDS or elasticache. But my fallout was using the elastic-bean-stalk EBS which is basically a framework that warps elastic-compute-cloud EC2, RDS and s3 and can be supplemented with all the other million products they offer.

comparing them for features of the two will be for another day

The advantages are

  • easily point-and-click and tell your app to scale up and down automatically
  • all services/servers are in one virtual-private-cloud
  • scalable online for block storage
  • create database, cache replications and clusters on the go
  • of course, elastic load balancer… its so elastic it scales up and forgoes provisions after the peak

all of it points towards high availability for delivering your app the two things I would point out and not be embarrassed about them are

  • the cost at which it comes, mostly any startup wants to be lean

thats a framework, its also well supported by packages for most of the app development frameworks and libraries that you can import and give the URI, username and password and good to go.

Now, looking at the digital ocean, there is no product offering except for selling a highly available, well distributed cloud. start a VPS, migrate to any location, make it available at any data centre at the same price. The whole look is amazing at digital ocean and I love them for being progressive as a company and to offer a service that is startup – developer friendly.

Here, you get mailed with a IP and root password to ssh into and the rest is history. Everything is open and its up to the developer to set the standards and formalise rules for the app. Choose the stack for development and create a deploy strategy.

i feel pretty flexible and always suggest it with a strong positive feeling that anything is possible / everything is possible if one can envision the requirement.

So, that app we developed, I started out without a clear vision on EBS but we wanted to do something with consuming web API from social media. It was first text, then came the images and now we are working with rendering videos. The first library we wanted in python required third party system packages and that wasn’t being installed due to repository problem. that was the first fall out from EBS to EC2. EC2 proved clearly the way to go. its open, its also a stackless framework-less VPS which only supports SSH with key based authentication. that ended up in messing up with deploying from different machines and losing access to the machine. Of course there is a guide to get your stuff back, but its just a pain during development. Then we switched to digital ocean, devOps took some time but three weeks later it was time to ship the first version!

Digital ocean wins on two grounds for me is framework-less development with developer friendly billing. Write me if you want know more about all the other issues because the deployment is 98% and I will continue to setup the system 🙂