Category: Internet

decentralized and distributed systems

Pointers for distributed vs decentralised:

Some bookmarks to read if you are a distributed systems researcher [1]

  • A decentralized system is a subset of a distributed system.
  • The primary difference is how/where the “decision” is made and how the information is shared throughout the control nodes in the system.
  • Decentralized means that there is no single point where the decision is made.
  • Every node makes a decision for it’s own behaviour and the resulting system behaviour is the aggregate response.
  • A key characteristic of decentral systems is that typically no single node will have complete system information.
  • “Distributed means that the processing is shared across multiple nodes, but the decisions may still be centralized and use complete system knowledge” says coinbase in their blog post
  • A scenario to think of:
    • Some control algorithms I’ve seen for multiple quad-copter control are purely distributed in that a central over-seer gives optimization problems to each copter to solve then return the solution.
    • The over-seer will then issue commands based on the aggregate result.
  • Here’s a philosophical question:
    • if the over-seer is “voted” into office by the nodes, is it decentralized or centralized?
    • I’d have to say decentralized, but it is arguable says MaRi Eagar
    • then I tend to ask what is distributed systems?

Keywords that matter

when you start the debate (here central systems are included): [2]

  • Points of Failure / Maintenance
  • Fault Tolerance / Stability
  • Scalability / Max Population
  • Ease of development / Creation
  • Evolution / Diversity

References:

  1. What is the difference between decentralized and distributed systems?
  2. Centralized vs Decentralized vs Distributed

DNSed: A simple useable internet name for large scale IoT

Intent

When there is a city wide Wireless Sensor Network (WSN), this has to be resolved with anything meaningful, say name of where it is located. then this has to be the name that it should be resolved to.

to creat DNSed, as in past tense in English for DNS. To DNS the shit out of the IoT. To create an amazing network and to overload the DNS servers. To create everything out in the open, so everything in the Hollywood movies can come true.

Scenario

  • When a water tank level indicator at place called “Meereen”.
  • There is a number for the water tank which is 256.

then, this can be described in the internet nomenclature as

  • domain name: meereen.got
  • object: water tank
  • number: 256
  • sensor type: level indicator

this would translate to an internet name of “level.256.water.meereen.got”

usually, there is only a sensor in a an area and they are organised in cluster.

in that case, Meereen’s water tank is being referenced, and the sensor 256 is being addressed where the level is being asked for, https://256.water.meereen.got/level where level is a service provided by the sensor

 

Possibility:

It is possible to use BIND9 to build a DNS server, then create the zone records to point to the different sensors. the tutorial gives a good insight on deploying a BIND9 DNS server [3]. There are very good python libraries available in such an instance to build an DNS server and then work with their API. The following are some interesting DNS APIs for python to process DNS records on server.

[1] http://www.dnspython.org

[2] https://pypi.python.org/pypi/easyzone/1.2.1

Implementation:

In this effort, the implementation takes advantage of the REST API provided by DigitalOcean to make the DNS. as the idea is to create a very dynamic name resolution for IoT devices and this does not involve building a DNS server from scratch. The DNSed will be a library that will used in the IoT devices as well to communicate to the central server which will be further proxied to the DNS Server of DigitalOcean. This architecture would provide a method to have a better security by not distributing the API keys throughout the network on every device.

The functional block diagram below tries to describes this exact architecture

drawit-diagram-2

References:

[1] http://www.dnspython.org

[2] https://pypi.python.org/pypi/easyzone/1.2.1

[3] https://www.digitalocean.com/community/tutorials/how-to-configure-bind-as-a-private-network-dns-server-on-ubuntu-14-04

[4] https://developers.digitalocean.com/documentation/v2/#domain-records

[5]
drawit-diagram-1

SensyLight: sensible atmosphere using Internet of Things

https://vimeo.com/192196471

The above video is from a research lab at the MIT Media Lab called Responsive Environments. They have a really interesting article [1] about a multimodal mediated work environment.

Internet of Things has been of great buzz these days. It is interesting, but why is it interesting? Just made a project/home lighting on this project.

So, here is the scenario for the internet of things. The thing in the Internet of Things is the web controlled lights – LED strip. The “control” part of the lights is managed by the Arduino. The task of Arduino would be to “GET” data that matters and send that info to the light strip. The ways in which the LED strip can be manipulated from the Arduino can be listed as

  • 1 LED can have 3 inputs, R G B.
  • Each R G B can value from 0 – 255 which is 256 values.
  • There are 32 LEDs in the strip.

Which makes a lot of math and logical decisions for the Arduino to handle. That is the whole point of networking these devices, now they have access to on-demand computing resources. This means we need the medium to connect the Arduino to the internet. That is done using the WiFi module which communicates with the Arduino using USART communication.

There are two ways to handle the information flow:

  • To directly to give out RGB information an hour
    • A lot of data transfer between the devices, but all of the computing, the decision is taken care of by the remote server
    • the device is highly dependent on the connection for operation
  • To query the server for Sunlight information (sunrise and sunset) and compute the colour information.
    • Having an update system to change the “compute” algorithm makes it highly robust.
    • Gets information for a day / a week and then “compute” with the algorithm. Also, listen for any settings like
      • Party lights
      • Work lights
      • get back to sunlight operation

Tools used:

Low cost, easy implementation of mediated atmospheres, to make your apartment to provide sensible lighting that can help harmonize the body and mind with the circadian rhythm.

[1] 17161.JosephA.Paradiso.Preprint1.pdf

[2] 192196471

Developing for ESP8266 using Arduino IDE

Cut to action steps:

  1. Install Arduino IDE
  2. Install board support packages to compile for ESP8266 
    1. http://arduino.esp8266.com/stable/package_esp8266com_index.json
    2. How and where to do it!
      1. first add that URL in the additional board managers URL
      2. now goto the board manager and look for ESP8266 and install the package
      3. compile your first program, don’t upload yet
      4. To upload, if you have NodeMCU, Arduino takes care of turning the boot-loader mode. if any other, then better to put it yourself into boot-loader mode.

Enjoy ESP8266 development!

 

I have an app idea, which cloud provider to choose?

There are millions of cloud providers out there, may be not, but there are enough to confuse you with their features and creative billing terms.

Ofcouse, its traditional VPS style and not using Docker to pack your app.

Now, the two cloud provider I choose to choose between are

Why this two?

I am not paid any one of those cloud providers, that further more vaguer for the reasoning. Being said that, to see from the perspective of highly scalable apps there are two philosophies.

  • I build my app using a framework from my cloud provider and my service provider will make it easier to deploy and scale up or down on demand
  • I setup everything and orchestrate the deployment process using chef, puppet or similar application and write my own logic for processing the on demand scaling

yes, ordered in the same order as the service provider. I will put my two cents of my philosophy in software development and this totally depends on the type of developer.

one follows the guidelines laid out by a framework to strictly develop using those or to use libraries to build ones own app philosophy which dictates its own framework as you build it up.

The fight is between libraries and frameworks, when you know what you are building and the final line is the light at the end of the tunnel, its probably much easier to find the suitable framework and nail it! important thing here would be that everything would have been already thought out for you by the developers. Whereas when you are building something that improvises over time, I prefer developing using libraries which gives a greater scope for solving problems.

Libraries are my way of doing. Choose generic libraries build out the app and improvise over time. frameworks might die out but standard libraries usually stay.

After those two cents, comes the drool factor, Does someone have a command line fetish in your team, stop reading here and starting using digitalocean.

So, whats the framework am I getting myself when i use AWS would be the next question! AWS has an amazing product line and most of them are amazing. I love how they built simple-storage-service s3, relational-database-service RDS or elasticache. But my fallout was using the elastic-bean-stalk EBS which is basically a framework that warps elastic-compute-cloud EC2, RDS and s3 and can be supplemented with all the other million products they offer.

comparing them for features of the two will be for another day

The advantages are

  • easily point-and-click and tell your app to scale up and down automatically
  • all services/servers are in one virtual-private-cloud
  • scalable online for block storage
  • create database, cache replications and clusters on the go
  • of course, elastic load balancer… its so elastic it scales up and forgoes provisions after the peak

all of it points towards high availability for delivering your app the two things I would point out and not be embarrassed about them are

  • the cost at which it comes, mostly any startup wants to be lean

thats a framework, its also well supported by packages for most of the app development frameworks and libraries that you can import and give the URI, username and password and good to go.

Now, looking at the digital ocean, there is no product offering except for selling a highly available, well distributed cloud. start a VPS, migrate to any location, make it available at any data centre at the same price. The whole look is amazing at digital ocean and I love them for being progressive as a company and to offer a service that is startup – developer friendly.

Here, you get mailed with a IP and root password to ssh into and the rest is history. Everything is open and its up to the developer to set the standards and formalise rules for the app. Choose the stack for development and create a deploy strategy.

i feel pretty flexible and always suggest it with a strong positive feeling that anything is possible / everything is possible if one can envision the requirement.

So, that app we developed, I started out without a clear vision on EBS but we wanted to do something with consuming web API from social media. It was first text, then came the images and now we are working with rendering videos. The first library we wanted in python required third party system packages and that wasn’t being installed due to repository problem. that was the first fall out from EBS to EC2. EC2 proved clearly the way to go. its open, its also a stackless framework-less VPS which only supports SSH with key based authentication. that ended up in messing up with deploying from different machines and losing access to the machine. Of course there is a guide to get your stuff back, but its just a pain during development. Then we switched to digital ocean, devOps took some time but three weeks later it was time to ship the first version!

Digital ocean wins on two grounds for me is framework-less development with developer friendly billing. Write me if you want know more about all the other issues because the deployment is 98% and I will continue to setup the system 🙂

Installing MEAN stack for Fullstack application development

The best one liner for what MEAN stack is given in the following picture.Screen Shot 2015-01-18 at 22.48.40

MEAN stack is more like LAMP.

M = MongoDB

E = ExpressJS

A = AngularJS

N = NodeJS

MEAN Stack Logo

There has been a lot of development from Google for the AngularJS and NodeJS. They are minimalistic and javascript based, for now cutting edge web development tools.

For installing MEAN stack in ubuntu, follow the commands to set up your MEAN Stack. I deployed my server in Digitalocean which is like my obsessed service for cloud deployment.

Install MongoDB
#~ sudo apt-get install mongodb
Install NodeJS and Npm
#~ sudo apt-get install nodejs npm
and create a symlink
#~ sudo ln -s /usr/bin/nodejs /usr/bin/node
Install Bower
#~ sudo npm install -g bower
Install Grunt
#~ sudo npm install -g grunt-cli
Install MEAN
sudo npm install -g mean-cli
Init your first App
$ mean init testApp
type in your IPaddress with port 3000
http://xxx.xxx.xxx.xxx:3000

TI CC2530DK – Contiki OS – Internet of Things

This blog post is an effort to create a one stop information for starting off your Contiki project with cc2530DK.

There is enough information already available on Github wiki. This post is more than an extension to have all information under one place and as usual my way of documenting my projects!

Continue reading “TI CC2530DK – Contiki OS – Internet of Things” →

WiFi Positioning System – Using Python and Google Geolocation API

I consider there is these following steps involved for such a program.

I will keep this as a documentation as well as a progress report to keep track of the project.

  1. Reading all the required information
    1. Reading SSID in MAC
      • Using Airport Shell command
      • Here is how it should look: [viraldownloader id=180]
    2. Reading SSID in Linux (Ubuntu), Windows
      • identify shell command for linux/Ubuntu
      • identify shell command for windows
  2. Parsing them in JSON format
    1. Parsing information into required information into prescribed format
  3. CURL and Get the Location information in JSON
  4. Parse it and display on google
  5. Building a GUI

Here is the gitHub repository for the project that I am working on.

https://github.com/akrv/pyWPSLocalisation

Feel free to contribute/contact it is interesting for you aswin[at]southpaw.in

All about Google WiFi Positioning System

WiFi Positioning system is a new way to get precise localisation. It is used in conjunction with GPS to achieve better accuracy. Read through to know how there is a slight privacy intrusion and how you can take care of your privacy issues. I am also sharing some interesting things about this WiFi Positioning System (WPS) that Google offers through its developer API.

It is interesting that Google asks for you to sign up for logging the access point (AP) information in your android phone. Well if you knew when and where it is asked, you will surely hit NO!

They collect your AP information mostly without clear notice and then you have to pay to use the API to access this information. Atleast this is what Google does.

Well some digging says it is just done periodically. My conclusions of how AP information tagged with location is logged are as follows

  • It is somewhere connected to location reporting from Google Location Services.
  • It could be just the localisation data that is sent to Google to while you are using your apps that require location.

So Google actually logs all the Access Point (AP) a device connects to and with the help of GPS it is geolocated and the info is tagged and sent to Google servers!

How to opt out of this WPS?