Real-time Web Applications (All About Real-time Web)

 Introduction

Over the period of time there has been lots of changes in the world of Web. We thought of the simplest web sites specific to devices then adaptive and responsive ones. The one thing that always pinches a user of a website is to refresh this all the time in order to see the latest changes on the web. Think of a scenario where you are monitoring the scores of a cricket match which is going on... and you need to keep on refreshing your web site every time when you want to see the latest scores. What if this automatically pops up to you !! Here comes the importance of Real-time Web.
                        Think of Twitter. You are viewing one twit and at the same time you see another twit popping up. You didn't refresh the page. Same with other social sites as well for example : Facebook.

Let's Define Real-time Web
"A set of technologies and practices that enable users to receive information as it is published."
Modern web applications go well beyond what the traditional limitations of the browsers. State of the art web applications involve real-time data feeds, video and audio streams. Machine-to-Machine-to-Browser communications and many more...

 Reasons For The Requirement Of Real-time Web

1. Improved Server Power [ Now-a-days the servers are becoming stronger. Processors and memories are going cheaper & faster. More data now can be processed. Connections can now be dealt with (C10K Problem). At the same time scaling has now become easier  ] 
2. Web Browsers Capabilities and Consistencies [ There are many factors like : Cross browserXMLHttpRequest Support, CORS(Cross-Origin Resource Sharing), Server Sent Events, Web Socket, WebRTC ]
3. Software Choices [ There are lots of languages and run time options available today. More and more open source solutions are coming up  ]
4. Massive Increasing Internet Usages

Below snap shot shows an approximate usage of internet in various areas

 Looking Back In The History (A Journey...)

• 1991 , the first documented version of HTTP was released. [ HTTP is a request response protocol where a client initiates a connection and requests for data. The server then responds and the connection is closed. ]
• It was required to have a web communication where the connection could be maintained. [ So that either side of  the connection if had new data , it could be instantly sent to the other party. ]
• Until 1996 , it was possible to create a persistent client-server connection in the web browsers using Java Applet.
• Unfortunately this never became popular due to the lawsuit [ lack of support of Java Applets on MS IE. ] 
• By 2000 , developers managed to hack some purely native web-browser solutions using persistent HTTP connection.
• 2009 , Robert Scoble [ a good name in UI world] wrote a blog post using a service called “FriendFeed”. The beauty of this service was , the moment you finish posting this instantly becomes discoverable through search
• Even the Wikipedia’s first definition to real-time web was given in mid 2009

• Soon we started seeing it partly enabled in social networks : Facebook, Twitter
• Expert Says "This was not exactly the real-time web". This could not be used to control extensive things like Safety Systems in nuclear plants or applying breaks in your car
• Remarkable point is : They were able to send data in < 300 MS
• Today we have lots of technologies and architecture patterns for achieving this

Below snap shot shows some of the technologies/patterns etc. for the real-time web

Who Needs The Real-time Web?

• Social Media : [ Twitter, Facebook etc. dont't wait till you refresh the page to let us know whether newer twits have arrived or new notifications have popped up ]
• Internet Of Things(IOT) : [ Building web applications that interact with internet-enabled devices and displaying the latest data from devices to M2M networks on the browsers in real time by pushing the data ]
• Data Dashboards : [ Modern dashboards are much more than reports. Real-time messaging can be used to push data alerts and updates to the dashboards automatically ]
• Corporate Communications : [ One can build his/her own collaborative communication tools directly into other enterprise applications to that other team can communicate using data, video and audio
• Video/Audio Manipulations : [ Advantage of WenRTC (Web Real-time communication) integration with browser can be taken to implement sound and video processing that happens in real-time as the data flows in  ]
• Education & Telemedicines : [ Real-time communication via video or audio channels in to someone's application in a secured, peer to peer, encrypted standards to enable educators, doctors or others to work reportedly   ].

Some Examples...

• Social Sites 

• Cloud 9 IDE : [ It's an IDE based on browser. It allows multiple developers to code together. Developers can edit code, chat and even execute and debug together. ]

• Super Sync Sports : [ Super Sync Sports is a Chrome experiment built to demonstrate WebSocket capabilities. One can user his/her mobile as a controller while the game runs in the desktop browser. It supports both single and multi player. It demonstrates real-time web technology to create fun experience ] 

• Murally : [ Murally is for the teams who want to collaborate in visual way. Murals are online walls with web contents (i.e Video, links, photos etc.). Contents can be easily dragged and dropped. Additions and edits are instantly shown to all collaborators. Link : https://mural.co/ can be visited for more details.  ]

• FirePad : [ An Open source web app and framework - to be used by multiple users. It's a real-time application which allows more than one users to write and share. This can also be used as a framework in any web application. Additions and edits are instantly shown to all collaborators. Link : https://firepad.io/#1 can be visited for more details. ]

• Caplin Trader : [ One of the pioneers of real-time web technologies. Their front end products continue to demonstrate cutting-edge use of technology. Includes real-time charting, grids with thousands of rows getting updated. Real-time news and trades titles for executing high value trades. Link : http://www.caplin.com/ can be visited for more details  ]

• Arduinos : [ Lots of cool things being created with Arduinos that uses real-time web technology. Some of the examples are : controlling power switches via SMS, Webhooks, and WebSockets. Link : https://www.arduinos.co/ ] 

Options Available To Develop Real-time Web Applications:

• Polling
• AJAX Long Polling
• Forever-Frames
• Server Sent Events
• Web Sockets

There are couple of Questions and Considerations
What are the core concepts used in real-time web technologies?
Pretty clear and obvious, you need to connect to the source of data in order to receive new information from it.
What are the fundamental raw techniques?
Polling - Short or Long.
Is it possible to break the types of functionalities up into categories?
Yes, we are going to explore all of them.
What happens in case of Polling?

• A client requests for resources from the server and server responds
• The client keeps on requesting data from the server at an interval
• The server then responds and passes the data back to the client
• Short polling is likely to use resources unnecessarily 
• Example of Polling : Average 10,000 users for a site | This site runs around 1 hour | 5 seconds polling . Let's say page load + resource estimate (HTML , CSS, JS, Image) = 50,000. Poll requests per minute = (60/5) = 12. Poll requests per hour = 12 * 60 = 720. Poll requests site wide per hour = 720 * 10,000 = 7,2000,000 i.e. Total : 7.2 Million Requests.
• With any of the standard real-time web technologies solution : You have dedicated and specialized resource to maintain those 10,000 persistent connections. Site average of 10,000 users online at any point of time. Around 1 hour , No Polling. Page load + Resource estimate (HTML , CSS, JS, Images) = 50,000. That's it ! Our standard web servers handle 50,000 requests.  

We have seen above the issue involved with polling implementation. We really need to implement some other mechanism for achieve this.

Factors To Select The Best Solution
As we have seen the drawbacks of short polling mechanism, we really need to look for other possibilities.
Number of Factors:
Use the existing solution
Use the existing solution , unless you have a very unique requirement. There is no point building something from ground up. There can be many reasons for the same

• Fallback/upgrade hacks still required
• Support/community
• Maintenance
• Future features
• Scaling

Use The Language You Are Comfortable With:

Native Mobile Support:

• Initial focus with most of the technologies was : the desktops web clients
• we cannot forget or ignore mobile these days. And we see that only few have mobile client support
• How much data are you sending? [ Using PuSub or Sync with deltas ] 
• SSL required for 3G /4G networks 
• HTTP fallback required
• Faye, Firebase , Hydna, PubNub , Pusher, SignalR , Socket.IO are considered to be good for native mobile as well

Understand The Application's Functionality

• Information Architecture [ Data ]
• Interaction complexity [ How users will interact with your application. Client <-> Server interactions. Understand the types of functionality you need in your applications- OnMessage, PubSub, Sync, RMI 

Sever Sent Event

• Server Sent Event [SSE] is a recent HTML 05 specification
• This is used in combination with EventSource API for streaming updates
• The specification says “API for opening an HTTP connection for receiving push notifications from a server in the form of DOM events. The API is designed such that it can be extended to work with other push notification schemes”
• SSE provides two types of components
• EventSource Interface [Allows the client to receive the push notification from the server ]
• Event Stream [Event stream is the data format used to deliver the individual updates ]
• Server Sent Events are found to be working fine in the applications such as Stock Ticker, Feed reader where the server responds with the message to the client. The client receives it and shows it to the user
• In these kind of applications the client doesn’t post the messages to the server very often
• Another advantage of SSE is , it operates over HTTP/HTTPS meaning it doesn’t require a special protocol or any other server side implementation as well as it works on the same proxy settings
• The EventSource would be working in the background without blocking any other scripts
• You don’t need any new language , whatever backend language you are using will work

Server Sent Event - Demo
Let's have a look on how are we going to achieve real-time web using one of the features called as Server Sent Events.

Below is the snap shot capturing on the UI (page) design done using HTML 5 & CSS 3 under ASP .Net.

Below is the snap shot of the rows from the employee table (after performing SELECT *  )

Below is the client side code which sets Event Source with one of the ASP .Net (aspx) pages which contains the logic to get data from the DB. Based on the data received they are read and populated to the table created in the UI.
We are creating a source object here. This provides us different events. onopen is the one where we say that the connection with server is now open.
source.onmessage , we get event.data which is the data sent from the server.

Below snap shot shows the code written in the test.aspx.cs file.

• A new EventSource object is created by specifying the URL of the page that is going to send the updates (in our case this is : test.aspx)
• Each time an update is received, the onmessage event occurs
• When onmessage event occurs, the data is captured by the client and populated in the UI

Controlling The Reconnection Time With Server Sent Event

• Roughly the updates from server take place at the interval of 3 seconds after each connection is closed
• One can change the time/interval by including an hint from the server code => retry: milli seconds (for example : retry: 20000\n will wait for 20 seconds)

Below snap shot shows how do we control the reconnection timings in case of Server Sent Events

Server Side Event Supporting Multiple Events
As of now in the above example , we have seen server sent event supporting only one event. Means only one update. There are possibilities and requirement as well to have multiple updates. Below example shows multiples events so that there would be many updates.