Information Technology Evolution & Intelligent Data Crunching

 Today's Data 

The Information Technology is being driven by intellect and intelligence obtained based on data. Initially Databases were at one place and it was sufficient to provide all required data. when networking among people flourished through internet, networking among department and organizations mandated Data Warehousing units, to manage data stores. Social Networking brought many data which were considered irrelevant earlier to a relevant state and it has made a mesh of inter networking among individuals, systems, devices and organizations which was not feasible earlier. Today Data Sources are not just in personal computers, but are on Embedded Devices & numerous embedded devices are flourishing. And Data consumers lies through out the Internet in the form of manufactures, retailers, eCommerce vendors etc.,

The below depiction is relevant only to show that today static Intelligence is bare minimum and Adaptive Intelligence is on demand with the kind of data obtained and its sources.



Data Crunching

Data obtained from various sources has to be crunched to make intelligence out of it. The data is being crunched by machines in binary. Machines do have constraints in crunching huge data and cannot provide the crunched outcome in time. Machines capacity and Speed depends on the Moore's law and limit of speed of Light, hence there is little one could do to increase individual processing machines capacity. Mostly now a days multicore processors are only produced instead of trying to increase the data crunching capacity of individual processors. Mutlicore processors are like an individual having more number of brains, each of which could crunch data independent of other and co-ordinate with each other at specific intervals or once the crunching complete or co-ordination happens in a master-slave, peer to peer strategy. Even this multi-core processing is not enough for crunching live data streams which requires multiple processors & huge capacity than a multi-core processors of today. hence it had resulted in networking of many processors through Internet that is what we call it as Cloud Computing (Computing with Inter networked machines and inter networking is so loosely coupled and it would vary dynamically) & Internet of Things (Inter networked devices to share data & communicate).

Data Management

When many people co-ordinate to maintain data, management & organization are vital to increase efficiency & reduces conflict and to avoid inconsistency. In Cloud Computing apart from people co-ordination, machine co-ordination to crunch data had become vital. ACID & BASE principles are governing principles of Data management for machines & CAP theorem gives a wholesome picture. Nowadays more preference is given for partitioning of work among various machines leaving consistency out of the picture. Leaving C or A or P depends upon the kind of data we are processing and hence choice of our Data Base Technology and Database Management system should depend upon our choice. Earlier the choice of Data Base technology was based on ACID and it could have been a great choice by then but in today's demand for Adaptive Intelligence, it is certainly not sufficient and we are in demand to make choices based on kind of data.

Partitioning of data for processing demands to look at Compression & Security. Data are usually checked for redundancy and are compressed & decompressed to increase store capacity and networking capacity. Security of Data is another aspect when networking & data sharing are inevitable, hence encrypting and decrypting do have to happen.