Category Archives: Real Time

From Big Data To Small Data In Real Time – What does this 1 per million part score card mean to you?

There are many takeaways. 

Seven powerful metrics that bring down a specific type of big data opportunity (security) into small data.

A 1 per million part identification system bringing together different data types in real time, a big data opportunity.

Behavioral interpretation and social analytics are still key to make sense of data, to quickly bring big data into interpretable and usable small data

I am working on a weighting system that is what makes this a very effective, better than six sigma identification system – this is my intellectual property

The following is a difficult thing for me to discuss as this indicates an unsettling prospect of

  • how resourceful organizations are going to be watching you and me going forward with big data, 
  • how privacy is going to be a challenge to maintain, 
  • how we are going to loose our moral superiority.  In difficult times like this, after Boston Marathon, it is important we have a tool like this, though. 

This only makes why discussions about Type I and Type II errors are becoming more and more important and, added to that, how the idea of bias (pre-conceived notions) are going to undermine real intelligence.

The following, based on my initial estimates, is a 1 per million part identification system, better than six sigma, using big data and a weighting system that would help identify that 1 per million part measure of a dangerous person who is lost in our day to day hurry-burry life of innocence, dream, and celebration of love and accomplishment.


– Sudden changes in behavior or performance or relationships

Their usual metric of performance will be lost.  For a student, he/she will get outlying set of grades from his normal performance or performance evaluation or angry exchanges

Loose commonly known best friends or other gender relationships

– Sudden changes in the watchful eyes of organization or people; will travel to not so common places and will acquire new relationships who are in turn in the watchful eyes of security organizations.  At least the chatter inside the security organization has just got elevated or elevated chatter comes and goes, but not able to stick to a well defined resolution of not dropping the ball.  Resolving clearly does not mean putting people in jail, but have an officer report on the latest activities, log in the details so that it flows through the right people for right action, making sure the watchful eyes not sleeping. 

– Unprecedented access changes happening around one’s neighborhood, social relationships, and one’s lifestyle interests, that would provide opportunities for dark side to show up its head

– Firearms, crude bomb, illegal activities blip on the intelligence radar or the correlated words of “bomb” or “firearms” or “mass danger” materials are popping up in the radar

– physical (becoming a post teenager – things get hardened around this time) or emotional changes at home (death or separation) or with closely related people (friends lost)

– New buying/shopping activities of apparently looking unrelated items; this will be a second or third blip, almost always, if they are innocent looking items but used as an aid to complete the intended action

– sleep patterns, telephone call patterns, internet information access patterns, even visiting one’s own home are changing unreliably, but with consistency one’s change started

I say, these are seven metrics of highly dangerous people who need help badly.  What caught my attention is that our security agencies were so close to the …. and yet the tragedy happened.

Of course, it takes a lot of fine intelligence to be careful about Type I and Type II errors that is also respectful of privacy of citizens.

My prayers are with the innocent victims of all ages, an innocent child who cared for kindness, a couple starting their dream life, and an accomplished elderly who did not give up running a marathon at age 78.

From Data Monster & Insight Monster