Author: hosurfamily

Google Cloud Platform: Reference architecture for Data Warehouse

It had been a great journey to learn and understand Google Cloud Platform also called as GCP. Among the top cloud providers, Google seemed to have nailed the cloud technology very well. As I was exploring the services, I got some Proof of concepts working and some reference architecture defined. One of the reference architecture defined is for the Data Warehouse. Our use case is creating an analytics dashboard and reporting platform for the internal and external users. The solution requires three standard serverless services from Google Cloud Platform:

  1. Cloud Dataflow – fully managed service from Google for streaming, batch-processing and enriching the data ingested into various storage options in Google
  2. BigQuery – serverless, highly-scalable, cloud Data Warehouse with a built-in In-memory BI Engine and Machine learning capabilities
  3. DataStudio – serverless BI engine, highly-scalable with flexible suite of data analytics tool

In this case, let us assume there are four sources Source 1, 2 and 3 residing within US and Source 4 residing outside of US requiring some data separation. Cloud Dataflow powered by Apache Beam can be utilized to stream or batch ingest the data from the sources. We can develop a pipeline for one source and leverage it for other sources. Dataflow can be created with Java or Python. Once we have the data, the industry practice is to have a Data Lake in BigQuery to store the raw data for in-depth analytics or run machine learning algorithms. From there dimensional modeling may or may not be required depending on the nature of the end output. For the benefit of clarity BigQuery is shown in both the Data Lake and Data Warehouse but the structures may reside as one. Both Google DataStudio which is now called Google Cloud BI Solution and Tableau are visualization solutions. Either of these could be extended to support the goal of the organization. This provides a high level overview for a Data Warehousing reference architecture in the Google Cloud Platform.

How to manage Mobile Apps?

Mobile apps have a tendency to proliferate and get out of control. We are all guilty of downloading the apps if one of our friends or family says it is good. What we will not know is when they delete them. In addition, the app companies promote their apps through various channels like TV, YouTube etc., Somehow, these apps clutter our phones and get out of control. If we run out space, then we begin deleting them. In the past our only opportunity to clean up is when we upgraded the phone. For few years, we just started fresh and began adding the apps. This reduced the clutter to some extent but nowadays there is software to port all the apps to the new phone. Besides phone memory has become cheaper and there is no need for deleting the apps. Does this mean we just keep on hoarding the apps? No, all these apps track your activities, listen to your phone calls and sell your data. God knows what these providers do with that information. For an interesting perspective check out Jeffrey Deaver’s Novel Broken Window in which the villain uses the data aggregated by the data aggregation company to stalk his victims.

We can separate the apps into three categories:

  1. Must have – These are the banking apps, email apps, photo apps and productivity apps
  2. May be – social apps, I would keep one chat app like whatsapp, delete messenger, Instagram etc
  3. Not used anymore – game apps that may been used when popular e.g., Words with friends, Pokemon Go (trust me I found this few months ago)

These need to be checked on a monthly basis to see if there is a need for the app. It would take few minutes and will save your privacy. With that said, here are my top 10 apps.

  1. Flipboard – latest news
  2. Feedly – create your own feed of information
  3. Medium – tech articles
  4. Sticher – Only podcast app you need
  5. Google Keep – Reminder and notes app
  6. Bored Panda – potpourri of information
  7. Google arts & culture – interactively walk the museums in 3-D
  8. Simple Habit – Meditation app
  9. Be Focused – “Pomodaro” app, great for productivity
  10. Personal Capital – track your networth

My 2018 Book Recommendation

Every once in a while we come across an interesting book that questions your assumptions, derails your beliefs and yet in a profound way affects your thoughts. Last year I picked up this innocuous looking, although bold book “Courage to be Disliked – The Japanese Phenomenon that shows you How to change your life and achieve Real Happiness” by Ichiro Kishimi and Fumitake Kaga. The title itself is quite a mouthful but the book is a light read of around 288 pages written in a conversational manner between a philosopher and a student. Whole book talks about Alfred Adler’s Teleology (study of the purpose of a given phenomenon rather than cause as believed by Freud)

The important thing is not what one is born with but what use one makes of that equipment – Adler

While the Freudean Idea believes that a person’s past psychic wounds (traumas) cause present unhappiness, Adler proposes that we don’t suffer from traumas but we make out of them whatever suits our purposes. We lack courage to change our lives and find excuses in our past. No matter what has occurred in your life up to this point, it should have no bearing at all on how you live from now on. Attributes all problems to interpersonal relationship problems.
Here are some other ideas:
  • It’s enough to move forward and no need to compare with others. Try to strive for your better self than trying to “keep up with the joneses”
  • Human beings are not the same but are equal. Concept of promoting horizontal hierarchies (equal) rather than vertical hierarchies (hierarchical)
  • One must not seek recognition which promotes vertical relationships. When one person praises another, the goal is to manipulate someone who has less ability than you not due to gratitude or respect. Encouragement enables one to complete their tasks.
  • Identify your tasks from others’ tasks and ensure you don’t trample on others’ tasks and don’t let others trample on yours
  • Happiness is a result of feeling of contribution to the society, i.e., by doing your job that indirectly helps others or doing social service or anything else
  • Life is always simple not something one needs to get too serious about
  • Greatest life-lie of all is not to live here and now
  • People are comrades, and the world is a wonderful place
  • Do not look at the past, and do not look at the future, live in the earnest
  • If “I” change, the world will change
  • and many others ….
I had to read twice to make sense out of this simple narrative. It certainly helped me put things in perspective and highly encourage you to read it.

Leadership in exceptional situations

**Spoiler alert for Episode 4, Season 7**

Being an avid fan of Game of Thrones, I constantly review the situations and how it relates to leadership and strategy. Daenerys Targaryen took a decision and attacked the Lannister army against the counsel of Tyrion and Varys. Is this the right decision? Let us review the context in which the decision was made. Euron attacked Greyjoy fleet and the took away the Sand Snakes destroying her entire naval fleet. Jaime attacked High Garden and took the Gold and food back to King’s landing. The victory of Grey Worm attacking Casterly Rock was useless as the Lannisters removed everything from the castle and left skeletal army to defend it.

She could have listened to Tyrion and followed the approach he recommended. This would have been a conservative move. She was seething with rage and had to take an action. Here is where her leadership shines. She listened to the wise men but not satisfied with the strategy and came up with her own plan. As leaders we encounter such situations where we listen to the team for guidance but make our own decisions. This is critically important to establish yourself as a leader.

You may encounter a setback while executing the decision similar to how the dragon gets injured by the arrow. We need to anticipate such failures but they need to be few and far in between, otherwise you risk getting fired from your position. While it is easy to conform to the decision of bosses, peers and team members, there are situations where you may have to make your own decisions. Daenerys has altruistic intentions for ruling the seven kingdoms. As a result she listened to Tyrion and did not attack King’s landing and raze it to ground with her dragons. However she felt that such inaction has costed her naval fleet and food supplies from High Garden. Therefore she took the dragon and dothrakis and destroyed Lannister. Viewers got satisfaction in punishing the evil Lannisters.

As leaders, we need to get as much information about the situation through data, wise counsels and/or AI to evaluate but make decisions that would benefit the organization. With that said, in a mature organizations decisions are made collectively and stakeholders may disagree but commit to the common goal.

Share your thoughts below.

Why Domain Driven Design (DDD) is getting popular?

The concept of Domain Driven Design has been popularized by Eric Even’s book “Domain Driven Design: Tackling Complexity in the Heart of Software” in 2003. Not much attention was paid at that time since the industry was recovering from the “Dotcom Bubble” and the applications developed were not as complex as we see today. Fast forward fifteen years and our worlds have changed. Businesses has evolved and startups have disrupted traditional industries successfully.

At the core of Domain Driven Design, Eric describes three main premises:

  • Project’s focus is on the domain and domain logic
  • Complex designs are based on a model of domain
  • Continuous collaboration between technical and domain experts to iteratively refine the model to address domain problems

By following this philosophy, big projects are chunked out into smaller projects based on distinct domain components. These small projects are assigned to independent teams for completion. By dividing the project into business specific domain and domain components a modular system is built. These business functions in the domains are reusable and over a period of time, there will enough parts that could be assembled together to solve an entirely new business domain problem.

The DDD encourages continuous collaboration with the business domain owners which is not unlike any Agile project that has business/product owners. The difference is that in DDD, both technical and business teams use the same jargon while discussing the domain problem and build software components that will directly address the business need. An important point to note is that development is not driven by latest and greatest technology but by the business and the software development teams can adopt a new technology only when the business domain problem couldn’t be solved using existing software.

In summary, DDD is not a new architectural pattern, but it affects how architectural decisions are made and the functions within the teams. Overall organizational leadership support, both business and technology, is essential since adoption of DDD may increase the cost initially but will taper off as more and more modular components are built.

College Decision

I clearly remember the day when my dad enrolled me in college and left. Education in India and for that matter specifically in Tamil Nadu is very straightforward. You work hard to get good grades and take the entrance exam. Your selection to the top colleges is based on the rankings which is a combination of the two. If you are on top, there are no doubts about getting into the top ranking colleges. Very simple and efficient.

Whereas, the higher education in the US is very complex and the choices are limitless. For a senior, especially one with 3.96 GPA, 12 APs above 4, working a part-time job, participating in National honor society by volunteering hours at the community, doing a senior project in robotics, achieved brown belt in Karate, likes mainstream rappers as well as obscure ones and of course crazy about the NFL, NBA team stats of the players, the choices are even more complicated.

His dream was to become a Mechanical Engineer and complete the MBA program to become an entrepreneur. I am glad he was so specific about his objectives considering the fact that, I wanted to become a Civil Engineer but within few weeks I changed to Electrical and Electronics, now I work in IT.

He was admitted to University of California, Berkeley and Barrett Honors College in Arizona State University. Both very different colleges with very different offerings. A quick comparison will tell you that Berkeley far surpasses Arizona State University and there should not even be a decision involved here to pick the right choice. This is what meets everybody’s eyes but my son chose Barrett in ASU for the reasons below. I am writing this blog post for anyone who are debating on the choices.

The comparison cited above compares the bigger Arizona State University with 70,000 students with UC Berkeley and does not take into consideration the Barrett College within ASU. Barrett is the honors college with around 2000 freshman and was started to capture and retain the extraordinary student talent within Arizona. Berkeley does not have a comparable college and their take is that everybody here is  an exceptional achiever. This is good and bad. Good because it is a level playing field for all the resources available and bad because the students have to work very very hard to the point of burning out to compete with each other.

With Barrett honors, the students are given priority student advisement and are provided with world class housing and dining options which are otherwise not available to the regular students. The Barrett program is isolated from the University and has its own boundaries. The housing is spectacular when compared to the old dilapidated structures in Berkeley (they really are and we were surprised). These Honors students are given opportunities to participate in Research during their undergraduate year wheres in Berkeley it would be next to impossible to get access to these opportunities. Another distinguishing factor was that the entire faculty in Barrett are dedicated to supporting the limited number of freshman when compared to the large number in Berkeley.

With regards to the machine shop, we were made aware in Berkeley that only Junior students are provided access and can tinker things. With Barrett the students will have the opportunities in Freshman when they can sign up for EPICS program that will make them tinker to solve a community problem.

Being the resident of the state and for his academic excellence he got a full tuition ride in Barrett with a Presidential scholarship. It may be a different situation if the college costs were comparable to Barrett when the decision would have been equalized. However, our situation was unique and will be different for different individuals.

Though part of me was siding with Berkeley, my son made the decision easier when he signed up for the orientation and enrolled himself in Barrett. According to him, he would like to complete the undergraduate in Barrett and pursue his graduate studies in out-of-state university although Barrett has a 4+1 program in which he can earn his graduation in 5 years. He said, “Dad I would rather be a big fish in small pond rather than a small fish in a big pond” and I was speechless.

Fast forward four years, he is out of ASU and came out with flying colors. We don’t doubt his decision and he is pursuing his dreams.

New Book published

For many years, it has been nagging me to write a book to help new comers from India to orient themselves in USA. There are many books available that are acting as travel guides but none of them get into the nitty, gritty details about day to day life in US. This is my attempt to help folks coming from India or for that matter any other country to review and be prepared for the ordeal. Having lived and worked here for the past 17 years, it wasn’t easy. However with friends and co-workers I was able to navigate and come out what I call to be a reasonably successful adjustment in the country. It should be out in Amazon within the next few days.

Hacking the sirens!

“If security were all that mattered, computers would never be turned on, let alone hooked into a network with literally millions of potential intruders”
― Dan Farmer

Alarm display

Imagine waking up in the morning and commuting to work when you hear the sirens all over the city. In panic, you pull over and try to search social media for clues and find nothing. Your survival instinct kicks in and start calling people who can provide no additional information. This is the hypothetical situation that is seen and observed during calamities or dooms day Hollywood Movies. Yet this became a reality today in Dallas when hackers pulled off a stunt activating all the sirens in the city. The hack was very sophisticated that the city officials are scratching their heads to understand how the system was breached whether it is an insider or outsider job. While we wait for the reports, one thing is for certain that the hackers continue to push their limits in everything we do.