Monday, October 17, 2016

#FailOps

Many years ago, Oleg Vishnepolsky and I worked together at IBM's T. J. Watson Research Center on the OS/2 1.1 port of the TCP/IP protocol stack.  Recently, I had a conversation with Oleg, who is currently the CTO at The Daily Mail Online, about DevOps and how it seems companies miss the whole point of what DevOps promotes. 

During our conversation (via email), it came out that he was of the opinion that DevOps is not a successful paradigm to follow and asked me for my thoughts on the matter.  I decided (with his permission) to publish the conversation because it seems that a lot of people are of a similar mind when it comes to DevOps (just as there were similar attitudes when the OGC released ITIL in the late 1980's), and while I do agree that many companies are not seeing the benefits of DevOps that it is supposed to bring to an organization, I don't believe that the problem is with the paradigm.  Hopefully, my answers to Oleg will not only clarify why DevOps is important but will also underscore why care must be taken to ensure that it is embraced correctly in order to fully realize the potential benefits that it can bring to your organization.



Oleg,

Here's my take on your blog entry, which I just read: DevOps is truly nothing new, like ITIL was nothing new either. Both were simply a formal means of describing things that were occurring for many years already upon their release. What ITIL and, to a lesser extent, DevOps does is describe the best practices that are needed to increase the chances of success. You note this "best practices" connection in your article, but I don't think you fully appreciate the value in doing so.

The biggest "problem" here (and I use that term loosely) is that you seem to have little experience working with organizations where the "software development pipeline" is so poorly implemented that it causes more problems than the extra value that Agile is supposed to bring to the organization. (I jokingly call this "frAgile".) I have, however, run into these types of organizations on a high enough frequency that it is scary.

One thing that is clear about DevOps, which was similar in the days when ITIL was initially released in 1988 by the OGC: it is not meant to be a "you must do it this way" process. Instead it is meant to describe ways that application development and operations can work together. The biggest benefit of this is by highlighting what many people like you and me with the appropriate level of experience already know: development is all about writing code while rarely giving the appropriate amount of consideration to quality, while operations is all about ensuring stability in the production environments, meaning they aren't too happy about software releases because that means constant change, which is counter to the concept of ensuring stability.

This is a critical point because although application development organizations are frequently measured by the number of defects, true, getting applications out the door on time trumps everything including software quality. I can recall one such time during my days in application development when development was halted for an entire month so that we (a team of 40+ developers) could all do testing (like a staff aug for the QA group). That sounds great until you realize that we only tested for and fixed severity 1 and 2 defects and ignored severity 3 and 4, which dwarfed the former categories of defects in terms of quantity by an order of magnitude.

So DevOps screams at developers that operations wants stability so they need to make quality a priority. At the same time, it screams at operations that developers are trying to release things quickly to get new business functionality in the hands of the end users. That last point cannot be understated. In March 2014, Gartner told a group of CIOs the following: "CEOs must focus on leading their organizations to think like and become more like ‘tech’ companies, because within a few years, digital business capabilities will dominate every industry. Urgent action is needed because first-mover advantage is common in digital business, and fast followers must be very fast."

To put that more succinctly, "constantly innovate or rapidly become irrelevant."

So DevOps is not only a buzzword but it is, in a very real sense, a "must have" since without this coordation between the two organizations, a company will lack the velocity to do exactly what Mark Raskino at Gartner said in that quote, above. Consider this for a few moments: in May 2011, Amazon was noted as releasing something to production every 12 seconds. One of my customers, Bet365, releases to production every 4 minutes. And when I was at a previous employer, one of our large financial services customers in EMEA had 60,000 production releases every month. This need to scale to massive throughput from development to production is here now, and so the coordination proscribed by the DevOps movement is needed now as well. 


I hope this helps.

Kind regards,
Larry




Thank you. What i see happening though in the industry is that devops is taken as a mandate to let developers loose on production environment.

Oleg



Oleg,

Well, that's kinda what I meant by "poorly implemented." That's similar to when people thought (and, unfortunately, frequently still think) that Agile means "release more quickly." In the latter, the part they miss is "release more quickly without incurring more operational risk." (To wit, the official mantra of Agile is to "fail fast" but that ultimately has the goal of allowing an organization to release more quickly with less risk because bugs are identified earlier in the SDLC and fixed there when the amount of effort to remediate is significantly less.) 

Organizations that take the word "DevOps" and use that as a shield to justify poor decisions related to operational impact are headed for failure. But that's not the fault of the DevOps movement and is instead the fault of the leadership at the company that allows this to happen.

Kind regards,
Larry



Your comments on this subject are highly encouraged because it's important to have thoughtful discussions about this and how you and your company can reap the greatest benefit from having fully and correctly embraced the DevOps mindset.

Thursday, September 8, 2016

Agility Has Its Limits

Automation and orchestration have given us free two-day shipping via Amazon Prime; next day shipping via the major postal companies; delivery via Uber or drone; and so on. These are all concepts that we are either intimately familiar with as consumers of these services or at least are cognizant of them because of news reports, TV commercials or YouTube videos.

Convenience and Digital Transformation

The value of convenience cannot be overstated.  However, there will always be times when these methods of delivery will not be enough.  For example, a freak snow blizzard in my area wasn't a big deal for me since I have snow removal equipment.  But as I drove past kids sledding in the field at the end of the street I wanted to buy a sled for my children also.  Next day shipping wouldn't have satisfied my need to have this now. Of course, Uber or drones could have delivered such an item theoretically.  But unless you live in NYC or a similarly sized metropolitan area and the goods purchased were something that either service would deliver then you're still forced to get in your car and drive to the nearest Walmart.

Measuring your way to revenue
It is during times like these that we must keep in the proper perspective the benefit that agility from a technology perspective gives a company.  For online shopping, the ability to deliver new functionality on an as-needed basis is critical to staying ahead of the competition.  For example, Amazon has a production deployment once every 25 seconds on average. For the brick and mortar companies, however, the need for agility is somewhat different since the quality of the end user experience isn't dictated by how good the Point of Sale software looks.  The focus in this situation is in the quality of the in store inventory, the cleanliness of the store, the helpfulness of the staff, etc.

Regardless of what defines "a quality experience" for a business, it can be measured.  And if it can be measured, then the measurements can be gathered and analyzed.  And if the measurements can be gathered and analyzed, then there is a need for the gathering and analysis.  This need exists whether you are a brick and mortar business, an online business, or a hybrid of the two.  The results can be stunning.
  • Take Netflix, for example.  Netflix does real time analysis of their customers' viewing habits to suggest new movies they may want to watch.  This starts with an ETL process that gathers the data and normalizes it into a format that is more easily digestible by the balance of the algorithmic analysis engine that results in recommendations.

  • Or consider LinkedIn.  LinkedIn does real time analysis of their users' network to suggest people that they may know based on degrees of separation.  This starts with several ETL processes that run on Hadoop, Teradata, and other data sources to enable this functionality to work.

  • Finally, look at AMC Theatres.  AMC does real time analysis of ticket and concession sales to automate their social media feeds.  This starts with an ETL process on Hadoop to gather, normalize, and process information to post updates to various social media sites with no user intervention.
Digital Growth

Anything that defines "a quality experience" for a business is quantifiable and should be measured. Why not? After all, this need exists whether you are a brick and mortar business, an online business, or a hybrid of the two like AMC. The process of measuring, gathering, analyzing and reporting unearths what is and is not working so success can be capitalized on in real time.

As you can see from the examples above, the results can be stunning. But the quantity of data that these companies generate means none of this would be possible without data automation, which is geared specifically at real-time execution of jobs to gather and then process large amounts of data in real-time.

Orchestration Backbone

There are a number of platforms to process large amounts of data (Hadoop, Hortonworks, and Cloudera, to name a few). However, the orchestration to coordinate the activities around gathering and reporting, is critical. Without it, you would be stuck with the task of building a custom set of solutions to do these two very important activities. It is this orchestration that distinguishes a large data processing platform like Hadoop from a data automation platform that enables you to effect a change in your company based on what the data contains.

The other thing all the above examples have in common? They all chose Automic to provide the orchestration capabilities to allow them to transform their businesses into something that responds in real time to the needs of their customers and users.

As Vijay Aruswamy, Big Data Operations at LinkedIn, put it, “We're using Automic to automate data transformation on Hadoop, Teradata, User Input and External sources of information. It supports our data driven products like ‘People You May Know’ or ‘Jobs You May Be Interested In’.”

Automation, when properly harnessed, means the difference between being able to continuously innovate and rapidly becoming irrelevant. And though large scale adoption of automation is not an endeavor to be undertaken on a whim, examples of data automation like those listed above show the outstanding benefits that come as a result of this investment that you make for your company’s future.

Wednesday, March 9, 2016

Digital Darwinism

"In 2014, CEOs must focus on leading their organizations to think like and become more like ‘tech’ companies, because within a few years, digital business capabilities will dominate every industry. Urgent action is needed because first-mover advantage is common in digital business, and fast followers must be very fast."- Gartner, "CEO Resolutions for 2014 - Time to Act on Digital Business," published in March 2014.

A more succinct way to state the above is "continuously evolve or rapidly become irrelevant."

You have to look no further to see the effects of gross negligence to adapt than the U.S. Post Office. Last year, I got divorced, sold the house, and moved to a new location.  My ex-wife moved to a different location.  And although we both created mail forwarding requests, she kept getting mail that was addressed for me. She moved over 4 months ago, and even today she is still getting my mail when it is mailed to my former address.

When I asked the USPS about it, I was told this:  only the first initial of the first name and the last name are used by the USPS for processing mail forwarding at a specific address.  Since my ex-wife has the same first initial as I do, that explained why she was receiving my mail (and that of my deceased father, who also shares this characteristic). I was in shock then, and it still leaves me shaking my head when I consider that statement because there is no need to restrict the amount of data that is used when considering whether or not a name on an envelope matches an entry in a database.

To describe a specific alternative, the concept of fuzzy searches have been around since my days at IBM in the 1980's.  The linked article describes how it is used, but the underlying idea is to remove all vowels, replace certain digraphs with functional equivalents, replace duplicate consonants with singular instances, and replace certain consonants with their functional equivalents. For example, "Larry Salomon" may become "Lry Slmn"; "CEOs must focus on leading" may become "C mst fks n ldng"; etc.

Combine this with a conversion to uppercase / lowercase, and then apply an MD5 hashing algorithm (a 16-byte value) to the result to get a (relatively) unique fingerprint for the name.  Couple this with the ridiculously cheap cost of disk storage per gigabyte and you have the ability to support whatever mail forwarding needs could possibly exist.

To put this in perspective, as of 2013 there were 7.125 billion people living in the world.  Not excluding a single person, even infants, if there were a single mail forwarding request for every one of those people, the storage requirements would be:  7,125,000,000 x 16 bytes per hash / (1024 x 1024 x 1024 bytes per terabyte) = 106 terabytes of storage. Contrast this with Facebook, which (as of 2012) collects 500 terabytes of data on its users per day. And while no single cause can fully explain the dire financial straits of the USPS, symptoms like this do not inspire confidence that the leadership has any clue how to meet the ever evolving needs of its customers.

"Adapt or die."  Live by this mantra, and you will continue to stay ahead of your competition.