How do you hide big data?
I’m no longer surprised when I hear someone talk about a big data project that isn’t on the official radar of the IT department, but I sure used to be. How in the world does someone hide a big data project? That’s a bit like hiding a tank in your back yard, isn’t it?
Here are some things that are happening “behind the scenes” so to speak that are causing big data seeds to be planted all over the place, without the gardener even knowing it…
- Shadow IT is alive and well… Shadow IT is created by the business when they get the impression that the IT department proper is too difficult (or simply too busy) to work with. They will go out and hire technical folks on their own and ask them to create solutions that (initially) live outside of IT.
- The Cloud… While hosting has been around for quite a long time, the ability to take out your credit card and fire up multiple instances to house your big data, hasn’t. That phenomena has led to many projects being spun-up off premise, far away from the confines of the local data center.
- Small starts… Rome wasn’t built in a day, and neither are big data sets. Teams are starting out with projects that they believe will be large one day, but in the initial phase have a relatively low amount of data. This allows them to set up clusters with just two or three machines just to prove the concept. Often, these prototypes slyly make their way into staging, then production, all without going through the regular IT process.
- Open source… the products are only a download away, which means developers and operations people can be up and running immediately. No purchase orders or budget approvals required. They just download the open-source databases and away they go.
For many developers and business people, these things are great. They can get started fast and small, and at very low cost, without any encumbrances of rules and standards.
For IT people, these things are scary. They know that sooner or later, they are going to get left holding the bag and will have to support these systems from a production standpoint.
The market has spoken and it seems these big data systems are here to stay. Therefore, the best things both sides can do is simply communicate. I’m finding in my conversations that when that happens, IT tends to be good about not slowing down the prototypes, and the developers and business people are more apt to having planning discussions about a proper transition into IT when the time is right. In the end, that little bit of up-front communication and solve a lot of long-term headaches.