What emerging technology are you most excited about and why?
Personally, real time data technologies represent the most
exciting technology frontier. There are many offerings from paid
providers such as Amazon, Microsoft, and Google. Looking under the hood
of these services will inevitably lead to the most exciting place of
them all, the Apache Project. Apache is an Open-Source Software
Foundation committed to furthering the advancement of breakthrough and
revolutionizing technologies. They are home to projects in a variety of
areas and scopes, including real time data processing.
Several Apache projects, some more well-known than others, have
been integrated real time data processing such as: Kafka, Lucene,
Synapse, Spark, Beam, NiFi, Storm, Flink, Pulsar, and many more.
Underneath the eye-catching codenames, these projects represent the
bleeding edge of the technological frontier. Apache projects are
typically born in organizations or companies as a byproduct of
development or research, these projects are then donated to the Apache
Foundation to grow and mature due to the vast resources the Foundation
can attract as an open-source community. For example, NiFi started life
at the NSA, Spark from the University of California Berkley, Superset
donated by Airbnb.
Kafka facilitates software applications to broadcasting events
in real time, essentially allowing any application to broadcast “hey I
just did this”, “Oh this just came in”, “I did this other thing too” in
messages which can be consumed by another application for use as it sees
fit. NiFi is a platform enabling data movement at a real-time pace, for
example the “hey I just did this” message could be sent to system B,
while routing “I did this other thing too” to system C and disposing the
“Oh this just came in”. Lucene is a library able to handle full text
searching within documents at an incredible pace and has redefined how
the world searches for things. Lucene is at the core of another
open-source technology, Elasticsearch. Elasticsearch handles ingestion,
analysis, searching of data (in the form of JSON documents) in
real-time. Elastic can be used to monitor things like server traffic in
real time, searching for and presenting anomalies for assessment or
intervention.
Real time data is transforming the way the world works, all by
eliminating the window of time between when data is created and when
data is actionable.