NETFLIX AND… STARGAZING?!?
Big Data and the Next Generation of Large Scale Astronomical Surveys.
When it comes to binge watching your favorite show on a streaming platform, or performing the next generation of large scale astronomical surveys, the one thing they both have in common is data, and lots of it! Astronomical surveys such as PAN-STAARS, the Zwicky Transit Facility (ZTF), and the upcoming Large Scale Synoptic Survey Telescope (LSST), monitor the sky for changes in our local Universe and beyond, and in sometimes covering vast areas of sky in a given night. These types of astronomical surveys can observe changes in hundreds of thousands, or potentially millions of objects per night. With such a high cadence across multiple wavelengths, these surveys can generate volumes of data ranging from a few Gigabytes, to a many Terabytes per night. With this overwhelming volume of data, how do we find objects of interest such as supernovae, transiting exoplanets, near-Earth Asteroids, or other exotic astrophysical phenomena, against the background noise? How do we cope with the sheer volume of detections that such surveys produce, and how do we store, process and make available, all this data? Solving this is one of the most pressing challenges in astronomy, in the era of 'Big Data'.
We will explore how we solve these problems within the context of the ZTF survey, and consider what we can learn from other in the big data business, such as streaming platforms like Netflix, cloud computing services like Amazon S3. We will also explore virtualizing our infrastructure can help make it as efficient and cost-effective as possible.
Christopher Phillips is an astrophysicist and research scientist working on the Zwicky Transit Facility, at the University of Washington. His research interests lie in the realm of time-domain astronomy and the detection and classification of transient astrophysical objects. His previous research includes using statistical methods to identify exoplanet candidates in Kepler space telescope data. Christopher is an 'Imiloa Astronomy Center alumni, and a member of the Maunakea 'ohana.