Naslov
AXS: Making End-User Petascale Analyses Possible, Scalable, and Usable

Autori
Zečević, Petar ; Slater, C. T. ; Jurić, Mario ; Lončarić, Sven

Vrsta, podvrsta i kategorija rada
Radovi u zbornicima skupova, cjeloviti rad (in extenso), znanstveni

Izvornik
Astronomical Data Analysis Software and Systems XXVIII / Teuben, Peter J. ; Pound, Marc W. ; Thomas, Brian A. ; Warner, Elizabeth M. - College Park (MD) : Astronomical Society of the Pacific (ASP), 2019, 401-404

ISBN
978-1-58381-933-3

Skup
Astronomical Data Analysis Software and Systems XXVIII

Mjesto i datum
College Park (MD), Sjedinjene Američke Države, 11.11.2018. - 15.11.2018

Vrsta sudjelovanja
Predavanje

Vrsta recenzije
Međunarodna recenzija

Ključne riječi
astronomy ; data analysis

Sažetak
We introduce AXS (Astronomy eXtensions for Spark), a scalable open-source astronomical data analysis framework built on Apache Spark, a state-of-the-art industry-standard engine for big data processing. In the age when the most challenging questions of the day demand repeated, complex processing of large information-rich tabular datasets, scalable and stable tools that are easy to use by domain practitioners are crucial. Building on capabilities present in Spark, AXS enables querying and analyzing almost arbitrarily large astronomical catalogs using familiar Python/AstroPy concepts, DataFrame APIs, and SQL statements. AXS supports complex analysis workflows with astronomy-specific operations such as spatial selection or on-line cross-matching. Special attention has been given to usability, from conda packaging to enabling ready-to- use cloud deployments. AXS is regularly used within the University of Washington's DIRAC Institute, enabling the analysis of ZTF (Zwicky Transient Facility) and other datasets. As an example, AXS is able to cross-match Gaia DR2 (1.7 billion rows) and SDSS (710 million rows) in 25 seconds, with the data of interest (photometry) being passed to Python routines for further processing. Here, we will present current AXS capabilities, give an overview of future plans, and discuss some implications to analysis of LSST and similarly sized datasets. The long-term goal of AXS is to enable petascale catalog and stream analyses by individual researchers and groups.

Izvorni jezik
Engleski

Znanstvena područja
Računarstvo

POVEZANOST RADA