Naslov
PySpark and RDKit: Moving towards Big Data in Cheminformatics

Autori
Lovrić, Mario ; Molero, José Manuel ; Kern, Roman

Izvornik
Molecular Informatics (1868-1743) 38 (2019), 6; 1800082, 5

Vrsta, podvrsta i kategorija rada
Radovi u časopisima, ostalo, ostalo

Ključne riječi
QSAR ; Hadoop ; Apache Spark ; Python ; pandas

Sažetak
The authors present an implementation of the cheminformatics toolkit RDKit in a distributed computing environment, Apache Hadoop. Together with the Apache Spark analytics engine, wrapped by PySpark, resources from commodity scalable hardware can be employed for cheminformatic calculations and query operations with basic knowledge in Python programming and understanding of the resilient distributed datasets (RDD). Three use cases of cheminfomatical computing in Spark on the Hadoop cluster are presented ; querying substructures, calculating fingerprint similarity and calculating molecular descriptors. The source code for the PySpark-RDKit implementation is provided. The use cases showed that Spark provides a reasonable scalability depending on the use case and can be a suitable choice for datasets too big to be processed with current low-end workstations.

Izvorni jezik
Engleski

Znanstvena područja
Kemija, Interdisciplinarne prirodne znanosti

POVEZANOST RADA