Our Snippets Of Knowledge Weekly Blog Series
Databricks has always been a powerful distributed compute engine, sitting on top of Apache Spark. When it was introduced as a first class offering in Azure, it was a turning point in the creation of big data analytics solutions in the Microsoft cloud platform. Previous offerings such as Azure HD Insight and Azure Data Lake Analytics quickly became overshadowed by the Databricks Runtime and versatile Notebook development experience.
From a deployed Azure workspace, Databricks offers the flexible provisioning of autoscaling clusters, avoiding any complex configuration, with further options for single user and single node compute resources.
For data engineering and advanced analytics Azure Databricks supports Python, Scala, R, Java, and Spark SQL as well as data science frameworks and libraries including TensorFlow, PyTorch, and Scikit-learn. All supported by the Unity Catalogue and Hive Metastore for entities created as Delta Lake tables. Also referred to as a Lakehouse.
To avoid cloud run costs, check out the Databricks Connect capability available locally as part of the Databricks extension for VSCode. Develop Notebooks on your laptop before deploying to full Azure based clusters.
See MS Learn for more information on this Resource here.
We hope you found this knowledge snippet helpful.
Check out all our posts in this series here.
Comments