Software development has moved to so-called agentic development. We no longer use AI just as a smarter code completion, but we run agents that go through the project on their own, modify files and iteratively fix the code based on error messages.
Whether you prefer writing queries in Spark SQL or you are more used to calling functions via the Dataframe API, it’s only a matter of time before you run into a classic problem.
Apache Spark 4.0 introduces a significant enhancement to its SQL capabilities: experimental support for procedural language.
Working with data using Apache Spark is possible in several ways. If you come more from a software development background, you will likely lean towards using the Dataframe API
Apache Spark, as one of the main representatives of distributed computing systems, supports several formats for reading and writing. Probably the simplest of these is the delimited text format
A key concept of object-oriented programming (OOP) is inheritance, which allows grouping properties and behaviors into a hierarchy. This way, new objects can automatically acquire functionality (methods, variables, constants, etc.) from their ancestor.
Object-oriented programming is not exclusive to higher-level languages; a very similar principle can be achieved in PL/SQL using the special OBJECT type. A class, or rather an object, is defined similarly to a package, i.e., with a specification and a body.
Savepoints are a mechanism within a PL/SQL block that allows you to divide the transaction being executed into partial units, or milestones. You can then easily return to these milestones without losing the entire transaction.
In older versions, limiting sorted data by the number of rows had to be implemented using a nested query. This first sorted the table, and only then was it possible to filter by count.
The option to define a PL/SQL function or procedure inside an SQL query might seem strange at first glance, but it has one significant advantage – the database does not need to switch context, which in certain cases leads to a massive increase in performance.
Recent comments
-
TechSavvySam
You've explained this better than anyone else I've…
Add a comment...