Skip to content

Databricks Dialect Configuration

The Databricks Dialect Configuration extends the base converter config and allows you to configure additional aspects of the conversion process. The following sections describe the available configuration classes and their fields.

Databricks Converter Config

Field Description Default Value Localizable Environment Variable
default_output_extension Default output file extension. .py True
render_unsupported_output_as_comment If True converter will render whatever it was able to convert for unsupported source chunk as a comment. The output by definition will not be correct and may not even resemble valid code, but may be useful for creating user templates or reporting feature requests.

This is especially true for large statements/expressions where only a small part is not supported. In this case the output may be very close to a fully converted one.

Alchemist will determine a reasonable level at which to render the output. This is the same level as for render_unsupported_source_as_comment.
True True
user_template_paths List of folders with user templates to make available for conversion. Empty list False
template_configs List of template configs. DBRSASTemplateConfig False
template_tag_filter Filter config for template tags to inclusion/exclusion from matching. StringFilterConfig True
node_filter Filter config for node inclusion/exclusion from rendering. NodeFilterConfig True
use_runtime If True converter will use runtime user-defined functions where appropriate.

Be aware, that setting this to False may reduce the amount of automatically converted code, since for some of the constructs there may not be a non-runtime static inline version.
True True ALC_USE_RUNTIME
custom_udf_name_mapping Custom name mapping for runtime functions.

This allows to use custom names for runtime functions. For the name of specific functions, consult target dialect documentation.
Empty dict True
conversion_comment_verbosity_level Verbosity level for conversion comments (code, todo, warning, debug).

Verbosity levels:

- code: outputs only regular code comments retained from the source or considered part of the output.

- todo: outputs code and todo comments; todo comments are used for output that has to be adjusted manually.

- warning: outputs code, todo and warning comments; warning comments are used for potentially invalid code.

- debug: outputs all comments, including developer warnings; debug comments are used for code that is unlikely to be invalid.
todo True
conversion_mode Conversion mode (normal, strict, lax).

Changes code generation and how the comment verbosity level is handled:

- NORMAL: The default mode. Balances correctness & readability,
by allowing some heuristics about the common cases when
achieving 100% match would generate overly verbose and complex
code, while still allowing short constructs that ensure
correctness.
- STRICT: Prioritizes correctness over readability, striving to
mark anything that is potentially not 100% correct in all
scenarios as a TODO item and reducing heuristics to a minimum.
- LAX: Prioritizes readability over correctness, assuming the best
case scenario and avoiding generating additiona expressions
that would be needed to handle edge cases.

In addition to that, the verbosity level of conversion comments is adjusted based on the mode: - In strict mode, the warning comment is treated as todo, and debug is treated as warning. So more todo comments are generated. - In lax mode, the todo comment is treated as warning, and warning is treated as debug. Meaning no todo comments are generated at all.
normal True
llm Configuration for GenAI base conversion. LLMConfig False
spark_conf_ansi_enabled Whether to generate code that assumes ANSI SQL mode is enabled in Spark. True True
sas SAS to Spark conversion options. SparkSASConfig True
group_nodes_into_paragraphs Whether consecutive nodes of similar type should be grouped into a single notebook paragraph. True True
render_all_source_code Whether notebooks should include entire original SAS code before the converted code. True True
render_markdown_headers Whether notebooks should include Markdown cells with headers reflecting original SAS program structure. True True
file_path_map File path mapping. Empty dict True

Fila Path Map

File path mapping is used to convert source file location to a new cloud location. Result is always a posix path.

Mapping may specify a prefix part of the full path as it appears in the source, and how it should be converted to the target part of the path itself.

The longest matching prefix will be used. If no prefix matches, the original path will be used (which is probably not what you want).

The resulting path will be automatically converted to posix path.

Example:

  • for the path C:\User\any_local_dir\file.xlsx
  • the mapping can be {"C:\\User\\": "dbfs:/mnt/User/}
  • and the final path will be dbfs:/mnt/User/any_local_dir/file.xlsx

Databricks SAS Specific Converter Config

Field Description Default Value
year_cutoff SAS YEARCUTOFF option value (see docs) 40
libref_to_schema Mapping of SAS librefs to Spark schemas. Empty dictionary

Example

Here's an example of how you can define libref_to_schema in the configuration file:

converter:
  sas:
    libref_to_schema:
      libref1: spark_schema1
      libref2: "{spark_schema_var}"

In this example, libref1 will be converted to spark_schema1, and libref2 will be converted to {spark_schema_var}, assuming that it will be used in f-strings and the variable spark_schema_var will be defined in the output code.