Databricks Dialect Configuration¶
The Databricks Dialect Configuration extends the base converter config and allows you to configure additional aspects of the conversion process. The following sections describe the available configuration classes and their fields.
Databricks Converter Config¶
Field | Description | Default Value | Localizable | Environment Variable |
---|---|---|---|---|
default_output_extension | Default output file extension. | .py | True | |
render_unsupported_output_as_comment | If True converter will render whatever it was able to convert for unsupported source chunk as a comment. The output by definition will not be correct and may not even resemble valid code, but may be useful for creating user templates or reporting feature requests. This is especially true for large statements/expressions where only a small part is not supported. In this case the output may be very close to a fully converted one. Alchemist will determine a reasonable level at which to render the output. This is the same level as for render_unsupported_source_as_comment . |
True | True | |
user_template_paths | List of folders with user templates to make available for conversion. | Empty list | False | |
template_configs | List of template configs. | DBRSASTemplateConfig | False | |
template_tag_filter | Filter config for template tags to inclusion/exclusion from matching. | StringFilterConfig | True | |
node_filter | Filter config for node inclusion/exclusion from rendering. | NodeFilterConfig | True | |
use_runtime | If True converter will use runtime user-defined functions where appropriate. Be aware, that setting this to False may reduce the amount of automatically converted code, since for some of the constructs there may not be a non-runtime static inline version. | True | True | ALC_USE_RUNTIME |
custom_udf_name_mapping | Custom name mapping for runtime functions. This allows to use custom names for runtime functions. For the name of specific functions, consult target dialect documentation. | Empty dict | True | |
conversion_comment_verbosity_level | Verbosity level for conversion comments (code , todo , warning , debug ). Verbosity levels: - code : outputs only regular code comments retained from the source or considered part of the output. - todo : outputs code and todo comments; todo comments are used for output that has to be adjusted manually. - warning : outputs code , todo and warning comments; warning comments are used for potentially invalid code. - debug : outputs all comments, including developer warnings; debug comments are used for code that is unlikely to be invalid. |
todo | True | |
conversion_mode | Conversion mode (normal , strict , lax ). Changes code generation and how the comment verbosity level is handled: - NORMAL: The default mode. Balances correctness & readability, by allowing some heuristics about the common cases when achieving 100% match would generate overly verbose and complex code, while still allowing short constructs that ensure correctness. - STRICT: Prioritizes correctness over readability, striving to mark anything that is potentially not 100% correct in all scenarios as a TODO item and reducing heuristics to a minimum. - LAX: Prioritizes readability over correctness, assuming the best case scenario and avoiding generating additiona expressions that would be needed to handle edge cases. In addition to that, the verbosity level of conversion comments is adjusted based on the mode: - In strict mode, the warning comment is treated as todo , and debug is treated as warning . So more todo comments are generated. - In lax mode, the todo comment is treated as warning , and warning is treated as debug . Meaning no todo comments are generated at all. |
normal | True | |
llm | Configuration for GenAI base conversion. | LLMConfig | False | |
spark_conf_ansi_enabled | Whether to generate code that assumes ANSI SQL mode is enabled in Spark. | True | True | |
sas | SAS to Spark conversion options. | SparkSASConfig | True | |
group_nodes_into_paragraphs | Whether consecutive nodes of similar type should be grouped into a single notebook paragraph. | True | True | |
render_all_source_code | Whether notebooks should include entire original SAS code before the converted code. | True | True | |
render_markdown_headers | Whether notebooks should include Markdown cells with headers reflecting original SAS program structure. | True | True | |
file_path_map | File path mapping. | Empty dict | True |
Fila Path Map¶
File path mapping is used to convert source file location to a new cloud location. Result is always a posix path.
Mapping may specify a prefix part of the full path as it appears in the source, and how it should be converted to the target part of the path itself.
The longest matching prefix will be used. If no prefix matches, the original path will be used (which is probably not what you want).
The resulting path will be automatically converted to posix path.
Example:
- for the path
C:\User\any_local_dir\file.xlsx
- the mapping can be
{"C:\\User\\": "dbfs:/mnt/User/}
- and the final path will be
dbfs:/mnt/User/any_local_dir/file.xlsx
Databricks SAS Specific Converter Config¶
Field | Description | Default Value |
---|---|---|
year_cutoff | SAS YEARCUTOFF option value (see docs) | 40 |
libref_to_schema | Mapping of SAS librefs to Spark schemas. | Empty dictionary |
Example¶
Here's an example of how you can define libref_to_schema
in the configuration file:
converter:
sas:
libref_to_schema:
libref1: spark_schema1
libref2: "{spark_schema_var}"
In this example, libref1
will be converted to spark_schema1
, and libref2
will be converted to {spark_schema_var}
, assuming that it will be used in f-strings and the variable spark_schema_var
will be defined in the output code.