Operators allow for generation of certain types of tasks that become nodes in
the DAG when instantiated. All operators derive from BaseOperator
and
inherit many attributes and methods that way. Refer to the BaseOperator
documentation for more details.
There are 3 main types of operators:
BaseSensorOperator
and run a poke
method at a specified poke_interval
until it returns True
.All operators are derived from BaseOperator
and acquire much
functionality through inheritance. Since this is the core of the engine,
it’s worth taking the time to understand the parameters of BaseOperator
to understand the primitive features that can be leveraged in your
DAGs.
class airflow.models.BaseOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, weight_rule=u'downstream', queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)
Bases: airflow.utils.log.logging_mixin.LoggingMixin
Abstract base class for all operators. Since operators create objects that become nodes in the dag, BaseOperator contains many recursive methods for dag crawling behavior. To derive this class, you are expected to override the constructor as well as the ‘execute’ method.
Operators derived from this class should perform or trigger certain tasks synchronously (wait for completion). Example of operators could be an operator that runs a Pig job (PigOperator), a sensor operator that waits for a partition to land in Hive (HiveSensorOperator), or one that moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these operators (tasks) target specific operations, running specific scripts, functions or data transfers.
This class is abstract and shouldn’t be instantiated. Instantiating a class derived from this one results in the creation of a task object, which ultimately becomes a node in DAG objects. Task dependencies should be set by using the set_upstream and/or set_downstream methods.
Parameters: |
|
---|
clear(**kwargs)
Clears the state of task instances associated with the task, following the parameters specified.
dag
Returns the Operator’s DAG if set, otherwise raises an error
deps
Returns the list of dependencies for the operator. These differ from execution context dependencies in that they are specific to tasks and can be extended/overridden by subclasses.
downstream_list
@property: list of tasks directly downstream
execute(context)
This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
get_direct_relative_ids(upstream=False)
Get the direct relative ids to the current task, upstream or downstream.
get_direct_relatives(upstream=False)
Get the direct relatives to the current task, upstream or downstream.
get_flat_relative_ids(upstream=False, found_descendants=None)
Get a flat list of relatives’ ids, either upstream or downstream.
get_flat_relatives(upstream=False)
Get a flat list of relatives, either upstream or downstream.
get_task_instances(session, start_date=None, end_date=None)
Get a set of task instance related to this task for a specific date range.
has_dag()
Returns True if the Operator has been assigned to a DAG.
on_kill()
Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.
post_execute(context, *args, **kwargs)
This hook is triggered right after self.execute() is called. It is passed the execution context and any results returned by the operator.
pre_execute(context, *args, **kwargs)
This hook is triggered right before self.execute() is called.
prepare_template()
Hook that is triggered after the templated fields get replaced by their content. If you need your operator to alter the content of the file before the template is rendered, it should override this method to do so.
render_template(attr, content, context)
Renders a template either from a file or directly in a field, and returns the rendered result.
render_template_from_field(attr, content, context, jinja_env)
Renders a template from a field. If the field is a string, it will simply render the string and return the result. If it is a collection or nested set of collections, it will traverse the structure and render all strings in it.
run(start_date=None, end_date=None, ignore_first_depends_on_past=False, ignore_ti_state=False, mark_success=False)
Run a set of task instances for a date range.
schedule_interval
The schedule interval of the DAG always wins over individual tasks so that tasks within a DAG always line up. The task still needs a schedule_interval as it may not be attached to a DAG.
set_downstream(task_or_task_list)
Set a task or a task list to be directly downstream from the current task.
set_upstream(task_or_task_list)
Set a task or a task list to be directly upstream from the current task.
upstream_list
@property: list of tasks directly upstream
xcom_pull(context, task_ids=None, dag_id=None, key=u'return_value', include_prior_dates=None)
See TaskInstance.xcom_pull()
xcom_push(context, key, value, execution_date=None)
See TaskInstance.xcom_push()
All sensors are derived from BaseSensorOperator
. All sensors inherit
the timeout
and poke_interval
on top of the BaseOperator
attributes.
class airflow.sensors.base_sensor_operator.BaseSensorOperator(poke_interval=60, timeout=604800, soft_fail=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
, airflow.models.SkipMixin
Sensor operators are derived from this class an inherit these attributes.
Sensor operators keep executing at a time interval and succeed whena criteria is met and fail if and when they time out.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.operators.bash_operator.BashOperator(bash_command, xcom_push=False, env=None, output_encoding='utf-8', *args, **kwargs)
Bases: airflow.models.BaseOperator
Execute a Bash script, command or set of commands.
Parameters: |
|
---|
execute(context)
Execute the bash command in a temporary directory which will be cleaned afterwards
class airflow.operators.python_operator.BranchPythonOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)
Bases: airflow.operators.python_operator.PythonOperator
, airflow.models.SkipMixin
Allows a workflow to “branch” or follow a single path following the execution of this task.
It derives the PythonOperator and expects a Python function that returns
the task_id to follow. The task_id returned should point to a task
directly downstream from {self}. All other “branches” or
directly downstream tasks are marked with a state of skipped
so that
these paths can’t move forward. The skipped
states are propageted
downstream to allow for the DAG state to fill up and the DAG run’s state
to be inferred.
Note that using tasks with depends_on_past=True
downstream from
BranchPythonOperator
is logically unsound as skipped
status
will invariably lead to block tasks that depend on their past successes.
skipped
states propagates where all directly upstream tasks are
skipped
.
class airflow.operators.check_operator.CheckOperator(sql, conn_id=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Performs checks against a db. The CheckOperator
expects
a sql query that will return a single row. Each value on that
first row is evaluated using python bool
casting. If any of the
values return False
the check is failed and errors out.
Note that Python bool casting evals the following as False
:
False
0
""
)[]
){}
)Given a query like SELECT COUNT(*) FROM foo
, it will fail only if
the count == 0
. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today’s partition is
greater than yesterday’s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.
This operator can be used as a data quality check in your pipeline, and depending on where you put it in your DAG, you have the choice to stop the critical path, preventing from publishing dubious data, or on the side and receive email alerts without stopping the progress of the DAG.
Note that this is an abstract class and get_db_hook needs to be defined. Whereas a get_db_hook is hook that gets a single record from an external source.
Parameters: | sql (string) – the sql to be executed. (templated) |
---|
class airflow.operators.docker_operator.DockerOperator(image, api_version=None, command=None, cpus=1.0, docker_url='unix://var/run/docker.sock', environment=None, force_pull=False, mem_limit=None, network_mode=None, tls_ca_cert=None, tls_client_cert=None, tls_client_key=None, tls_hostname=None, tls_ssl_version=None, tmp_dir='/tmp/airflow', user=None, volumes=None, working_dir=None, xcom_push=False, xcom_all=False, docker_conn_id=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Execute a command inside a docker container.
A temporary directory is created on the host and
mounted into a container to allow storing files
that together exceed the default disk size of 10GB in a container.
The path to the mounted directory can be accessed
via the environment variable AIRFLOW_TMP_DIR
.
If a login to a private registry is required prior to pulling the image, a
Docker connection needs to be configured in Airflow and the connection ID
be provided with the parameter docker_conn_id
.
Parameters: |
|
---|
class airflow.operators.dummy_operator.DummyOperator(*args, **kwargs)
Bases: airflow.models.BaseOperator
Operator that does literally nothing. It can be used to group tasks in a DAG.
class airflow.operators.druid_check_operator.DruidCheckOperator(sql, druid_broker_conn_id='druid_broker_default', *args, **kwargs)
Bases: airflow.operators.check_operator.CheckOperator
Performs checks against Druid. The DruidCheckOperator
expects
a sql query that will return a single row. Each value on that
first row is evaluated using python bool
casting. If any of the
values return False
the check is failed and errors out.
Note that Python bool casting evals the following as False
:
False
0
""
)[]
){}
)Given a query like SELECT COUNT(*) FROM foo
, it will fail only if
the count == 0
. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today’s partition is
greater than yesterday’s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.
This operator can be used as a data quality check in your pipeline, and
depending on where you put it in your DAG, you have the choice to
stop the critical path, preventing from
publishing dubious data, or on the side and receive email alterts
without stopping the progress of the DAG.
Parameters: |
|
---|
get_db_hook()
Return the druid db api hook.
get_first(sql)
Executes the druid sql to druid broker and returns the first resulting row.
Parameters: | sql (str) – the sql statement to be executed (str) |
---|
class airflow.operators.email_operator.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, mime_subtype='mixed', mime_charset='us_ascii', *args, **kwargs)
Bases: airflow.models.BaseOperator
Sends an email.
Parameters: |
|
---|
class airflow.operators.generic_transfer.GenericTransfer(sql, destination_table, source_conn_id, destination_conn_id, preoperator=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from a connection to another, assuming that they both provide the required methods in their respective hooks. The source hook needs to expose a get_records method, and the destination a insert_rows method.
This is meant to be used on small-ish datasets that fit in memory.
Parameters: |
|
---|
class airflow.operators.hive_to_druid.HiveToDruidTransfer(sql, druid_datasource, ts_dim, metric_spec=None, hive_cli_conn_id='hive_cli_default', druid_ingest_conn_id='druid_ingest_default', metastore_conn_id='metastore_default', hadoop_dependency_coordinates=None, intervals=None, num_shards=-1, target_partition_size=-1, query_granularity='NONE', segment_granularity='DAY', hive_tblproperties=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from Hive to Druid, [del]note that for now the data is loaded into memory before being pushed to Druid, so this operator should be used for smallish amount of data.[/del]
Parameters: |
|
---|
construct_ingest_query(static_path, columns)
Builds an ingest query for an HDFS TSV load.
Parameters: |
|
---|
class airflow.operators.hive_to_mysql.HiveToMySqlTransfer(sql, mysql_table, hiveserver2_conn_id='hiveserver2_default', mysql_conn_id='mysql_default', mysql_preoperator=None, mysql_postoperator=None, bulk_load=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from Hive to MySQL, note that for now the data is loaded into memory before being pushed to MySQL, so this operator should be used for smallish amount of data.
Parameters: |
|
---|
class airflow.operators.hive_to_samba_operator.Hive2SambaOperator(hql, destination_filepath, samba_conn_id='samba_default', hiveserver2_conn_id='hiveserver2_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes hql code in a specific Hive database and loads the results of the query as a csv to a Samba location.
Parameters: |
|
---|
class airflow.operators.hive_operator.HiveOperator(hql, hive_cli_conn_id=u'hive_cli_default', schema=u'default', hiveconfs=None, hiveconf_jinja_translate=False, script_begin_tag=None, run_as_owner=False, mapred_queue=None, mapred_queue_priority=None, mapred_job_name=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes hql code or hive script in a specific Hive database.
Parameters: |
|
---|
class airflow.operators.hive_stats_operator.HiveStatsCollectionOperator(table, partition, extra_exprs=None, col_blacklist=None, assignment_func=None, metastore_conn_id='metastore_default', presto_conn_id='presto_default', mysql_conn_id='airflow_db', *args, **kwargs)
Bases: airflow.models.BaseOperator
Gathers partition statistics using a dynamically generated Presto query, inserts the stats into a MySql table with this format. Stats overwrite themselves if you rerun the same date/partition.
CREATE TABLE hive_stats (
ds VARCHAR(16),
table_name VARCHAR(500),
metric VARCHAR(200),
value BIGINT
);
Parameters: |
|
---|
class airflow.operators.check_operator.IntervalCheckOperator(table, metrics_thresholds, date_filter_column='ds', days_back=-7, conn_id=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Checks that the values of metrics given as SQL expressions are within a certain tolerance of the ones from days_back before.
Note that this is an abstract class and get_db_hook needs to be defined. Whereas a get_db_hook is hook that gets a single record from an external source.
Parameters: |
|
---|
class airflow.operators.jdbc_operator.JdbcOperator(sql, jdbc_conn_id='jdbc_default', autocommit=False, parameters=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a database using jdbc driver.
Requires jaydebeapi.
Parameters: |
|
---|
class airflow.operators.latest_only_operator.LatestOnlyOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, weight_rule=u'downstream', queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
, airflow.models.SkipMixin
Allows a workflow to skip tasks that are not running during the most recent schedule interval.
If the task is run outside of the latest schedule interval, all directly downstream tasks will be skipped.
class airflow.operators.mssql_operator.MsSqlOperator(sql, mssql_conn_id='mssql_default', parameters=None, autocommit=False, database=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a specific Microsoft SQL database
Parameters: |
|
---|
class airflow.operators.mssql_to_hive.MsSqlToHiveTransfer(sql, hive_table, create=True, recreate=False, partition=None, delimiter=u'x01', mssql_conn_id='mssql_default', hive_cli_conn_id='hive_cli_default', tblproperties=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from Microsoft SQL Server to Hive. The operator runs
your query against Microsoft SQL Server, stores the file locally
before loading it into a Hive table. If the create
or
recreate
arguments are set to True
,
a CREATE TABLE
and DROP TABLE
statements are generated.
Hive data types are inferred from the cursor’s metadata.
Note that the table generated in Hive uses STORED AS textfile
which isn’t the most efficient serialization format. If a
large amount of data is loaded and/or if the table gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a HiveOperator
.
Parameters: |
|
---|
class airflow.operators.mysql_operator.MySqlOperator(sql, mysql_conn_id='mysql_default', parameters=None, autocommit=False, database=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a specific MySQL database
Parameters: |
|
---|
class airflow.operators.mysql_to_hive.MySqlToHiveTransfer(sql, hive_table, create=True, recreate=False, partition=None, delimiter=u'x01', mysql_conn_id='mysql_default', hive_cli_conn_id='hive_cli_default', tblproperties=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from MySql to Hive. The operator runs your query against
MySQL, stores the file locally before loading it into a Hive table.
If the create
or recreate
arguments are set to True
,
a CREATE TABLE
and DROP TABLE
statements are generated.
Hive data types are inferred from the cursor’s metadata. Note that the
table generated in Hive uses STORED AS textfile
which isn’t the most efficient serialization format. If a
large amount of data is loaded and/or if the table gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a HiveOperator
.
Parameters: |
|
---|
class airflow.operators.oracle_operator.OracleOperator(sql, oracle_conn_id='oracle_default', parameters=None, autocommit=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a specific Oracle database :param oracle_conn_id: reference to a specific Oracle database :type oracle_conn_id: string :param sql: the sql code to be executed. (templated) :type sql: Can receive a str representing a sql statement,
a list of str (sql statements), or reference to a template file. Template reference are recognized by str ending in ‘.sql’
class airflow.operators.pig_operator.PigOperator(pig, pig_cli_conn_id='pig_cli_default', pigparams_jinja_translate=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes pig script.
Parameters: |
|
---|
class airflow.operators.postgres_operator.PostgresOperator(sql, postgres_conn_id='postgres_default', autocommit=False, parameters=None, database=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a specific Postgres database
Parameters: |
|
---|
class airflow.operators.presto_check_operator.PrestoCheckOperator(sql, presto_conn_id='presto_default', *args, **kwargs)
Bases: airflow.operators.check_operator.CheckOperator
Performs checks against Presto. The PrestoCheckOperator
expects
a sql query that will return a single row. Each value on that
first row is evaluated using python bool
casting. If any of the
values return False
the check is failed and errors out.
Note that Python bool casting evals the following as False
:
False
0
""
)[]
){}
)Given a query like SELECT COUNT(*) FROM foo
, it will fail only if
the count == 0
. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today’s partition is
greater than yesterday’s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.
This operator can be used as a data quality check in your pipeline, and depending on where you put it in your DAG, you have the choice to stop the critical path, preventing from publishing dubious data, or on the side and receive email alterts without stopping the progress of the DAG.
Parameters: |
|
---|
class airflow.operators.presto_check_operator.PrestoIntervalCheckOperator(table, metrics_thresholds, date_filter_column='ds', days_back=-7, presto_conn_id='presto_default', *args, **kwargs)
Bases: airflow.operators.check_operator.IntervalCheckOperator
Checks that the values of metrics given as SQL expressions are within a certain tolerance of the ones from days_back before.
Parameters: |
|
---|
class airflow.operators.presto_to_mysql.PrestoToMySqlTransfer(sql, mysql_table, presto_conn_id='presto_default', mysql_conn_id='mysql_default', mysql_preoperator=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from Presto to MySQL, note that for now the data is loaded into memory before being pushed to MySQL, so this operator should be used for smallish amount of data.
Parameters: |
|
---|
class airflow.operators.presto_check_operator.PrestoValueCheckOperator(sql, pass_value, tolerance=None, presto_conn_id='presto_default', *args, **kwargs)
Bases: airflow.operators.check_operator.ValueCheckOperator
Performs a simple value check using sql code.
Parameters: |
|
---|
class airflow.operators.python_operator.PythonOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes a Python callable
Parameters: |
|
---|
class airflow.operators.python_operator.PythonVirtualenvOperator(python_callable, requirements=None, python_version=None, use_dill=False, system_site_packages=True, op_args=None, op_kwargs=None, string_args=None, templates_dict=None, templates_exts=None, *args, **kwargs)
Bases: airflow.operators.python_operator.PythonOperator
Allows one to run a function in a virtualenv that is created and destroyed automatically (with certain caveats).
The function must be defined using def, and not be part of a class. All imports must happen inside the function and no variables outside of the scope may be referenced. A global scope variable named virtualenv_string_args will be available (populated by string_args). In addition, one can pass stuff through op_args and op_kwargs, and one can use a return value.
Note that if your virtualenv runs in a different Python major version than Airflow, you cannot use return values, op_args, or op_kwargs. You can use string_args though.
Parameters: |
|
---|
class airflow.operators.s3_file_transform_operator.S3FileTransformOperator(source_s3_key, dest_s3_key, transform_script=None, select_expression=None, source_aws_conn_id='aws_default', dest_aws_conn_id='aws_default', replace=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Copies data from a source S3 location to a temporary location on the local filesystem. Runs a transformation on this file as specified by the transformation script and uploads the output to a destination S3 location.
The locations of the source and the destination files in the local filesystem is provided as an first and second arguments to the transformation script. The transformation script is expected to read the data from source, transform it and write the output to the local destination file. The operator then takes over control and uploads the local destination file to S3.
S3 Select is also available to filter the source contents. Users can omit the transformation script if S3 Select expression is specified.
Parameters: |
|
---|
class airflow.operators.s3_to_hive_operator.S3ToHiveTransfer(s3_key, field_dict, hive_table, delimiter=', ', create=True, recreate=False, partition=None, headers=False, check_headers=False, wildcard_match=False, aws_conn_id='aws_default', hive_cli_conn_id='hive_cli_default', input_compressed=False, tblproperties=None, select_expression=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from S3 to Hive. The operator downloads a file from S3,
stores the file locally before loading it into a Hive table.
If the create
or recreate
arguments are set to True
,
a CREATE TABLE
and DROP TABLE
statements are generated.
Hive data types are inferred from the cursor’s metadata from.
Note that the table generated in Hive uses STORED AS textfile
which isn’t the most efficient serialization format. If a
large amount of data is loaded and/or if the tables gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a HiveOperator
.
Parameters: |
|
---|
class airflow.operators.s3_to_redshift_operator.S3ToRedshiftTransfer(schema, table, s3_bucket, s3_key, redshift_conn_id='redshift_default', aws_conn_id='aws_default', copy_options=(), autocommit=False, parameters=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes an COPY command to load files from s3 to Redshift
Parameters: |
|
---|
class airflow.operators.python_operator.ShortCircuitOperator(python_callable, op_args=None, op_kwargs=None, provide_context=False, templates_dict=None, templates_exts=None, *args, **kwargs)
Bases: airflow.operators.python_operator.PythonOperator
, airflow.models.SkipMixin
Allows a workflow to continue only if a condition is met. Otherwise, the workflow “short-circuits” and downstream tasks are skipped.
The ShortCircuitOperator is derived from the PythonOperator. It evaluates a condition and short-circuits the workflow if the condition is False. Any downstream tasks are marked with a state of “skipped”. If the condition is True, downstream tasks proceed as normal.
The condition is determined by the result of python_callable.
class airflow.operators.http_operator.SimpleHttpOperator(endpoint, method='POST', data=None, headers=None, response_check=None, extra_options=None, xcom_push=False, http_conn_id='http_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Calls an endpoint on an HTTP system to execute an action
Parameters: |
|
---|
class airflow.operators.slack_operator.SlackAPIOperator(slack_conn_id=None, token=None, method=None, api_params=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Base Slack Operator The SlackAPIPostOperator is derived from this operator. In the future additional Slack API Operators will be derived from this class as well
Parameters: |
|
---|
construct_api_call_params()
Used by the execute function. Allows templating on the source fields of the api_call_params dict before construction
Override in child classes. Each SlackAPIOperator child class is responsible for having a construct_api_call_params function which sets self.api_call_params with a dict of API call parameters (https://api.slack.com/methods)
execute(**kwargs)
SlackAPIOperator calls will not fail even if the call is not unsuccessful. It should not prevent a DAG from completing in success
class airflow.operators.slack_operator.SlackAPIPostOperator(channel='#general', username='Airflow', text='No message has been set.nHere is a cat video insteadnhttps://www.youtube.com/watch?v=J---aiyznGQ', icon_url='https://raw.githubusercontent.com/airbnb/airflow/master/airflow/www/static/pin_100.png', attachments=None, *args, **kwargs)
Bases: airflow.operators.slack_operator.SlackAPIOperator
Posts messages to a slack channel
Parameters: |
|
---|
construct_api_call_params()
Used by the execute function. Allows templating on the source fields of the api_call_params dict before construction
Override in child classes. Each SlackAPIOperator child class is responsible for having a construct_api_call_params function which sets self.api_call_params with a dict of API call parameters (https://api.slack.com/methods)
class airflow.operators.sqlite_operator.SqliteOperator(sql, sqlite_conn_id='sqlite_default', parameters=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a specific Sqlite database
Parameters: |
|
---|
class airflow.operators.subdag_operator.SubDagOperator(**kwargs)
Bases: airflow.models.BaseOperator
class airflow.operators.dagrun_operator.TriggerDagRunOperator(trigger_dag_id, python_callable=None, execution_date=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Triggers a DAG run for a specified dag_id
Parameters: |
|
---|
class airflow.operators.check_operator.ValueCheckOperator(sql, pass_value, tolerance=None, conn_id=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Performs a simple value check using sql code.
Note that this is an abstract class and get_db_hook needs to be defined. Whereas a get_db_hook is hook that gets a single record from an external source.
Parameters: | sql (string) – the sql to be executed. (templated) |
---|
class airflow.operators.redshift_to_s3_operator.RedshiftToS3Transfer(schema, table, s3_bucket, s3_key, redshift_conn_id='redshift_default', aws_conn_id='aws_default', unload_options=(), autocommit=False, parameters=None, include_header=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes an UNLOAD command to s3 as a CSV with headers
Parameters: |
|
---|
class airflow.sensors.external_task_sensor.ExternalTaskSensor(external_dag_id, external_task_id, allowed_states=None, execution_delta=None, execution_date_fn=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a task to complete in a different DAG
Parameters: |
|
---|
poke(**kwargs)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.hdfs_sensor.HdfsSensor(filepath, hdfs_conn_id='hdfs_default', ignored_ext=['_COPYING_'], ignore_copying=True, file_size=None, hook=<class 'airflow.hooks.hdfs_hook.HDFSHook'>, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a file or folder to land in HDFS
static filter_for_filesize(result, size=None)
Will test the filepath result and test if its size is at least self.filesize
Parameters: |
|
---|---|
Returns: | (bool) depending on the matching criteria |
static filter_for_ignored_ext(result, ignored_ext, ignore_copying)
Will filter if instructed to do so the result to remove matching criteria
Parameters: |
|
---|---|
Returns: | (list) of dicts which were not removed |
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.hive_partition_sensor.HivePartitionSensor(table, partition="ds='{{ ds }}'", metastore_conn_id='metastore_default', schema='default', poke_interval=180, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a partition to show up in Hive.
Note: Because partition
supports general logical operators, it
can be inefficient. Consider using NamedHivePartitionSensor instead if
you don’t need the full flexibility of HivePartitionSensor.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.http_sensor.HttpSensor(endpoint, http_conn_id='http_default', method='GET', request_params=None, headers=None, response_check=None, extra_options=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Executes a HTTP get statement and returns False on failure:404 not found or response_check function returned False
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.metastore_partition_sensor.MetastorePartitionSensor(table, partition_name, schema='default', mysql_conn_id='metastore_mysql', *args, **kwargs)
Bases: airflow.sensors.sql_sensor.SqlSensor
An alternative to the HivePartitionSensor that talk directly to the MySQL db. This was created as a result of observing sub optimal queries generated by the Metastore thrift service when hitting subpartitioned tables. The Thrift service’s queries were written in a way that wouldn’t leverage the indexes.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.named_hive_partition_sensor.NamedHivePartitionSensor(partition_names, metastore_conn_id='metastore_default', poke_interval=180, hook=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a set of partitions to show up in Hive.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.s3_key_sensor.S3KeySensor(bucket_key, bucket_name=None, wildcard_match=False, aws_conn_id='aws_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a key (a file-like instance on S3) to be present in a S3 bucket. S3 being a key/value it does not support folders. The path is just a key a resource.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.s3_prefix_sensor.S3PrefixSensor(bucket_name, prefix, delimiter='/', aws_conn_id='aws_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a prefix to exist. A prefix is the first part of a key, thus enabling checking of constructs similar to glob airfl* or SQL LIKE ‘airfl%’. There is the possibility to precise a delimiter to indicate the hierarchy or keys, meaning that the match will stop at that delimiter. Current code accepts sane delimiters, i.e. characters that are NOT special characters in the Python regex engine.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.sql_sensor.SqlSensor(conn_id, sql, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Runs a sql statement until a criteria is met. It will keep trying while sql returns no row, or if the first cell in (0, ‘0’, ‘’).
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.time_sensor.TimeSensor(target_time, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits until the specified time of the day.
Parameters: | target_time (datetime.time) – time after which the job succeeds |
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.time_delta_sensor.TimeDeltaSensor(delta, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a timedelta after the task’s execution_date + schedule_interval.
In Airflow, the daily task stamped with execution_date
2016-01-01 can only start running on 2016-01-02. The timedelta here
represents the time after the execution period has closed.
Parameters: | delta (datetime.timedelta) – time length to wait after execution_date before succeeding |
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.sensors.web_hdfs_sensor.WebHdfsSensor(filepath, webhdfs_conn_id='webhdfs_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a file or folder to land in HDFS
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.operators.awsbatch_operator.AWSBatchOperator(job_name, job_definition, job_queue, overrides, max_retries=4200, aws_conn_id=None, region_name=None, **kwargs)
Bases: airflow.models.BaseOperator
Execute a job on AWS Batch Service
Parameters: |
|
---|---|
Param: | overrides: the same parameter that boto3 will receive on containerOverrides (templated): http://boto3.readthedocs.io/en/latest/reference/services/batch.html#submit_job |
Type: | overrides: dict |
class airflow.contrib.operators.bigquery_check_operator.BigQueryCheckOperator(sql, bigquery_conn_id='bigquery_default', *args, **kwargs)
Bases: airflow.operators.check_operator.CheckOperator
Performs checks against BigQuery. The BigQueryCheckOperator
expects
a sql query that will return a single row. Each value on that
first row is evaluated using python bool
casting. If any of the
values return False
the check is failed and errors out.
Note that Python bool casting evals the following as False
:
False
0
""
)[]
){}
)Given a query like SELECT COUNT(*) FROM foo
, it will fail only if
the count == 0
. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today’s partition is
greater than yesterday’s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.
This operator can be used as a data quality check in your pipeline, and depending on where you put it in your DAG, you have the choice to stop the critical path, preventing from publishing dubious data, or on the side and receive email alterts without stopping the progress of the DAG.
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_check_operator.BigQueryValueCheckOperator(sql, pass_value, tolerance=None, bigquery_conn_id='bigquery_default', *args, **kwargs)
Bases: airflow.operators.check_operator.ValueCheckOperator
Performs a simple value check using sql code.
Parameters: | sql (string) – the sql to be executed |
---|
class airflow.contrib.operators.bigquery_check_operator.BigQueryIntervalCheckOperator(table, metrics_thresholds, date_filter_column='ds', days_back=-7, bigquery_conn_id='bigquery_default', *args, **kwargs)
Bases: airflow.operators.check_operator.IntervalCheckOperator
Checks that the values of metrics given as SQL expressions are within a certain tolerance of the ones from days_back before.
This method constructs a query like so
SELECT {metrics_threshold_dict_key} FROM {table}
WHERE {date_filter_column}=<date>
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_get_data.BigQueryGetDataOperator(dataset_id, table_id, max_results='100', selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Fetches the data from a BigQuery table (alternatively fetch data for selected columns) and returns data in a python list. The number of elements in the returned list will be equal to the number of rows fetched. Each element in the list will again be a list where element would represent the columns values for that row.
Example Result: [['Tony', '10'], ['Mike', '20'], ['Steve', '15']]
Note
If you pass fields to selected_fields
which are in different order than the
order of columns already in
BQ table, the data will still be in the order of BQ table.
For example if the BQ table has 3 columns as
[A,B,C]
and you pass ‘B,A’ in the selected_fields
the data would still be of the form 'A,B'
.
Example:
get_data = BigQueryGetDataOperator(
task_id='get_data_from_bq',
dataset_id='test_dataset',
table_id='Transaction_partitions',
max_results='100',
selected_fields='DATE',
bigquery_conn_id='airflow-service-account'
)
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_operator.BigQueryCreateEmptyTableOperator(dataset_id, table_id, project_id=None, schema_fields=None, gcs_schema_object=None, time_partitioning={}, bigquery_conn_id='bigquery_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Creates a new, empty table in the specified BigQuery dataset, optionally with schema.
The schema to be used for the BigQuery table may be specified in one of two ways. You may either directly pass the schema fields in, or you may point the operator to a Google cloud storage object name. The object in Google cloud storage must be a JSON file with the schema fields in it. You can also create a table without schema.
Parameters: |
|
---|
Example (with schema JSON in GCS):
CreateTable = BigQueryCreateEmptyTableOperator(
task_id='BigQueryCreateEmptyTableOperator_task',
dataset_id='ODS',
table_id='Employees',
project_id='internal-gcp-project',
gcs_schema_object='gs://schema-bucket/employee_schema.json',
bigquery_conn_id='airflow-service-account',
google_cloud_storage_conn_id='airflow-service-account'
)
Corresponding Schema file (employee_schema.json
):
[
{
"mode": "NULLABLE",
"name": "emp_name",
"type": "STRING"
},
{
"mode": "REQUIRED",
"name": "salary",
"type": "INTEGER"
}
]
Example (with schema in the DAG):
CreateTable = BigQueryCreateEmptyTableOperator(
task_id='BigQueryCreateEmptyTableOperator_task',
dataset_id='ODS',
table_id='Employees',
project_id='internal-gcp-project',
schema_fields=[{"name": "emp_name", "type": "STRING", "mode": "REQUIRED"},
{"name": "salary", "type": "INTEGER", "mode": "NULLABLE"}],
bigquery_conn_id='airflow-service-account',
google_cloud_storage_conn_id='airflow-service-account'
)
class airflow.contrib.operators.bigquery_operator.BigQueryCreateExternalTableOperator(bucket, source_objects, destination_project_dataset_table, schema_fields=None, schema_object=None, source_format='CSV', compression='NONE', skip_leading_rows=0, field_delimiter=', ', max_bad_records=0, quote_character=None, allow_quoted_newlines=False, allow_jagged_rows=False, bigquery_conn_id='bigquery_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, src_fmt_configs={}, *args, **kwargs)
Bases: airflow.models.BaseOperator
Creates a new external table in the dataset with the data in Google Cloud Storage.
The schema to be used for the BigQuery table may be specified in one of two ways. You may either directly pass the schema fields in, or you may point the operator to a Google cloud storage object name. The object in Google cloud storage must be a JSON file with the schema fields in it.
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_operator.BigQueryOperator(bql=None, sql=None, destination_dataset_table=False, write_disposition='WRITE_EMPTY', allow_large_results=False, flatten_results=False, bigquery_conn_id='bigquery_default', delegate_to=None, udf_config=False, use_legacy_sql=True, maximum_billing_tier=None, maximum_bytes_billed=None, create_disposition='CREATE_IF_NEEDED', schema_update_options=(), query_params=None, priority='INTERACTIVE', time_partitioning={}, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes BigQuery SQL queries in a specific BigQuery database
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_table_delete_operator.BigQueryTableDeleteOperator(deletion_dataset_table, bigquery_conn_id='bigquery_default', delegate_to=None, ignore_if_missing=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Deletes BigQuery tables
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_to_bigquery.BigQueryToBigQueryOperator(source_project_dataset_tables, destination_project_dataset_table, write_disposition='WRITE_EMPTY', create_disposition='CREATE_IF_NEEDED', bigquery_conn_id='bigquery_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Copies data from one BigQuery table to another.
See also
For more details about these parameters: https://cloud.google.com/bigquery/docs/reference/v2/jobs#configuration.copy
Parameters: |
|
---|
class airflow.contrib.operators.bigquery_to_gcs.BigQueryToCloudStorageOperator(source_project_dataset_table, destination_cloud_storage_uris, compression='NONE', export_format='CSV', field_delimiter=', ', print_header=True, bigquery_conn_id='bigquery_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Transfers a BigQuery table to a Google Cloud Storage bucket.
See also
For more details about these parameters: https://cloud.google.com/bigquery/docs/reference/v2/jobs
Parameters: |
|
---|
class airflow.contrib.operators.cassandra_to_gcs.CassandraToGoogleCloudStorageOperator(cql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, cassandra_conn_id=u'cassandra_default', google_cloud_storage_conn_id=u'google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Copy data from Cassandra to Google cloud storage in JSON format
Note: Arrays of arrays are not supported.
classmethod convert_map_type(name, value)
Converts a map to a repeated RECORD that contains two fields: ‘key’ and ‘value’, each will be converted to its corresopnding data type in BQ.
classmethod convert_tuple_type(name, value)
Converts a tuple to RECORD that contains n fields, each will be converted to its corresponding data type in bq and will be named ‘field_<index>’, where index is determined by the order of the tuple elments defined in cassandra.
classmethod convert_user_type(name, value)
Converts a user type to RECORD that contains n fields, where n is the number of attributes. Each element in the user type class will be converted to its corresponding data type in BQ.
class airflow.contrib.operators.databricks_operator.DatabricksSubmitRunOperator(json=None, spark_jar_task=None, notebook_task=None, new_cluster=None, existing_cluster_id=None, libraries=None, run_name=None, timeout_seconds=None, databricks_conn_id='databricks_default', polling_period_seconds=30, databricks_retry_limit=3, do_xcom_push=False, **kwargs)
Bases: airflow.models.BaseOperator
Submits an Spark job run to Databricks using the api/2.0/jobs/runs/submit API endpoint.
There are two ways to instantiate this operator.
In the first way, you can take the JSON payload that you typically use
to call the api/2.0/jobs/runs/submit
endpoint and pass it directly
to our DatabricksSubmitRunOperator
through the json
parameter.
For example
json = {
'new_cluster': {
'spark_version': '2.1.0-db3-scala2.11',
'num_workers': 2
},
'notebook_task': {
'notebook_path': '/Users/airflow@example.com/PrepareData',
},
}
notebook_run = DatabricksSubmitRunOperator(task_id='notebook_run', json=json)
Another way to accomplish the same thing is to use the named parameters
of the DatabricksSubmitRunOperator
directly. Note that there is exactly
one named parameter for each top level parameter in the runs/submit
endpoint. In this method, your code would look like this:
new_cluster = {
'spark_version': '2.1.0-db3-scala2.11',
'num_workers': 2
}
notebook_task = {
'notebook_path': '/Users/airflow@example.com/PrepareData',
}
notebook_run = DatabricksSubmitRunOperator(
task_id='notebook_run',
new_cluster=new_cluster,
notebook_task=notebook_task)
In the case where both the json parameter AND the named parameters
are provided, they will be merged together. If there are conflicts during the merge,
the named parameters will take precedence and override the top level json
keys.
Currently the named parameters that DatabricksSubmitRunOperator supports are
spark_jar_task
notebook_task
new_cluster
existing_cluster_id
libraries
run_name
timeout_seconds
Parameters: |
|
---|
class airflow.contrib.operators.dataflow_operator.DataFlowJavaOperator(jar, dataflow_default_options=None, options=None, gcp_conn_id='google_cloud_default', delegate_to=None, poll_sleep=10, job_class=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Java Cloud DataFlow batch job. The parameters of the operation will be passed to the job.
It’s a good practice to define dataflow_* parameters in the default_args of the dag like the project, zone and staging location.
default_args = {
'dataflow_default_options': {
'project': 'my-gcp-project',
'zone': 'europe-west1-d',
'stagingLocation': 'gs://my-staging-bucket/staging/'
}
}
You need to pass the path to your dataflow as a file reference with the jar
parameter, the jar needs to be a self executing jar (see documentation here:
https://beam.apache.org/documentation/runners/dataflow/#self-executing-jar).
Use options
to pass on options to your job.
t1 = DataFlowOperation(
task_id='datapflow_example',
jar='{{var.value.gcp_dataflow_base}}pipeline/build/libs/pipeline-example-1.0.jar',
options={
'autoscalingAlgorithm': 'BASIC',
'maxNumWorkers': '50',
'start': '{{ds}}',
'partitionType': 'DAY',
'labels': {'foo' : 'bar'}
},
gcp_conn_id='gcp-airflow-service-account',
dag=my-dag)
Both jar
and options
are templated so you can use variables in them.
class airflow.contrib.operators.dataflow_operator.DataflowTemplateOperator(template, dataflow_default_options=None, parameters=None, gcp_conn_id='google_cloud_default', delegate_to=None, poll_sleep=10, *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Templated Cloud DataFlow batch job. The parameters of the operation will be passed to the job. It’s a good practice to define dataflow_* parameters in the default_args of the dag like the project, zone and staging location.
See also
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/LaunchTemplateParameters https://cloud.google.com/dataflow/docs/reference/rest/v1b3/RuntimeEnvironment
default_args = {
'dataflow_default_options': {
'project': 'my-gcp-project'
'zone': 'europe-west1-d',
'tempLocation': 'gs://my-staging-bucket/staging/'
}
}
}
You need to pass the path to your dataflow template as a file reference with the
template
parameter. Use parameters
to pass on parameters to your job.
Use environment
to pass on runtime environment variables to your job.
t1 = DataflowTemplateOperator(
task_id='datapflow_example',
template='{{var.value.gcp_dataflow_base}}',
parameters={
'inputFile': "gs://bucket/input/my_input.txt",
'outputFile': "gs://bucket/output/my_output.txt"
},
gcp_conn_id='gcp-airflow-service-account',
dag=my-dag)
template
, dataflow_default_options
and parameters
are templated so you can
use variables in them.
class airflow.contrib.operators.dataflow_operator.DataFlowPythonOperator(py_file, py_options=None, dataflow_default_options=None, options=None, gcp_conn_id='google_cloud_default', delegate_to=None, poll_sleep=10, *args, **kwargs)
Bases: airflow.models.BaseOperator
execute(context)
Execute the python dataflow job.
class airflow.contrib.operators.dataproc_operator.DataprocClusterCreateOperator(cluster_name, project_id, num_workers, zone, network_uri=None, subnetwork_uri=None, internal_ip_only=None, tags=None, storage_bucket=None, init_actions_uris=None, init_action_timeout='10m', metadata=None, image_version=None, properties=None, master_machine_type='n1-standard-4', master_disk_size=500, worker_machine_type='n1-standard-4', worker_disk_size=500, num_preemptible_workers=0, labels=None, region='global', gcp_conn_id='google_cloud_default', delegate_to=None, service_account=None, service_account_scopes=None, idle_delete_ttl=None, auto_delete_time=None, auto_delete_ttl=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Create a new cluster on Google Cloud Dataproc. The operator will wait until the creation is successful or an error occurs in the creation process.
The parameters allow to configure the cluster. Please refer to
https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters
for a detailed explanation on the different parameters. Most of the configuration parameters detailed in the link are available as a parameter to this operator.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataprocClusterScaleOperator(cluster_name, project_id, region='global', gcp_conn_id='google_cloud_default', delegate_to=None, num_workers=2, num_preemptible_workers=0, graceful_decommission_timeout=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Scale, up or down, a cluster on Google Cloud Dataproc. The operator will wait until the cluster is re-scaled.
Example:
t1 = DataprocClusterScaleOperator(task_id=’dataproc_scale’, project_id=’my-project’, cluster_name=’cluster-1’, num_workers=10, num_preemptible_workers=10, graceful_decommission_timeout=‘1h’ dag=dag)
See also
For more detail on about scaling clusters have a look at the reference: https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scaling-clusters
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataprocClusterDeleteOperator(cluster_name, project_id, region='global', gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Delete a cluster on Google Cloud Dataproc. The operator will wait until the cluster is destroyed.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataProcPigOperator(query=None, query_uri=None, variables=None, job_name='{{task.task_id}}_{{ds_nodash}}', cluster_name='cluster-1', dataproc_pig_properties=None, dataproc_pig_jars=None, gcp_conn_id='google_cloud_default', delegate_to=None, region='global', *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Pig query Job on a Cloud DataProc cluster. The parameters of the operation will be passed to the cluster.
It’s a good practice to define dataproc_* parameters in the default_args of the dag like the cluster name and UDFs.
default_args = {
'cluster_name': 'cluster-1',
'dataproc_pig_jars': [
'gs://example/udf/jar/datafu/1.2.0/datafu.jar',
'gs://example/udf/jar/gpig/1.2/gpig.jar'
]
}
You can pass a pig script as string or file reference. Use variables to pass on variables for the pig script to be resolved on the cluster or use the parameters to be resolved in the script as template parameters.
Example:
t1 = DataProcPigOperator(
task_id='dataproc_pig',
query='a_pig_script.pig',
variables={'out': 'gs://example/output/{{ds}}'},
dag=dag)
See also
For more detail on about job submission have a look at the reference: https://cloud.google.com/dataproc/reference/rest/v1/projects.regions.jobs
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataProcHiveOperator(query=None, query_uri=None, variables=None, job_name='{{task.task_id}}_{{ds_nodash}}', cluster_name='cluster-1', dataproc_hive_properties=None, dataproc_hive_jars=None, gcp_conn_id='google_cloud_default', delegate_to=None, region='global', *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Hive query Job on a Cloud DataProc cluster.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataProcSparkSqlOperator(query=None, query_uri=None, variables=None, job_name='{{task.task_id}}_{{ds_nodash}}', cluster_name='cluster-1', dataproc_spark_properties=None, dataproc_spark_jars=None, gcp_conn_id='google_cloud_default', delegate_to=None, region='global', *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Spark SQL query Job on a Cloud DataProc cluster.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataProcSparkOperator(main_jar=None, main_class=None, arguments=None, archives=None, files=None, job_name='{{task.task_id}}_{{ds_nodash}}', cluster_name='cluster-1', dataproc_spark_properties=None, dataproc_spark_jars=None, gcp_conn_id='google_cloud_default', delegate_to=None, region='global', *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Spark Job on a Cloud DataProc cluster.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataProcHadoopOperator(main_jar=None, main_class=None, arguments=None, archives=None, files=None, job_name='{{task.task_id}}_{{ds_nodash}}', cluster_name='cluster-1', dataproc_hadoop_properties=None, dataproc_hadoop_jars=None, gcp_conn_id='google_cloud_default', delegate_to=None, region='global', *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Hadoop Job on a Cloud DataProc cluster.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataProcPySparkOperator(main, arguments=None, archives=None, pyfiles=None, files=None, job_name='{{task.task_id}}_{{ds_nodash}}', cluster_name='cluster-1', dataproc_pyspark_properties=None, dataproc_pyspark_jars=None, gcp_conn_id='google_cloud_default', delegate_to=None, region='global', *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a PySpark Job on a Cloud DataProc cluster.
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator(project_id, region='global', gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
class airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateOperator(template_id, *args, **kwargs)
Bases: airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator
Instantiate a WorkflowTemplate on Google Cloud Dataproc. The operator will wait until the WorkflowTemplate is finished executing.
See also
Please refer to: https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiate
Parameters: |
|
---|
class airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateInstantiateInlineOperator(template, *args, **kwargs)
Bases: airflow.contrib.operators.dataproc_operator.DataprocWorkflowTemplateBaseOperator
Instantiate a WorkflowTemplate Inline on Google Cloud Dataproc. The operator will wait until the WorkflowTemplate is finished executing.
See also
Please refer to: https://cloud.google.com/dataproc/docs/reference/rest/v1beta2/projects.regions.workflowTemplates/instantiateInline
Parameters: |
|
---|
class airflow.contrib.operators.datastore_export_operator.DatastoreExportOperator(bucket, namespace=None, datastore_conn_id='google_cloud_default', cloud_storage_conn_id='google_cloud_default', delegate_to=None, entity_filter=None, labels=None, polling_interval_in_seconds=10, overwrite_existing=False, xcom_push=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Export entities from Google Cloud Datastore to Cloud Storage
Parameters: |
|
---|
class airflow.contrib.operators.datastore_import_operator.DatastoreImportOperator(bucket, file, namespace=None, entity_filter=None, labels=None, datastore_conn_id='google_cloud_default', delegate_to=None, polling_interval_in_seconds=10, xcom_push=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Import entities from Cloud Storage to Google Cloud Datastore
Parameters: |
|
---|
class airflow.contrib.operators.discord_webhook_operator.DiscordWebhookOperator(http_conn_id=None, webhook_endpoint=None, message='', username=None, avatar_url=None, tts=False, proxy=None, *args, **kwargs)
Bases: airflow.operators.http_operator.SimpleHttpOperator
This operator allows you to post messages to Discord using incoming webhooks. Takes a Discord connection ID with a default relative webhook endpoint. The default endpoint can be overridden using the webhook_endpoint parameter (https://discordapp.com/developers/docs/resources/webhook).
Each Discord webhook can be pre-configured to use a specific username and avatar_url. You can override these defaults in this operator.
Parameters: |
|
---|
execute(context)
Call the DiscordWebhookHook to post message
class airflow.contrib.operators.druid_operator.DruidOperator(json_index_file, druid_ingest_conn_id='druid_ingest_default', max_ingestion_time=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Allows to submit a task directly to druid
Parameters: |
|
---|
class airflow.contrib.operators.ecs_operator.ECSOperator(task_definition, cluster, overrides, aws_conn_id=None, region_name=None, launch_type='EC2', **kwargs)
Bases: airflow.models.BaseOperator
Execute a task on AWS EC2 Container Service
Parameters: |
|
---|---|
Param: | overrides: the same parameter that boto3 will receive (templated): http://boto3.readthedocs.org/en/latest/reference/services/ecs.html#ECS.Client.run_task |
Type: | overrides: dict |
Type: | launch_type: str |
class airflow.contrib.operators.emr_add_steps_operator.EmrAddStepsOperator(job_flow_id, aws_conn_id='s3_default', steps=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
An operator that adds steps to an existing EMR job_flow.
Parameters: |
|
---|
class airflow.contrib.operators.emr_create_job_flow_operator.EmrCreateJobFlowOperator(aws_conn_id='s3_default', emr_conn_id='emr_default', job_flow_overrides=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Creates an EMR JobFlow, reading the config from the EMR connection. A dictionary of JobFlow overrides can be passed that override the config from the connection.
Parameters: |
|
---|
class airflow.contrib.operators.emr_terminate_job_flow_operator.EmrTerminateJobFlowOperator(job_flow_id, aws_conn_id='s3_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Operator to terminate EMR JobFlows.
Parameters: |
|
---|
class airflow.contrib.operators.file_to_gcs.FileToGoogleCloudStorageOperator(src, dst, bucket, google_cloud_storage_conn_id='google_cloud_default', mime_type='application/octet-stream', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Uploads a file to Google Cloud Storage
Parameters: |
|
---|
execute(context)
Uploads the file to Google cloud storage
class airflow.contrib.operators.file_to_wasb.FileToWasbOperator(file_path, container_name, blob_name, wasb_conn_id='wasb_default', load_options=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Uploads a file to Azure Blob Storage.
Parameters: |
|
---|
execute(context)
Upload a file to Azure Blob Storage.
class airflow.contrib.operators.gcp_container_operator.GKEClusterCreateOperator(project_id, location, body={}, gcp_conn_id='google_cloud_default', api_version='v2', *args, **kwargs)
Bases: airflow.models.BaseOperator
class airflow.contrib.operators.gcp_container_operator.GKEClusterDeleteOperator(project_id, name, location, gcp_conn_id='google_cloud_default', api_version='v2', *args, **kwargs)
Bases: airflow.models.BaseOperator
class airflow.contrib.operators.gcs_download_operator.GoogleCloudStorageDownloadOperator(bucket, object, filename=None, store_to_xcom_key=None, google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Downloads a file from Google Cloud Storage.
Parameters: |
|
---|
class airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator(bucket, prefix=None, delimiter=None, google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
List all objects from the bucket with the give string prefix and delimiter in name.
This operator returns a python list with the name of objects which can be used byxcom in the downstream task.
Parameters: |
|
---|
Example:
The following Operator would list all the Avro files from sales/sales-2017
folder in data
bucket.
GCS_Files = GoogleCloudStorageListOperator(
task_id='GCS_Files',
bucket='data',
prefix='sales/sales-2017/',
delimiter='.avro',
google_cloud_storage_conn_id=google_cloud_conn_id
)
class airflow.contrib.operators.gcs_operator.GoogleCloudStorageCreateBucketOperator(bucket_name, storage_class='MULTI_REGIONAL', location='US', project_id=None, labels=None, google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Creates a new bucket. Google Cloud Storage uses a flat namespace, so you can’t create a bucket with a name that is already in use.
See also
For more information, see Bucket Naming Guidelines: https://cloud.google.com/storage/docs/bucketnaming.html#requirements
Parameters: |
|
---|
Example:
The following Operator would create a new bucket test-bucket
with MULTI_REGIONAL
storage class in EU
region
CreateBucket = GoogleCloudStorageCreateBucketOperator(
task_id='CreateNewBucket',
bucket_name='test-bucket',
storage_class='MULTI_REGIONAL',
location='EU',
labels={'env': 'dev', 'team': 'airflow'},
google_cloud_storage_conn_id='airflow-service-account'
)
class airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator(bucket, source_objects, destination_project_dataset_table, schema_fields=None, schema_object=None, source_format='CSV', compression='NONE', create_disposition='CREATE_IF_NEEDED', skip_leading_rows=0, write_disposition='WRITE_EMPTY', field_delimiter=', ', max_bad_records=0, quote_character=None, ignore_unknown_values=False, allow_quoted_newlines=False, allow_jagged_rows=False, max_id_key=None, bigquery_conn_id='bigquery_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, schema_update_options=(), src_fmt_configs={}, external_table=False, time_partitioning={}, *args, **kwargs)
Bases: airflow.models.BaseOperator
Loads files from Google cloud storage into BigQuery.
The schema to be used for the BigQuery table may be specified in one of two ways. You may either directly pass the schema fields in, or you may point the operator to a Google cloud storage object name. The object in Google cloud storage must be a JSON file with the schema fields in it.
Parameters: |
|
---|
class airflow.contrib.operators.gcs_to_gcs.GoogleCloudStorageToGoogleCloudStorageOperator(source_bucket, source_object, destination_bucket=None, destination_object=None, move_object=False, google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Copies objects from a bucket to another, with renaming if requested.
Parameters: |
|
---|
where the object should be. (templated) :type destination_bucket: string :param destination_object: The destination name of the object in the
destination Google cloud storage bucket. (templated) If a wildcard is supplied in the source_object argument, this is the prefix that will be prepended to the final destination objects’ paths. Note that the source path’s part before the wildcard will be removed; if it needs to be retained it should be appended to destination_object. For example, with prefixfoo/*
and destination_object ‘blah/`, the filefoo/baz
will be copied toblah/baz
; to retain the prefix write the destination_object as e.g.blah/foo
, in which case the copied file will be namedblah/foo/baz
.
Parameters: | move_object – When move object is True, the object is moved instead |
---|
of copied to the new location.This is the equivalent of a mv command as opposed to a cp command.
Parameters: |
|
---|
Examples:
The following Operator would copy a single file named
sales/sales-2017/january.avro
in the data
bucket to the file named
copied_sales/2017/january-backup.avro` in the ``data_backup
bucket
copy_single_file = GoogleCloudStorageToGoogleCloudStorageOperator(
task_id='copy_single_file',
source_bucket='data',
source_object='sales/sales-2017/january.avro',
destination_bucket='data_backup',
destination_object='copied_sales/2017/january-backup.avro',
google_cloud_storage_conn_id=google_cloud_conn_id
)
The following Operator would copy all the Avro files from sales/sales-2017
folder (i.e. with names starting with that prefix) in data
bucket to the
copied_sales/2017
folder in the data_backup
bucket.
copy_files = GoogleCloudStorageToGoogleCloudStorageOperator(
task_id='copy_files',
source_bucket='data',
source_object='sales/sales-2017/*.avro',
destination_bucket='data_backup',
destination_object='copied_sales/2017/',
google_cloud_storage_conn_id=google_cloud_conn_id
)
The following Operator would move all the Avro files from sales/sales-2017
folder (i.e. with names starting with that prefix) in data
bucket to the
same folder in the data_backup
bucket, deleting the original files in the
process.
move_files = GoogleCloudStorageToGoogleCloudStorageOperator(
task_id='move_files',
source_bucket='data',
source_object='sales/sales-2017/*.avro',
destination_bucket='data_backup',
move_object=True,
google_cloud_storage_conn_id=google_cloud_conn_id
)
class airflow.contrib.operators.gcs_to_s3.GoogleCloudStorageToS3Operator(bucket, prefix=None, delimiter=None, google_cloud_storage_conn_id='google_cloud_storage_default', delegate_to=None, dest_aws_conn_id=None, dest_s3_key=None, replace=False, *args, **kwargs)
Bases: airflow.contrib.operators.gcs_list_operator.GoogleCloudStorageListOperator
Synchronizes a Google Cloud Storage bucket with an S3 bucket.
Parameters: |
|
---|
class airflow.contrib.operators.hipchat_operator.HipChatAPIOperator(token, base_url='https://api.hipchat.com/v2', *args, **kwargs)
Bases: airflow.models.BaseOperator
Base HipChat Operator. All derived HipChat operators reference from HipChat’s official REST API documentation at https://www.hipchat.com/docs/apiv2. Before using any HipChat API operators you need to get an authentication token at https://www.hipchat.com/docs/apiv2/auth. In the future additional HipChat operators will be derived from this class as well.
Parameters: |
|
---|
prepare_request()
Used by the execute function. Set the request method, url, and body of HipChat’s REST API call. Override in child class. Each HipChatAPI child operator is responsible for having a prepare_request method call which sets self.method, self.url, and self.body.
class airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator(room_id, message, *args, **kwargs)
Bases: airflow.contrib.operators.hipchat_operator.HipChatAPIOperator
Send notification to a specific HipChat room. More info: https://www.hipchat.com/docs/apiv2/method/send_room_notification
Parameters: |
|
---|
prepare_request()
Used by the execute function. Set the request method, url, and body of HipChat’s REST API call. Override in child class. Each HipChatAPI child operator is responsible for having a prepare_request method call which sets self.method, self.url, and self.body.
class airflow.contrib.operators.hive_to_dynamodb.HiveToDynamoDBTransferOperator(sql, table_name, table_keys, pre_process=None, pre_process_args=None, pre_process_kwargs=None, region_name=None, schema='default', hiveserver2_conn_id='hiveserver2_default', aws_conn_id='aws_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from Hive to DynamoDB, note that for now the data is loaded into memory before being pushed to DynamoDB, so this operator should be used for smallish amount of data.
Parameters: |
|
---|
class airflow.contrib.operators.jenkins_job_trigger_operator.JenkinsJobTriggerOperator(jenkins_connection_id, job_name, parameters='', sleep_time=10, max_try_before_job_appears=10, *args, **kwargs)
Bases: airflow.models.BaseOperator
Trigger a Jenkins Job and monitor it’s execution. This operator depend on python-jenkins library, version >= 0.4.15 to communicate with jenkins server. You’ll also need to configure a Jenkins connection in the connections screen. :param jenkins_connection_id: The jenkins connection to use for this job :type jenkins_connection_id: string :param job_name: The name of the job to trigger :type job_name: string :param parameters: The parameters block to provide to jenkins. (templated) :type parameters: string :param sleep_time: How long will the operator sleep between each status request for the job (min 1, default 10) :type sleep_time: int :param max_try_before_job_appears: The maximum number of requests to make
while waiting for the job to appears on jenkins server (default 10)
build_job(jenkins_server)
This function makes an API call to Jenkins to trigger a build for ‘job_name’ It returned a dict with 2 keys : body and headers. headers contains also a dict-like object which can be queried to get the location to poll in the queue. :param jenkins_server: The jenkins server where the job should be triggered :return: Dict containing the response body (key body) and the headers coming along (headers)
poll_job_in_queue(location, jenkins_server)
This method poll the jenkins queue until the job is executed. When we trigger a job through an API call, the job is first put in the queue without having a build number assigned. Thus we have to wait the job exit the queue to know its build number. To do so, we have to add /api/json (or /api/xml) to the location returned by the build_job call and poll this file. When a ‘executable’ block appears in the json, it means the job execution started and the field ‘number’ then contains the build number. :param location: Location to poll, returned in the header of the build_job call :param jenkins_server: The jenkins server to poll :return: The build_number corresponding to the triggered job
class airflow.contrib.operators.jira_operator.JiraOperator(jira_conn_id='jira_default', jira_method=None, jira_method_args=None, result_processor=None, get_jira_resource_method=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
JiraOperator to interact and perform action on Jira issue tracking system. This operator is designed to use Jira Python SDK: http://jira.readthedocs.io
Parameters: |
|
---|
class airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator(namespace, image, name, cmds=None, arguments=None, volume_mounts=None, volumes=None, env_vars=None, secrets=None, in_cluster=False, cluster_context=None, labels=None, startup_timeout_seconds=120, get_logs=True, image_pull_policy='IfNotPresent', annotations=None, resources=None, affinity=None, config_file=None, xcom_push=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Execute a task in a Kubernetes Pod
Parameters: |
|
---|---|
Param: | namespace: the namespace to run within kubernetes |
Type: | namespace: str |
class airflow.contrib.operators.mlengine_operator.MLEngineBatchPredictionOperator(project_id, job_id, region, data_format, input_paths, output_path, model_name=None, version_name=None, uri=None, max_worker_count=None, runtime_version=None, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Start a Google Cloud ML Engine prediction job.
NOTE: For model origin, users should consider exactly one from the three options below: 1. Populate ‘uri’ field only, which should be a GCS location that points to a tensorflow savedModel directory. 2. Populate ‘model_name’ field only, which refers to an existing model, and the default version of the model will be used. 3. Populate both ‘model_name’ and ‘version_name’ fields, which refers to a specific version of a specific model.
In options 2 and 3, both model and version name should contain the minimal identifier. For instance, call
MLEngineBatchPredictionOperator(
...,
model_name='my_model',
version_name='my_version',
...)
if the desired model version is “projects/my_project/models/my_model/versions/my_version”.
See https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs for further documentation on the parameters.
Parameters: |
|
---|
Raises:
ValueError
: if a unique model/version origin cannot be determined.
class airflow.contrib.operators.mlengine_operator.MLEngineModelOperator(project_id, model, operation='create', gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Operator for managing a Google Cloud ML Engine model.
Parameters: |
|
---|
class airflow.contrib.operators.mlengine_operator.MLEngineVersionOperator(project_id, model_name, version_name=None, version=None, operation='create', gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Operator for managing a Google Cloud ML Engine version.
Parameters: |
|
---|
class airflow.contrib.operators.mlengine_operator.MLEngineTrainingOperator(project_id, job_id, package_uris, training_python_module, training_args, region, scale_tier=None, runtime_version=None, python_version=None, job_dir=None, gcp_conn_id='google_cloud_default', delegate_to=None, mode='PRODUCTION', *args, **kwargs)
Bases: airflow.models.BaseOperator
Operator for launching a MLEngine training job.
Parameters: |
|
---|
class airflow.contrib.operators.mongo_to_s3.MongoToS3Operator(mongo_conn_id, s3_conn_id, mongo_collection, mongo_query, s3_bucket, s3_key, mongo_db=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Mongo -> S3
A more specific baseOperator meant to move data from mongo via pymongo to s3 via boto
things to note.execute() is written to depend on .transform() .transform() is meant to be extended by child classes to perform transformations unique to those operators needs
execute(context)
Executed by task_instance at runtime
transform(docs)
Processes pyMongo cursor and returns an iterable with each element beinga JSON serializable dictionary
Base transform() assumes no processing is needed ie. docs is a pyMongo cursor of documents and cursor just needs to be passed through
Override this method for custom transformations
class airflow.contrib.operators.mysql_to_gcs.MySqlToGoogleCloudStorageOperator(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, mysql_conn_id='mysql_default', google_cloud_storage_conn_id='google_cloud_default', schema=None, delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Copy data from MySQL to Google cloud storage in JSON format.
classmethod type_map(mysql_type)
Helper function that maps from MySQL fields to BigQuery fields. Used when a schema_filename is set.
class airflow.contrib.operators.postgres_to_gcs_operator.PostgresToGoogleCloudStorageOperator(sql, bucket, filename, schema_filename=None, approx_max_file_size_bytes=1900000000, postgres_conn_id='postgres_default', google_cloud_storage_conn_id='google_cloud_default', delegate_to=None, parameters=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Copy data from Postgres to Google Cloud Storage in JSON format.
classmethod convert_types(value)
Takes a value from Postgres, and converts it to a value that’s safe for JSON/Google Cloud Storage/BigQuery. Dates are converted to UTC seconds. Decimals are converted to floats. Times are converted to seconds.
classmethod type_map(postgres_type)
Helper function that maps from Postgres fields to BigQuery fields. Used when a schema_filename is set.
class airflow.contrib.operators.pubsub_operator.PubSubTopicCreateOperator(project, topic, fail_if_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Create a PubSub topic.
By default, if the topic already exists, this operator will not cause the DAG to fail.
with DAG('successful DAG') as dag:
(
dag
>> PubSubTopicCreateOperator(project='my-project',
topic='my_new_topic')
>> PubSubTopicCreateOperator(project='my-project',
topic='my_new_topic')
)
The operator can be configured to fail if the topic already exists.
with DAG('failing DAG') as dag:
(
dag
>> PubSubTopicCreateOperator(project='my-project',
topic='my_new_topic')
>> PubSubTopicCreateOperator(project='my-project',
topic='my_new_topic',
fail_if_exists=True)
)
Both project
and topic
are templated so you can use
variables in them.
class airflow.contrib.operators.pubsub_operator.PubSubTopicDeleteOperator(project, topic, fail_if_not_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Delete a PubSub topic.
By default, if the topic does not exist, this operator will not cause the DAG to fail.
with DAG('successful DAG') as dag:
(
dag
>> PubSubTopicDeleteOperator(project='my-project',
topic='non_existing_topic')
)
The operator can be configured to fail if the topic does not exist.
with DAG('failing DAG') as dag:
(
dag
>> PubSubTopicCreateOperator(project='my-project',
topic='non_existing_topic',
fail_if_not_exists=True)
)
Both project
and topic
are templated so you can use
variables in them.
class airflow.contrib.operators.pubsub_operator.PubSubSubscriptionCreateOperator(topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Create a PubSub subscription.
By default, the subscription will be created in topic_project
. If
subscription_project
is specified and the GCP credentials allow, the
Subscription can be created in a different project from its topic.
By default, if the subscription already exists, this operator will not cause the DAG to fail. However, the topic must exist in the project.
with DAG('successful DAG') as dag:
(
dag
>> PubSubSubscriptionCreateOperator(
topic_project='my-project', topic='my-topic',
subscription='my-subscription')
>> PubSubSubscriptionCreateOperator(
topic_project='my-project', topic='my-topic',
subscription='my-subscription')
)
The operator can be configured to fail if the subscription already exists.
with DAG('failing DAG') as dag:
(
dag
>> PubSubSubscriptionCreateOperator(
topic_project='my-project', topic='my-topic',
subscription='my-subscription')
>> PubSubSubscriptionCreateOperator(
topic_project='my-project', topic='my-topic',
subscription='my-subscription', fail_if_exists=True)
)
Finally, subscription is not required. If not passed, the operator will generated a universally unique identifier for the subscription’s name.
with DAG('DAG') as dag:
(
dag >> PubSubSubscriptionCreateOperator(
topic_project='my-project', topic='my-topic')
)
topic_project
, topic
, subscription
, and
subscription
are templated so you can use variables in them.
class airflow.contrib.operators.pubsub_operator.PubSubSubscriptionDeleteOperator(project, subscription, fail_if_not_exists=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Delete a PubSub subscription.
By default, if the subscription does not exist, this operator will not cause the DAG to fail.
with DAG('successful DAG') as dag:
(
dag
>> PubSubSubscriptionDeleteOperator(project='my-project',
subscription='non-existing')
)
The operator can be configured to fail if the subscription already exists.
with DAG('failing DAG') as dag:
(
dag
>> PubSubSubscriptionDeleteOperator(
project='my-project', subscription='non-existing',
fail_if_not_exists=True)
)
project
, and subscription
are templated so you can use
variables in them.
class airflow.contrib.operators.pubsub_operator.PubSubPublishOperator(project, topic, messages, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Publish messages to a PubSub topic.
Each Task publishes all provided messages to the same topic in a single GCP project. If the topic does not exist, this task will fail.
from base64 import b64encode as b64e
m1 = {'data': b64e('Hello, World!'),
'attributes': {'type': 'greeting'}
}
m2 = {'data': b64e('Knock, knock')}
m3 = {'attributes': {'foo': ''}}
t1 = PubSubPublishOperator(
project='my-project',topic='my_topic',
messages=[m1, m2, m3],
create_topic=True,
dag=dag)
``project`` , ``topic``, and ``messages`` are templated so you can use
variables in them.
class airflow.contrib.operators.qubole_check_operator.QuboleCheckOperator(qubole_conn_id='qubole_default', *args, **kwargs)
Bases: airflow.operators.check_operator.CheckOperator
, airflow.contrib.operators.qubole_operator.QuboleOperator
Performs checks against Qubole Commands. QuboleCheckOperator
expects
a command that will be executed on QDS.
By default, each value on first row of the result of this Qubole Commmand
is evaluated using python bool
casting. If any of the
values return False
, the check is failed and errors out.
Note that Python bool casting evals the following as False
:
False
0
""
)[]
){}
)Given a query like SELECT COUNT(*) FROM foo
, it will fail only if
the count == 0
. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today’s partition is
greater than yesterday’s partition, or that a set of metrics are less
than 3 standard deviation for the 7 day average.
This operator can be used as a data quality check in your pipeline, and depending on where you put it in your DAG, you have the choice to stop the critical path, preventing from publishing dubious data, or on the side and receive email alerts without stopping the progress of the DAG.
Parameters: | qubole_conn_id (str) – Connection id which consists of qds auth_token |
---|
kwargs:
Arguments specific to Qubole command can be referred from QuboleOperator docs.
results_parser_callable: This is an optional parameter to extend the flexibility of parsing the results of Qubole command to the users. This is a python callable which can hold the logic to parse list of rows returned by Qubole command. By default, only the values on first row are used for performing checks. This callable should return a list of records on which the checks have to be performed.
Note
All fields in common with template fields of QuboleOperator and CheckOperator are template-supported.
class airflow.contrib.operators.qubole_check_operator.QuboleValueCheckOperator(pass_value, tolerance=None, qubole_conn_id='qubole_default', *args, **kwargs)
Bases: airflow.operators.check_operator.ValueCheckOperator
, airflow.contrib.operators.qubole_operator.QuboleOperator
Performs a simple value check using Qubole command. By default, each value on the first row of this Qubole command is compared with a pre-defined value. The check fails and errors out if the output of the command is not within the permissible limit of expected value.
Parameters: |
|
---|
kwargs:
Arguments specific to Qubole command can be referred from QuboleOperator docs.
results_parser_callable: This is an optional parameter to extend the flexibility of parsing the results of Qubole command to the users. This is a python callable which can hold the logic to parse list of rows returned by Qubole command. By default, only the values on first row are used for performing checks. This callable should return a list of records on which the checks have to be performed.
Note
All fields in common with template fields of QuboleOperator and ValueCheckOperator are template-supported.
class airflow.contrib.operators.qubole_operator.QuboleOperator(qubole_conn_id='qubole_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Execute tasks (commands) on QDS (https://qubole.com).
Parameters: | qubole_conn_id (str) – Connection id which consists of qds auth_token |
---|
kwargs:
command_type: | type of command to be executed, e.g. hivecmd, shellcmd, hadoopcmd |
---|---|
tags: | array of tags to be assigned with the command |
cluster_label: | cluster label on which the command will be executed |
name: | name to be given to command |
notify: | whether to send email on command completion or not (default is False) |
Arguments specific to command types
hivecmd:
query: | inline query statement |
---|---|
script_location: | |
s3 location containing query statement | |
sample_size: | size of sample in bytes on which to run query |
macros: | macro values which were used in query |
prestocmd:
query: | inline query statement |
---|---|
script_location: | |
s3 location containing query statement | |
macros: | macro values which were used in query |
hadoopcmd:
sub_commnad: | must be one these [“jar”, “s3distcp”, “streaming”] followed by 1 or more args |
---|
shellcmd:
script: | inline command with args |
---|---|
script_location: | |
s3 location containing query statement | |
files: | list of files in s3 bucket as file1,file2 format. These files will be copied into the working directory where the qubole command is being executed. |
archives: | list of archives in s3 bucket as archive1,archive2 format. These will be unarchived intothe working directory where the qubole command is being executed |
parameters: | any extra args which need to be passed to script (only when script_location is supplied) |
pigcmd:
script: | inline query statement (latin_statements) |
---|---|
script_location: | |
s3 location containing pig query | |
parameters: | any extra args which need to be passed to script (only when script_location is supplied |
sparkcmd:
program: | the complete Spark Program in Scala, SQL, Command, R, or Python |
---|---|
cmdline: | spark-submit command line, all required information must be specify in cmdline itself. |
sql: | inline sql query |
script_location: | |
s3 location containing query statement | |
language: | language of the program, Scala, SQL, Command, R, or Python |
app_id: | ID of an Spark job server app |
arguments: | spark-submit command line arguments |
user_program_arguments: | |
arguments that the user program takes in | |
macros: | macro values which were used in query |
dbtapquerycmd:
db_tap_id: | data store ID of the target database, in Qubole. |
---|---|
query: | inline query statement |
macros: | macro values which were used in query |
dbexportcmd:
mode: | 1 (simple), 2 (advance) |
---|---|
hive_table: | Name of the hive table |
partition_spec: | partition specification for Hive table. |
dbtap_id: | data store ID of the target database, in Qubole. |
db_table: | name of the db table |
db_update_mode: | allowinsert or updateonly |
db_update_keys: | columns used to determine the uniqueness of rows |
export_dir: | HDFS/S3 location from which data will be exported. |
fields_terminated_by: | |
hex of the char used as column separator in the dataset |
dbimportcmd:
mode: | 1 (simple), 2 (advance) |
---|---|
hive_table: | Name of the hive table |
dbtap_id: | data store ID of the target database, in Qubole. |
db_table: | name of the db table |
where_clause: | where clause, if any |
parallelism: | number of parallel db connections to use for extracting data |
extract_query: | SQL query to extract data from db. $CONDITIONS must be part of the where clause. |
boundary_query: | Query to be used get range of row IDs to be extracted |
split_column: | Column used as row ID to split data into ranges (mode 2) |
Note
Following fields are template-supported : query
, script_location
,
sub_command
, script
, files
, archives
, program
, cmdline
,
sql
, where_clause
, extract_query
, boundary_query
, macros
,
tags
, name
, parameters
, dbtap_id
, hive_table
, db_table
,
split_column
, note_id
, db_update_keys
, export_dir
,
partition_spec
, qubole_conn_id
, arguments
, user_program_arguments
.
You can also use.txt
files for template driven use cases.
Note
In QuboleOperator there is a default handler for task failures and retries, which generally kills the command running at QDS for the corresponding task instance. You can override this behavior by providing your own failure and retry handler in task definition.
class airflow.contrib.operators.s3_list_operator.S3ListOperator(bucket, prefix='', delimiter='', aws_conn_id='aws_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
List all objects from the bucket with the given string prefix in name.
This operator returns a python list with the name of objects which can be used by xcom in the downstream task.
Parameters: |
|
---|
Example:
The following operator would list all the files
(excluding subfolders) from the S3
customers/2018/04/
key in the data
bucket.
s3_file = S3ListOperator(
task_id='list_3s_files',
bucket='data',
prefix='customers/2018/04/',
delimiter='/',
aws_conn_id='aws_customers_conn'
)
class airflow.contrib.operators.s3_to_gcs_operator.S3ToGoogleCloudStorageOperator(bucket, prefix='', delimiter='', aws_conn_id='aws_default', dest_gcs_conn_id=None, dest_gcs=None, delegate_to=None, replace=False, *args, **kwargs)
Bases: airflow.contrib.operators.s3_list_operator.S3ListOperator
Synchronizes an S3 key, possibly a prefix, with a Google Cloud Storage destination path.
Parameters: |
|
---|
Example: .. code-block:: python
s3_to_gcs_op = S3ToGoogleCloudStorageOperator(task_id=’s3_to_gcs_example’, bucket=’my-s3-bucket’, prefix=’data/customers-201804’, dest_gcs_conn_id=’google_cloud_default’, dest_gcs=’gs://my.gcs.bucket/some/customers/’, replace=False, dag=my-dag)
Note that bucket
, prefix
, delimiter
and dest_gcs
are
templated, so you can use variables in them if you wish.
class airflow.contrib.operators.segment_track_event_operator.SegmentTrackEventOperator(user_id, event, properties=None, segment_conn_id='segment_default', segment_debug_mode=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
Send Track Event to Segment for a specified user_id and event
Parameters: |
|
---|
class airflow.contrib.operators.sftp_operator.SFTPOperator(ssh_hook=None, ssh_conn_id=None, remote_host=None, local_filepath=None, remote_filepath=None, operation='put', *args, **kwargs)
Bases: airflow.models.BaseOperator
SFTPOperator for transferring files from remote host to local or vice a versa. This operator uses ssh_hook to open sftp trasport channel that serve as basis for file transfer.
Parameters: |
|
---|
class airflow.contrib.operators.slack_webhook_operator.SlackWebhookOperator(http_conn_id=None, webhook_token=None, message='', channel=None, username=None, icon_emoji=None, link_names=False, proxy=None, *args, **kwargs)
Bases: airflow.operators.http_operator.SimpleHttpOperator
This operator allows you to post messages to Slack using incoming webhooks. Takes both Slack webhook token directly and connection that has Slack webhook token. If both supplied, Slack webhook token will be used.
Each Slack webhook token can be pre-configured to use a specific channel, username and icon. You can override these defaults in this hook.
Parameters: |
|
---|
execute(context)
Call the SparkSqlHook to run the provided sql query
class airflow.contrib.operators.snowflake_operator.SnowflakeOperator(sql, snowflake_conn_id='snowflake_default', parameters=None, autocommit=True, warehouse=None, database=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a Snowflake database
Parameters: |
|
---|
class airflow.contrib.operators.spark_jdbc_operator.SparkJDBCOperator(spark_app_name='airflow-spark-jdbc', spark_conn_id='spark-default', spark_conf=None, spark_py_files=None, spark_files=None, spark_jars=None, num_executors=None, executor_cores=None, executor_memory=None, driver_memory=None, verbose=False, keytab=None, principal=None, cmd_type='spark_to_jdbc', jdbc_table=None, jdbc_conn_id='jdbc-default', jdbc_driver=None, metastore_table=None, jdbc_truncate=False, save_mode=None, save_format=None, batch_size=None, fetch_size=None, num_partitions=None, partition_column=None, lower_bound=None, upper_bound=None, create_table_column_types=None, *args, **kwargs)
Bases: airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator
This operator extends the SparkSubmitOperator specifically for performing data transfers to/from JDBC-based databases with Apache Spark. As with the SparkSubmitOperator, it assumes that the “spark-submit” binary is available on the PATH.
Parameters: |
|
---|---|
Type: | jdbc_conn_id: str |
execute(context)
Call the SparkSubmitHook to run the provided spark job
class airflow.contrib.operators.spark_sql_operator.SparkSqlOperator(sql, conf=None, conn_id='spark_sql_default', total_executor_cores=None, executor_cores=None, executor_memory=None, keytab=None, principal=None, master='yarn', name='default-name', num_executors=None, yarn_queue='default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Execute Spark SQL query
Parameters: |
|
---|
execute(context)
Call the SparkSqlHook to run the provided sql query
class airflow.contrib.operators.spark_submit_operator.SparkSubmitOperator(application='', conf=None, conn_id='spark_default', files=None, py_files=None, driver_classpath=None, jars=None, java_class=None, packages=None, exclude_packages=None, repositories=None, total_executor_cores=None, executor_cores=None, executor_memory=None, driver_memory=None, keytab=None, principal=None, name='airflow-spark', num_executors=None, application_args=None, env_vars=None, verbose=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
This hook is a wrapper around the spark-submit binary to kick off a spark-submit job. It requires that the “spark-submit” binary is in the PATH or the spark-home is set in the extra on the connection.
Parameters: |
|
---|
execute(context)
Call the SparkSubmitHook to run the provided spark job
class airflow.contrib.operators.sqoop_operator.SqoopOperator(conn_id='sqoop_default', cmd_type='import', table=None, query=None, target_dir=None, append=None, file_type='text', columns=None, num_mappers=None, split_by=None, where=None, export_dir=None, input_null_string=None, input_null_non_string=None, staging_table=None, clear_staging_table=False, enclosed_by=None, escaped_by=None, input_fields_terminated_by=None, input_lines_terminated_by=None, input_optionally_enclosed_by=None, batch=False, direct=False, driver=None, verbose=False, relaxed_isolation=False, properties=None, hcatalog_database=None, hcatalog_table=None, create_hcatalog_table=False, extra_import_options=None, extra_export_options=None, *args, **kwargs)
Bases: airflow.models.BaseOperator
Execute a Sqoop job. Documentation for Apache Sqoop can be found here:
execute(context)
Execute sqoop job
class airflow.contrib.operators.ssh_operator.SSHOperator(ssh_hook=None, ssh_conn_id=None, remote_host=None, command=None, timeout=10, do_xcom_push=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
SSHOperator to execute commands on given remote host using the ssh_hook.
Parameters: |
|
---|
class airflow.contrib.operators.vertica_operator.VerticaOperator(sql, vertica_conn_id='vertica_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Executes sql code in a specific Vertica database
Parameters: |
|
---|
class airflow.contrib.operators.vertica_to_hive.VerticaToHiveTransfer(sql, hive_table, create=True, recreate=False, partition=None, delimiter=u'x01', vertica_conn_id='vertica_default', hive_cli_conn_id='hive_cli_default', *args, **kwargs)
Bases: airflow.models.BaseOperator
Moves data from Vertia to Hive. The operator runs
your query against Vertia, stores the file locally
before loading it into a Hive table. If the create
or
recreate
arguments are set to True
,
a CREATE TABLE
and DROP TABLE
statements are generated.
Hive data types are inferred from the cursor’s metadata.
Note that the table generated in Hive uses STORED AS textfile
which isn’t the most efficient serialization format. If a
large amount of data is loaded and/or if the table gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a HiveOperator
.
Parameters: |
|
---|
class airflow.contrib.operators.winrm_operator.WinRMOperator(winrm_hook=None, ssh_conn_id=None, remote_host=None, command=None, timeout=10, do_xcom_push=False, *args, **kwargs)
Bases: airflow.models.BaseOperator
WinRMOperator to execute commands on given remote host using the winrm_hook.
Parameters: |
|
---|
class airflow.contrib.sensors.aws_redshift_cluster_sensor.AwsRedshiftClusterSensor(cluster_identifier, target_status='available', aws_conn_id='aws_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a Redshift cluster to reach a specific status.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.bash_sensor.BashSensor(bash_command, env=None, output_encoding='utf-8', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Executes a bash command/script and returns True if and only if the return code is 0.
Parameters: |
|
---|
poke(context)
Execute the bash command in a temporary directory which will be cleaned afterwards
class airflow.contrib.sensors.bigquery_sensor.BigQueryTableSensor(project_id, dataset_id, table_id, bigquery_conn_id='bigquery_default_conn', delegate_to=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Checks for the existence of a table in Google Bigquery.
param project_id: The Google cloud project in which to look for the table. The connection supplied to the hook must provide access to the specified project. type project_id: string param dataset_id: The name of the dataset in which to look for the table. storage bucket. type dataset_id: string param table_id: The name of the table to check the existence of. type table_id: string param bigquery_conn_id: The connection ID to use when connecting to Google BigQuery. type bigquery_conn_id: string param delegate_to: The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled. type delegate_to: string
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.datadog_sensor.DatadogSensor(datadog_conn_id='datadog_default', from_seconds_ago=3600, up_to_seconds_from_now=0, priority=None, sources=None, tags=None, response_check=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
A sensor to listen, with a filter, to datadog event streams and determine if some event was emitted.
Depends on the datadog API, which has to be deployed on the same server where Airflow runs.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor(aws_conn_id='aws_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Contains general sensor behavior for EMR. Subclasses should implement get_emr_response() and state_from_response() methods. Subclasses should also implement NON_TERMINAL_STATES and FAILED_STATE constants.
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.emr_job_flow_sensor.EmrJobFlowSensor(job_flow_id, *args, **kwargs)
Bases: airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor
Asks for the state of the JobFlow until it reaches a terminal state. If it fails the sensor errors, failing the task.
Parameters: | job_flow_id (string) – job_flow_id to check the state of |
---|
class airflow.contrib.sensors.emr_step_sensor.EmrStepSensor(job_flow_id, step_id, *args, **kwargs)
Bases: airflow.contrib.sensors.emr_base_sensor.EmrBaseSensor
Asks for the state of the step until it reaches a terminal state. If it fails the sensor errors, failing the task.
Parameters: |
|
---|
class airflow.contrib.sensors.file_sensor.FileSensor(filepath, fs_conn_id='fs_default2', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a file or folder to land in a filesystem.
If the path given is a directory then this sensor will only return true if any files exist inside it (either directly, or within a subdirectory)
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.ftp_sensor.FTPSensor(path, ftp_conn_id='ftp_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a file or directory to be present on FTP.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.ftp_sensor.FTPSSensor(path, ftp_conn_id='ftp_default', *args, **kwargs)
Bases: airflow.contrib.sensors.ftp_sensor.FTPSensor
Waits for a file or directory to be present on FTP over SSL.
class airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectSensor(bucket, object, google_cloud_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Checks for the existence of a file in Google Cloud Storage. Create a new GoogleCloudStorageObjectSensor.
param bucket: The Google cloud storage bucket where the object is. type bucket: string param object: The name of the object to check in the Google cloud storage bucket. type object: string param google_cloud_storage_conn_id: The connection ID to use when connecting to Google cloud storage. type google_cloud_storage_conn_id: string param delegate_to: The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled. type delegate_to: string
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.gcs_sensor.GoogleCloudStorageObjectUpdatedSensor(bucket, object, ts_func=<function ts_function>, google_cloud_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Checks if an object is updated in Google Cloud Storage. Create a new GoogleCloudStorageObjectUpdatedSensor.
param bucket: The Google cloud storage bucket where the object is. type bucket: string param object: The name of the object to download in the Google cloud storage bucket. type object: string param ts_func: Callback for defining the update condition. The default callback returns execution_date + schedule_interval. The callback takes the context as parameter. type ts_func: function param google_cloud_storage_conn_id: The connection ID to use when connecting to Google cloud storage. type google_cloud_storage_conn_id: string param delegate_to: The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled. type delegate_to: string
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.gcs_sensor.GoogleCloudStoragePrefixSensor(bucket, prefix, google_cloud_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Checks for the existence of a files at prefix in Google Cloud Storage bucket. Create a new GoogleCloudStorageObjectSensor.
param bucket: The Google cloud storage bucket where the object is. type bucket: string param prefix: The name of the prefix to check in the Google cloud storage bucket. type prefix: string param google_cloud_storage_conn_id: The connection ID to use when connecting to Google cloud storage. type google_cloud_storage_conn_id: string param delegate_to: The account to impersonate, if any. For this to work, the service account making the request must have domain-wide delegation enabled. type delegate_to: string
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.hdfs_sensor.HdfsSensorFolder(be_empty=False, *args, **kwargs)
Bases: airflow.sensors.hdfs_sensor.HdfsSensor
poke(context)
poke for a non empty directory
Returns: | Bool depending on the search criteria |
---|
class airflow.contrib.sensors.hdfs_sensor.HdfsSensorRegex(regex, *args, **kwargs)
Bases: airflow.sensors.hdfs_sensor.HdfsSensor
poke(context)
poke matching files in a directory with self.regex
Returns: | Bool depending on the search criteria |
---|
class airflow.contrib.sensors.jira_sensor.JiraSensor(jira_conn_id='jira_default', method_name=None, method_params=None, result_processor=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Monitors a jira ticket for any change.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.pubsub_sensor.PubSubPullSensor(project, subscription, max_messages=5, return_immediately=False, ack_messages=False, gcp_conn_id='google_cloud_default', delegate_to=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Pulls messages from a PubSub subscription and passes them through XCom.
This sensor operator will pull up to max_messages
messages from the
specified PubSub subscription. When the subscription returns messages,
the poke method’s criteria will be fulfilled and the messages will be
returned from the operator and passed through XCom for downstream tasks.
If ack_messages
is set to True, messages will be immediately
acknowledged before being returned, otherwise, downstream tasks will be
responsible for acknowledging them.
project
and subscription
are templated so you can use
variables in them.
execute(context)
Overridden to allow messages to be passed
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.qubole_sensor.QuboleSensor(data, qubole_conn_id='qubole_default', *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Base class for all Qubole Sensors
Parameters: |
|
---|
Note
Both data
and qubole_conn_id
fields are template-supported. You can
also use .txt
files for template driven use cases.
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.redis_key_sensor.RedisKeySensor(key, redis_conn_id, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Checks for the existence of a key in a Redis database
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.sftp_sensor.SFTPSensor(path, sftp_conn_id='sftp_default', *args, **kwargs)
Bases: airflow.operators.sensors.BaseSensorOperator
Waits for a file or directory to be present on SFTP. :param path: Remote file or directory path :type path: str :param sftp_conn_id: The connection to run the sensor against :type sftp_conn_id: str
poke(context)
Function that the sensors defined while deriving this class should override.
class airflow.contrib.sensors.wasb_sensor.WasbBlobSensor(container_name, blob_name, wasb_conn_id='wasb_default', check_options=None, *args, **kwargs)
Bases: airflow.sensors.base_sensor_operator.BaseSensorOperator
Waits for a blob to arrive on Azure Blob Storage.
Parameters: |
|
---|
poke(context)
Function that the sensors defined while deriving this class should override.
Here’s a list of variables and macros that can be used in templates
The Airflow engine passes a few variables by default that are accessible in all templates
Variable | Description |
---|---|
{{ ds }} |
the execution date as YYYY-MM-DD |
{{ ds_nodash }} |
the execution date as YYYYMMDD |
{{ prev_ds }} |
the previous execution date as YYYY-MM-DD .
if {{ ds }} is 2016-01-08 and schedule_interval is @weekly ,
{{ prev_ds }} will be 2016-01-01 . |
{{ next_ds }} |
the next execution date as YYYY-MM-DD .
if {{ ds }} is 2016-01-01 and schedule_interval is @weekly ,
{{ prev_ds }} will be 2016-01-08 . |
{{ yesterday_ds }} |
yesterday’s date as YYYY-MM-DD |
{{ yesterday_ds_nodash }} |
yesterday’s date as YYYYMMDD |
{{ tomorrow_ds }} |
tomorrow’s date as YYYY-MM-DD |
{{ tomorrow_ds_nodash }} |
tomorrow’s date as YYYYMMDD |
{{ ts }} |
same as execution_date.isoformat() |
{{ ts_nodash }} |
same as ts without - and : |
{{ execution_date }} |
the execution_date, (datetime.datetime) |
{{ prev_execution_date }} |
the previous execution date (if available) (datetime.datetime) |
{{ next_execution_date }} |
the next execution date (datetime.datetime) |
{{ dag }} |
the DAG object |
{{ task }} |
the Task object |
{{ macros }} |
a reference to the macros package, described below |
{{ task_instance }} |
the task_instance object |
{{ end_date }} |
same as {{ ds }} |
{{ latest_date }} |
same as {{ ds }} |
{{ ti }} |
same as {{ task_instance }} |
{{ params }} |
a reference to the user-defined params dictionary which can be overridden by
the dictionary passed through trigger_dag -c if you enabled
dag_run_conf_overrides_params` in ``airflow.cfg |
{{ var.value.my_var }} |
global defined variables represented as a dictionary |
{{ var.json.my_var.path }} |
global defined variables represented as a dictionary with deserialized JSON object, append the path to the key within the JSON object |
{{ task_instance_key_str }} |
a unique, human-readable key to the task instance
formatted {dag_id}_{task_id}_{ds} |
{{ conf }} |
the full configuration object located at
airflow.configuration.conf which
represents the content of your
airflow.cfg |
{{ run_id }} |
the run_id of the current DAG run |
{{ dag_run }} |
a reference to the DagRun object |
{{ test_mode }} |
whether the task instance was called using the CLI’s test subcommand |
Note that you can access the object’s attributes and methods with simple
dot notation. Here are some examples of what is possible:
{{ task.owner }}
, {{ task.task_id }}
, {{ ti.hostname }}
, …
Refer to the models documentation for more information on the objects’
attributes and methods.
The var
template variable allows you to access variables defined in Airflow’s
UI. You can access them as either plain-text or JSON. If you use JSON, you are
also able to walk nested structures, such as dictionaries like:
{{ var.json.my_dict_var.key1 }}
Macros are a way to expose objects to your templates and live under the
macros
namespace in your templates.
A few commonly used libraries and methods are made available.
Variable | Description |
---|---|
macros.datetime |
The standard lib’s datetime.datetime |
macros.timedelta |
The standard lib’s datetime.timedelta |
macros.dateutil |
A reference to the dateutil package |
macros.time |
The standard lib’s time |
macros.uuid |
The standard lib’s uuid |
macros.random |
The standard lib’s random |
Some airflow specific macros are also defined:
airflow.macros.ds_add(ds, days)
Add or subtract days from a YYYY-MM-DD
Parameters: |
|
---|
>>> ds_add('2015-01-01', 5)
'2015-01-06'
>>> ds_add('2015-01-06', -5)
'2015-01-01'
airflow.macros.ds_format(ds, input_format, output_format)
Takes an input string and outputs another string as specified in the output format
Parameters: |
|
---|
>>> ds_format('2015-01-01', "%Y-%m-%d", "%m-%d-%y")
'01-01-15'
>>> ds_format('1/5/2015', "%m/%d/%Y", "%Y-%m-%d")
'2015-01-05'
airflow.macros.random() → x in the interval [0, 1).
airflow.macros.hive.closest_ds_partition(table, ds, before=True, schema='default', metastore_conn_id='metastore_default')
This function finds the date in a list closest to the target date. An optional parameter can be given to get the closest before or after.
Parameters: |
|
---|---|
Returns: | The closest date |
Return type: | str or None |
>>> tbl = 'airflow.static_babynames_partitioned'
>>> closest_ds_partition(tbl, '2015-01-02')
'2015-01-01'
airflow.macros.hive.max_partition(table, schema='default', field=None, filter_map=None, metastore_conn_id='metastore_default')
Gets the max partition for a table.
Parameters: |
|
---|
>>> max_partition('airflow.static_babynames_partitioned')
'2015-01-01'
Models are built on top of the SQLAlchemy ORM Base class, and instances are persisted in the database.
class airflow.models.BaseOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, weight_rule=u'downstream', queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, task_concurrency=None, executor_config=None, inlets=None, outlets=None, *args, **kwargs)
Bases: airflow.utils.log.logging_mixin.LoggingMixin
Abstract base class for all operators. Since operators create objects that become nodes in the dag, BaseOperator contains many recursive methods for dag crawling behavior. To derive this class, you are expected to override the constructor as well as the ‘execute’ method.
Operators derived from this class should perform or trigger certain tasks synchronously (wait for completion). Example of operators could be an operator that runs a Pig job (PigOperator), a sensor operator that waits for a partition to land in Hive (HiveSensorOperator), or one that moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these operators (tasks) target specific operations, running specific scripts, functions or data transfers.
This class is abstract and shouldn’t be instantiated. Instantiating a class derived from this one results in the creation of a task object, which ultimately becomes a node in DAG objects. Task dependencies should be set by using the set_upstream and/or set_downstream methods.
Parameters: |
|
---|
clear(**kwargs)
Clears the state of task instances associated with the task, following the parameters specified.
dag
Returns the Operator’s DAG if set, otherwise raises an error
deps
Returns the list of dependencies for the operator. These differ from execution context dependencies in that they are specific to tasks and can be extended/overridden by subclasses.
downstream_list
@property: list of tasks directly downstream
execute(context)
This is the main method to derive when creating an operator. Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
get_direct_relative_ids(upstream=False)
Get the direct relative ids to the current task, upstream or downstream.
get_direct_relatives(upstream=False)
Get the direct relatives to the current task, upstream or downstream.
get_flat_relative_ids(upstream=False, found_descendants=None)
Get a flat list of relatives’ ids, either upstream or downstream.
get_flat_relatives(upstream=False)
Get a flat list of relatives, either upstream or downstream.
get_task_instances(session, start_date=None, end_date=None)
Get a set of task instance related to this task for a specific date range.
has_dag()
Returns True if the Operator has been assigned to a DAG.
on_kill()
Override this method to cleanup subprocesses when a task instance gets killed. Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up or it will leave ghost processes behind.
post_execute(context, *args, **kwargs)
This hook is triggered right after self.execute() is called. It is passed the execution context and any results returned by the operator.
pre_execute(context, *args, **kwargs)
This hook is triggered right before self.execute() is called.
prepare_template()
Hook that is triggered after the templated fields get replaced by their content. If you need your operator to alter the content of the file before the template is rendered, it should override this method to do so.
render_template(attr, content, context)
Renders a template either from a file or directly in a field, and returns the rendered result.
render_template_from_field(attr, content, context, jinja_env)
Renders a template from a field. If the field is a string, it will simply render the string and return the result. If it is a collection or nested set of collections, it will traverse the structure and render all strings in it.
run(start_date=None, end_date=None, ignore_first_depends_on_past=False, ignore_ti_state=False, mark_success=False)
Run a set of task instances for a date range.
schedule_interval
The schedule interval of the DAG always wins over individual tasks so that tasks within a DAG always line up. The task still needs a schedule_interval as it may not be attached to a DAG.
set_downstream(task_or_task_list)
Set a task or a task list to be directly downstream from the current task.
set_upstream(task_or_task_list)
Set a task or a task list to be directly upstream from the current task.
upstream_list
@property: list of tasks directly upstream
xcom_pull(context, task_ids=None, dag_id=None, key=u'return_value', include_prior_dates=None)
See TaskInstance.xcom_pull()
xcom_push(context, key, value, execution_date=None)
See TaskInstance.xcom_push()
class airflow.models.Chart(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.Connection(conn_id=None, conn_type=None, host=None, login=None, password=None, schema=None, port=None, extra=None, uri=None)
Bases: sqlalchemy.ext.declarative.api.Base
, airflow.utils.log.logging_mixin.LoggingMixin
Placeholder to store information about different database instances connection information. The idea here is that scripts use references to database instances (conn_id) instead of hard coding hostname, logins and passwords when using operators or hooks.
extra_dejson
Returns the extra property by deserializing json.
class airflow.models.DAG(dag_id, description=u'', schedule_interval=datetime.timedelta(1), start_date=None, end_date=None, full_filepath=None, template_searchpath=None, user_defined_macros=None, user_defined_filters=None, default_args=None, concurrency=16, max_active_runs=16, dagrun_timeout=None, sla_miss_callback=None, default_view=u'tree', orientation='LR', catchup=True, on_success_callback=None, on_failure_callback=None, params=None)
Bases: airflow.dag.base_dag.BaseDag
, airflow.utils.log.logging_mixin.LoggingMixin
A dag (directed acyclic graph) is a collection of tasks with directional dependencies. A dag also has a schedule, a start end an end date (optional). For each schedule, (say daily or hourly), the DAG needs to run each individual tasks as their dependencies are met. Certain tasks have the property of depending on their own past, meaning that they can’t run until their previous schedule (and upstream tasks) are completed.
DAGs essentially act as namespaces for tasks. A task_id can only be added once to a DAG.
Parameters: |
|
---|
add_task(task)
Add a task to the DAG
Parameters: | task (task) – the task you want to add |
---|
add_tasks(tasks)
Add a list of tasks to the DAG
Parameters: | tasks (list of tasks) – a lit of tasks you want to add |
---|
clear(**kwargs)
Clears a set of task instances associated with the current dag for a specified date range.
cli()
Exposes a CLI specific to this DAG
concurrency_reached
Returns a boolean indicating whether the concurrency limit for this DAG has been reached
create_dagrun(**kwargs)
Creates a dag run from this dag including the tasks associated with this dag. Returns the dag run.
Parameters: |
|
---|
static deactivate_stale_dags(*args, **kwargs)
Deactivate any DAGs that were last touched by the scheduler before the expiration date. These DAGs were likely deleted.
Parameters: | expiration_date (datetime) – set inactive DAGs that were touched before this time |
---|---|
Returns: | None |
static deactivate_unknown_dags(*args, **kwargs)
Given a list of known DAGs, deactivate any other DAGs that are marked as active in the ORM
Parameters: | active_dag_ids (list[unicode]) – list of DAG IDs that are active |
---|---|
Returns: | None |
filepath
File location of where the dag object is instantiated
folder
Folder location of where the dag object is instantiated
following_schedule(dttm)
Calculates the following schedule for this dag in local time
Parameters: | dttm – utc datetime |
---|---|
Returns: | utc datetime |
get_active_runs(**kwargs)
Returns a list of dag run execution dates currently running
Parameters: | session – |
---|---|
Returns: | List of execution dates |
get_dagrun(**kwargs)
Returns the dag run for a given execution date if it exists, otherwise none.
Parameters: |
|
---|---|
Returns: | The DagRun if found, otherwise None. |
get_last_dagrun(**kwargs)
Returns the last dag run for this dag, None if there was none. Last dag run can be any type of run eg. scheduled or backfilled. Overridden DagRuns are ignored
get_num_active_runs(**kwargs)
Returns the number of active “running” dag runs
Parameters: |
|
---|---|
Returns: | number greater than 0 for active dag runs |
static get_num_task_instances(*args, **kwargs)
Returns the number of task instances in the given DAG.
Parameters: |
|
---|---|
Returns: | The number of running tasks |
Return type: | int |
get_run_dates(start_date, end_date=None)
Returns a list of dates between the interval received as parameter using this dag’s schedule interval. Returned dates can be used for execution dates.
Parameters: |
|
---|---|
Returns: | a list of dates within the interval following the dag’s schedule |
Return type: | list |
get_template_env()
Returns a jinja2 Environment while taking into account the DAGs template_searchpath, user_defined_macros and user_defined_filters
handle_callback(**kwargs)
Triggers the appropriate callback depending on the value of success, namely the on_failure_callback or on_success_callback. This method gets the context of a single TaskInstance part of this DagRun and passes that to the callable along with a ‘reason’, primarily to differentiate DagRun failures. .. note:
The logs end up in $AIRFLOW_HOME/logs/scheduler/latest/PROJECT/DAG_FILE.py.log
Parameters: |
|
---|
is_paused
Returns a boolean indicating whether this DAG is paused
latest_execution_date
Returns the latest date for which at least one dag run exists
normalize_schedule(dttm)
Returns dttm + interval unless dttm is first interval then it returns dttm
previous_schedule(dttm)
Calculates the previous schedule for this dag in local time
Parameters: | dttm – utc datetime |
---|---|
Returns: | utc datetime |
run(start_date=None, end_date=None, mark_success=False, local=False, executor=None, donot_pickle=False, ignore_task_deps=False, ignore_first_depends_on_past=False, pool=None, delay_on_limit_secs=1.0, verbose=False, conf=None, rerun_failed_tasks=False)
Runs the DAG.
Parameters: |
|
---|
set_dependency(upstream_task_id, downstream_task_id)
Simple utility method to set dependency between two tasks that already have been added to the DAG using add_task()
sub_dag(task_regex, include_downstream=False, include_upstream=True)
Returns a subset of the current dag as a deep copy of the current dag based on a regex that should match one or many tasks, and includes upstream and downstream neighbours based on the flag passed.
subdags
Returns a list of the subdag objects associated to this DAG
sync_to_db(**kwargs)
Save attributes about this DAG to the DB. Note that this method can be called for both DAGs and SubDAGs. A SubDag is actually a SubDagOperator.
Parameters: |
|
---|---|
Returns: | None |
test_cycle()
Check to see if there are any cycles in the DAG. Returns False if no cycle found, otherwise raises exception.
topological_sort()
Sorts tasks in topographical order, such that a task comes after any of its upstream dependencies.
Heavily inspired by: http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/
Returns: | list of tasks in topological order |
---|
tree_view()
Shows an ascii tree representation of the DAG
class airflow.models.DagBag(dag_folder=None, executor=None, include_examples=False)
Bases: airflow.dag.base_dag.BaseDagBag
, airflow.utils.log.logging_mixin.LoggingMixin
A dagbag is a collection of dags, parsed out of a folder tree and has high level configuration settings, like what database to use as a backend and what executor to use to fire off tasks. This makes it easier to run distinct environments for say production and development, tests, or for different teams or security profiles. What would have been system level settings are now dagbag level so that one system can run multiple, independent settings sets.
Parameters: |
|
---|
bag_dag(dag, parent_dag, root_dag)
Adds the DAG into the bag, recurses into sub dags. Throws AirflowDagCycleException if a cycle is detected in this dag or its subdags
collect_dags(dag_folder=None, only_if_updated=True)
Given a file path or a folder, this method looks for python modules, imports them and adds them to the dagbag collection.
Note that if a .airflowignore file is found while processing, the directory, it will behaves much like a .gitignore does, ignoring files that match any of the regex patterns specified in the file. Note: The patterns in .airflowignore are treated as un-anchored regexes, not shell-like glob patterns.
dagbag_report()
Prints a report around DagBag loading stats
get_dag(dag_id)
Gets the DAG out of the dictionary, and refreshes it if expired
kill_zombies(**kwargs)
Fails tasks that haven’t had a heartbeat in too long
process_file(filepath, only_if_updated=True, safe_mode=True)
Given a path to a python module or zip file, this method imports the module and look for dag objects within it.
size()
Returns: | the amount of dags contained in this dagbag |
---|
class airflow.models.DagModel(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.DagPickle(dag)
Bases: sqlalchemy.ext.declarative.api.Base
Dags can originate from different places (user repos, master repo, …) and also get executed in different places (different executors). This object represents a version of a DAG and becomes a source of truth for a BackfillJob execution. A pickle is a native python serialized object, and in this case gets stored in the database for the duration of the job.
The executors pick up the DagPickle id and read the dag definition from the database.
class airflow.models.DagRun(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
, airflow.utils.log.logging_mixin.LoggingMixin
DagRun describes an instance of a Dag. It can be created by the scheduler (for regular runs) or by an external trigger
static find(*args, **kwargs)
Returns a set of dag runs for the given search criteria.
Parameters: |
|
---|
Defaults to False :type no_backfills: bool :param session: database session :type session: Session
get_dag()
Returns the Dag associated with this DagRun.
Returns: | DAG |
---|
classmethod get_latest_runs(**kwargs)
Returns the latest DagRun for each DAG.
get_previous_dagrun(**kwargs)
The previous DagRun, if there is one
get_previous_scheduled_dagrun(**kwargs)
The previous, SCHEDULED DagRun, if there is one
static get_run(session, dag_id, execution_date)
Parameters: |
|
---|---|
Returns: | DagRun corresponding to the given dag_id and execution date |
if one exists. None otherwise. :rtype: DagRun
get_task_instance(**kwargs)
Returns the task instance specified by task_id for this dag run
Parameters: | task_id – the task id |
---|
get_task_instances(**kwargs)
Returns the task instances for this dag run
refresh_from_db(**kwargs)
Reloads the current dagrun from the database :param session: database session
update_state(**kwargs)
Determines the overall state of the DagRun based on the state of its TaskInstances.
Returns: | State |
---|
verify_integrity(**kwargs)
Verifies the DagRun by checking for removed tasks or tasks that are not in the database yet. It will set state to removed or add the task if required.
class airflow.models.DagStat(dag_id, state, count=0, dirty=False)
Bases: sqlalchemy.ext.declarative.api.Base
static create(*args, **kwargs)
Creates the missing states the stats table for the dag specified
Parameters: |
|
---|---|
Returns: |
static set_dirty(*args, **kwargs)
Parameters: |
|
---|---|
Returns: |
static update(*args, **kwargs)
Updates the stats for dirty/out-of-sync dags
Parameters: |
|
---|
class airflow.models.ImportError(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
exception airflow.models.InvalidFernetToken
Bases: exceptions.Exception
class airflow.models.KnownEvent(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.KnownEventType(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.KubeResourceVersion(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.KubeWorkerIdentifier(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.Log(event, task_instance, owner=None, extra=None, **kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
Used to actively log events to the database
class airflow.models.NullFernet
Bases: future.types.newobject.newobject
A “Null” encryptor class that doesn’t encrypt or decrypt but that presents a similar interface to Fernet.
The purpose of this is to make the rest of the code not have to know the difference, and to only display the message once, not 20 times when airflow initdb is ran.
class airflow.models.Pool(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
open_slots(**kwargs)
Returns the number of slots open at the moment
queued_slots(**kwargs)
Returns the number of slots used at the moment
used_slots(**kwargs)
Returns the number of slots used at the moment
class airflow.models.SlaMiss(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
Model that stores a history of the SLA that have been missed. It is used to keep track of SLA failures over time and to avoid double triggering alert emails.
class airflow.models.TaskFail(task, execution_date, start_date, end_date)
Bases: sqlalchemy.ext.declarative.api.Base
TaskFail tracks the failed run durations of each task instance.
class airflow.models.TaskInstance(task, execution_date, state=None)
Bases: sqlalchemy.ext.declarative.api.Base
, airflow.utils.log.logging_mixin.LoggingMixin
Task instances store the state of a task instance. This table is the authority and single source of truth around what tasks have run and the state they are in.
The SqlAlchemy model doesn’t have a SqlAlchemy foreign key to the task or dag model deliberately to have more control over transactions.
Database transactions on this table should insure double triggers and any confusion around what task instances are or aren’t ready to run even while multiple schedulers may be firing task instances.
are_dependencies_met(**kwargs)
Returns whether or not all the conditions are met for this task instance to be run given the context for the dependencies (e.g. a task instance being force run from the UI will ignore some dependencies).
Parameters: |
|
---|
are_dependents_done(**kwargs)
Checks whether the dependents of this task instance have all succeeded. This is meant to be used by wait_for_downstream.
This is useful when you do not want to start processing the next schedule of a task until the dependents are done. For instance, if the task DROPs and recreates a table.
clear_xcom_data(**kwargs)
Clears all XCom data from the database for the task instance
command(mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)
Returns a command that can be executed anywhere where airflow is installed. This command is part of the message sent to executors by the orchestrator.
command_as_list(mark_success=False, ignore_all_deps=False, ignore_task_deps=False, ignore_depends_on_past=False, ignore_ti_state=False, local=False, pickle_id=None, raw=False, job_id=None, pool=None, cfg_path=None)
Returns a command that can be executed anywhere where airflow is installed. This command is part of the message sent to executors by the orchestrator.
current_state(**kwargs)
Get the very latest state from the database, if a session is passed, we use and looking up the state becomes part of the session, otherwise a new session is used.
error(**kwargs)
Forces the task instance’s state to FAILED in the database.
static generate_command(dag_id, task_id, execution_date, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, local=False, pickle_id=None, file_path=None, raw=False, job_id=None, pool=None, cfg_path=None)
Generates the shell command required to execute this task instance.
Parameters: |
|
---|---|
Returns: | shell command that can be used to run the task instance |
get_dagrun(**kwargs)
Returns the DagRun for this TaskInstance
Parameters: | session – |
---|---|
Returns: | DagRun |
init_on_load()
Initialize the attributes that aren’t stored in the DB.
init_run_context(raw=False)
Sets the log context.
is_eligible_to_retry()
Is task instance is eligible for retry
is_premature
Returns whether a task is in UP_FOR_RETRY state and its retry interval has elapsed.
key
Returns a tuple that identifies the task instance uniquely
next_retry_datetime()
Get datetime of the next retry if the task instance fails. For exponential backoff, retry_delay is used as base and will be converted to seconds.
pool_full(**kwargs)
Returns a boolean as to whether the slot pool has room for this task to run
previous_ti
The task instance for the task that ran before this task instance
ready_for_retry()
Checks on whether the task instance is in the right state and timeframe to be retried.
refresh_from_db(**kwargs)
Refreshes the task instance from the database based on the primary key
Parameters: | lock_for_update – if True, indicates that the database should lock the TaskInstance (issuing a FOR UPDATE clause) until the session is committed. |
---|
try_number
Return the try number that this task number will be when it is acutally run.
If the TI is currently running, this will match the column in the databse, in all othercases this will be incremenetd
xcom_pull(task_ids=None, dag_id=None, key=u'return_value', include_prior_dates=False)
Pull XComs that optionally meet certain criteria.
The default value for key limits the search to XComs that were returned by other tasks (as opposed to those that were pushed manually). To remove this filter, pass key=None (or any desired value).
If a single task_id string is provided, the result is the value of the most recent matching XCom from that task_id. If multiple task_ids are provided, a tuple of matching values is returned. None is returned whenever no matches are found.
Parameters: |
|
---|
xcom_push(key, value, execution_date=None)
Make an XCom available for tasks to pull.
Parameters: |
|
---|
class airflow.models.User(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
class airflow.models.Variable(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
, airflow.utils.log.logging_mixin.LoggingMixin
classmethod setdefault(key, default, deserialize_json=False)
Like a Python builtin dict object, setdefault returns the current value for a key, and if it isn’t there, stores the default value and returns it.
Parameters: |
|
---|
isn’t already in the DB :type default: Mixed :param deserialize_json: Store this as a JSON encoded value in the DB
and un-encode it when retrieving a value
Returns: | Mixed |
---|
class airflow.models.XCom(**kwargs)
Bases: sqlalchemy.ext.declarative.api.Base
, airflow.utils.log.logging_mixin.LoggingMixin
Base class for XCom objects.
classmethod get_many(**kwargs)
Retrieve an XCom value, optionally meeting certain criteria TODO: “pickling” has been deprecated and JSON is preferred.
“pickling” will be removed in Airflow 2.0.
classmethod get_one(**kwargs)
Retrieve an XCom value, optionally meeting certain criteria. TODO: “pickling” has been deprecated and JSON is preferred.
“pickling” will be removed in Airflow 2.0.
Returns: | XCom value |
---|
classmethod set(**kwargs)
Store an XCom value. TODO: “pickling” has been deprecated and JSON is preferred.
“pickling” will be removed in Airflow 2.0.
Returns: | None |
---|
airflow.models.clear_task_instances(tis, session, activate_dag_runs=True, dag=None)
Clears a set of task instances, but makes sure the running ones get killed.
Parameters: |
|
---|
airflow.models.get_fernet()
Deferred load of Fernet key.
This function could fail either because Cryptography is not installed or because the Fernet key is invalid.
Returns: | Fernet object |
---|---|
Raises: | AirflowException if there’s a problem trying to load Fernet |
Hooks are interfaces to external platforms and databases, implementing a common interface when possible and acting as building blocks for operators.
class airflow.hooks.dbapi_hook.DbApiHook(*args, **kwargs)
Bases: airflow.hooks.base_hook.BaseHook
Abstract base class for sql hooks.
bulk_dump(table, tmp_file)
Dumps a database table into a tab-delimited file
Parameters: |
|
---|
bulk_load(table, tmp_file)
Loads a tab-delimited file into a database table
Parameters: |
|
---|
get_autocommit(conn)
Get autocommit setting for the provided connection. Return True if conn.autocommit is set to True. Return False if conn.autocommit is not set or set to False or conn does not support autocommit. :param conn: Connection to get autocommit setting from. :type conn: connection object. :return: connection autocommit setting. :rtype bool.
get_conn()
Returns a connection object
get_cursor()
Returns a cursor
get_first(sql, parameters=None)
Executes the sql and returns the first resulting row.
Parameters: |
|
---|
get_pandas_df(sql, parameters=None)
Executes the sql and returns a pandas dataframe
Parameters: |
|
---|
get_records(sql, parameters=None)
Executes the sql and returns a set of records.
Parameters: |
|
---|
insert_rows(table, rows, target_fields=None, commit_every=1000, replace=False)
A generic way to insert a set of tuples into a table, a new transaction is created every commit_every rows
Parameters: |
|
---|
run(sql, autocommit=False, parameters=None)
Runs a command or a list of commands. Pass a list of sql statements to the sql parameter to get them to execute sequentially
Parameters: |
|
---|
set_autocommit(conn, autocommit)
Sets the autocommit flag on the connection
class airflow.hooks.docker_hook.DockerHook(docker_conn_id='docker_default', base_url=None, version=None, tls=None)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Interact with a private Docker registry.
Parameters: | docker_conn_id (str) – ID of the Airflow connection where credentials and extra configuration are stored |
---|
class airflow.hooks.hive_hooks.HiveCliHook(hive_cli_conn_id=u'hive_cli_default', run_as=None, mapred_queue=None, mapred_queue_priority=None, mapred_job_name=None)
Bases: airflow.hooks.base_hook.BaseHook
Simple wrapper around the hive CLI.
It also supports the beeline
a lighter CLI that runs JDBC and is replacing the heavier
traditional CLI. To enable beeline
, set the use_beeline param in the
extra field of your connection as in { "use_beeline": true }
Note that you can also set default hive CLI parameters using the
hive_cli_params
to be used in your connection as in
{"hive_cli_params": "-hiveconf mapred.job.tracker=some.jobtracker:444"}
Parameters passed here can be overridden by run_cli’s hive_conf param
The extra connection parameter auth
gets passed as in the jdbc
connection string as is.
Parameters: |
|
---|
load_df(df, table, field_dict=None, delimiter=u', ', encoding=u'utf8', pandas_kwargs=None, **kwargs)
Loads a pandas DataFrame into hive.
Hive data types will be inferred if not passed but column names will not be sanitized.
Parameters: |
|
---|
load_file(filepath, table, delimiter=u', ', field_dict=None, create=True, overwrite=True, partition=None, recreate=False, tblproperties=None)
Loads a local file into Hive
Note that the table generated in Hive uses STORED AS textfile
which isn’t the most efficient serialization format. If a
large amount of data is loaded and/or if the tables gets
queried considerably, you may want to use this operator only to
stage the data into a temporary table before loading it into its
final destination using a HiveOperator
.
Parameters: |
|
---|
run_cli(hql, schema=None, verbose=True, hive_conf=None)
Run an hql statement using the hive cli. If hive_conf is specified it should be a dict and the entries will be set as key/value pairs in HiveConf
Parameters: | hive_conf (dict) – if specified these key value pairs will be passed
to hive as -hiveconf "key"="value" . Note that they will be
passed after the hive_cli_params and thus will override
whatever values are specified in the database. |
---|
>>> hh = HiveCliHook()
>>> result = hh.run_cli("USE airflow;")
>>> ("OK" in result)
True
test_hql(hql)
Test an hql statement using the hive cli and EXPLAIN
class airflow.hooks.hive_hooks.HiveMetastoreHook(metastore_conn_id=u'metastore_default')
Bases: airflow.hooks.base_hook.BaseHook
Wrapper to interact with the Hive Metastore
check_for_named_partition(schema, table, partition_name)
Checks whether a partition with a given name exists
Parameters: |
|
---|---|
Partition: | Name of the partitions to check for (eg a=b/c=d) |
Return type: | boolean |
>>> hh = HiveMetastoreHook()
>>> t = 'static_babynames_partitioned'
>>> hh.check_for_named_partition('airflow', t, "ds=2015-01-01")
True
>>> hh.check_for_named_partition('airflow', t, "ds=xxx")
False
check_for_partition(schema, table, partition)
Checks whether a partition exists
Parameters: |
|
---|---|
Partition: | Expression that matches the partitions to check for (eg a = ‘b’ AND c = ‘d’) |
Return type: | boolean |
>>> hh = HiveMetastoreHook()
>>> t = 'static_babynames_partitioned'
>>> hh.check_for_partition('airflow', t, "ds='2015-01-01'")
True
get_databases(pattern=u'*')
Get a metastore table object
get_metastore_client()
Returns a Hive thrift client.
get_partitions(schema, table_name, filter=None)
Returns a list of all partitions in a table. Works only for tables with less than 32767 (java short max val). For subpartitioned table, the number might easily exceed this.
>>> hh = HiveMetastoreHook()
>>> t = 'static_babynames_partitioned'
>>> parts = hh.get_partitions(schema='airflow', table_name=t)
>>> len(parts)
1
>>> parts
[{'ds': '2015-01-01'}]
get_table(table_name, db=u'default')
Get a metastore table object
>>> hh = HiveMetastoreHook()
>>> t = hh.get_table(db='airflow', table_name='static_babynames')
>>> t.tableName
'static_babynames'
>>> [col.name for col in t.sd.cols]
['state', 'year', 'name', 'gender', 'num']
get_tables(db, pattern=u'*')
Get a metastore table object
max_partition(schema, table_name, field=None, filter_map=None)
Returns the maximum value for all partitions with given field in a table. If only one partition key exist in the table, the key will be used as field. filter_map should be a partition_key:partition_value map and will be used to filter out partitions.
Parameters: |
|
---|
>>> hh = HiveMetastoreHook()
>>> filter_map = {'ds': '2015-01-01', 'ds': '2014-01-01'}
>>> t = 'static_babynames_partitioned'
>>> hh.max_partition(schema='airflow', ... table_name=t, field='ds', filter_map=filter_map)
'2015-01-01'
table_exists(table_name, db=u'default')
Check if table exists
>>> hh = HiveMetastoreHook()
>>> hh.table_exists(db='airflow', table_name='static_babynames')
True
>>> hh.table_exists(db='airflow', table_name='does_not_exist')
False
class airflow.hooks.hive_hooks.HiveServer2Hook(hiveserver2_conn_id=u'hiveserver2_default')
Bases: airflow.hooks.base_hook.BaseHook
Wrapper around the pyhive library
Note that the default authMechanism is PLAIN, to override it you
can specify it in the extra
of your connection in the UI as in
get_pandas_df(hql, schema=u'default')
Get a pandas dataframe from a Hive query
>>> hh = HiveServer2Hook()
>>> sql = "SELECT * FROM airflow.static_babynames LIMIT 100"
>>> df = hh.get_pandas_df(sql)
>>> len(df.index)
100
get_records(hql, schema=u'default')
Get a set of records from a Hive query.
>>> hh = HiveServer2Hook()
>>> sql = "SELECT * FROM airflow.static_babynames LIMIT 100"
>>> len(hh.get_records(sql))
100
class airflow.hooks.http_hook.HttpHook(method='POST', http_conn_id='http_default')
Bases: airflow.hooks.base_hook.BaseHook
Interact with HTTP servers. :param http_conn_id: connection that has the base API url i.e https://www.google.com/
and optional authentication credentials. Default headers can also be specified in the Extra field in json format.
Parameters: | method (str) – the API method to be called |
---|
check_response(response)
Checks the status code and raise an AirflowException exception on non 2XX or 3XX status codes :param response: A requests response object :type response: requests.response
get_conn(headers=None)
Returns http session for use with requests :param headers: additional headers to be passed through as a dictionary :type headers: dict
run(endpoint, data=None, headers=None, extra_options=None)
Performs the request :param endpoint: the endpoint to be called i.e. resource/v1/query? :type endpoint: str :param data: payload to be uploaded or request parameters :type data: dict :param headers: additional headers to be passed through as a dictionary :type headers: dict :param extra_options: additional options to be used when executing the request
i.e. {‘check_response’: False} to avoid checking raising exceptions on non 2XX or 3XX status codes
run_and_check(session, prepped_request, extra_options)
Grabs extra options like timeout and actually runs the request, checking for the result :param session: the session to be used to execute the request :type session: requests.Session :param prepped_request: the prepared request generated in run() :type prepped_request: session.prepare_request :param extra_options: additional options to be used when executing the request
i.e. {‘check_response’: False} to avoid checking raising exceptions on non 2XX or 3XX status codes
run_with_advanced_retry(_retry_args, *args, **kwargs)
Runs Hook.run() with a Tenacity decorator attached to it. This is useful for connectors which might be disturbed by intermittent issues and should not instantly fail. :param _retry_args: Arguments which define the retry behaviour.
See Tenacity documentation at https://github.com/jd/tenacity
Example: ::
hook = HttpHook(http_conn_id=’my_conn’,method=’GET’) retry_args = dict(
wait=tenacity.wait_exponential(), stop=tenacity.stop_after_attempt(10), retry=requests.exceptions.ConnectionError) hook.run_with_advanced_retry(
endpoint=’v1/test’, _retry_args=retry_args)
class airflow.hooks.druid_hook.DruidDbApiHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Druid broker
This hook is purely for users to query druid broker. For ingestion, please use druidHook.
get_conn()
Establish a connection to druid broker.
get_pandas_df(sql, parameters=None)
Executes the sql and returns a pandas dataframe
Parameters: |
|
---|
get_uri()
Get the connection uri for druid broker.
e.g: druid://localhost:8082/druid/v2/sql/
insert_rows(table, rows, target_fields=None, commit_every=1000)
A generic way to insert a set of tuples into a table, a new transaction is created every commit_every rows
Parameters: |
|
---|
set_autocommit(conn, autocommit)
Sets the autocommit flag on the connection
class airflow.hooks.druid_hook.DruidHook(druid_ingest_conn_id='druid_ingest_default', timeout=1, max_ingestion_time=None)
Bases: airflow.hooks.base_hook.BaseHook
Connection to Druid overlord for ingestion
Parameters: |
|
---|
class airflow.hooks.hdfs_hook.HDFSHook(hdfs_conn_id='hdfs_default', proxy_user=None, autoconfig=False)
Bases: airflow.hooks.base_hook.BaseHook
Interact with HDFS. This class is a wrapper around the snakebite library.
Parameters: |
|
---|
get_conn()
Returns a snakebite HDFSClient object.
class airflow.hooks.jdbc_hook.JdbcHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
General hook for jdbc db access.
JDBC URL, username and password will be taken from the predefined connection. Note that the whole JDBC URL must be specified in the “host” field in the DB. Raises an airflow error if the given connection id doesn’t exist.
get_conn()
Returns a connection object
set_autocommit(conn, autocommit)
Enable or disable autocommit for the given connection.
Parameters: | conn – The connection |
---|---|
Returns: |
class airflow.hooks.mssql_hook.MsSqlHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Microsoft SQL Server.
get_conn()
Returns a mssql connection object
set_autocommit(conn, autocommit)
Sets the autocommit flag on the connection
class airflow.hooks.mysql_hook.MySqlHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with MySQL.
You can specify charset in the extra field of your connection
as {"charset": "utf8"}
. Also you can choose cursor as
{"cursor": "SSCursor"}
. Refer to the MySQLdb.cursors for more details.
bulk_dump(table, tmp_file)
Dumps a database table into a tab-delimited file
bulk_load(table, tmp_file)
Loads a tab-delimited file into a database table
get_autocommit(conn)
MySql connection gets autocommit in a different way. :param conn: connection to get autocommit setting from. :type conn: connection object. :return: connection autocommit setting :rtype bool
get_conn()
Returns a mysql connection object
set_autocommit(conn, autocommit)
MySql connection sets autocommit in a different way.
class airflow.hooks.oracle_hook.OracleHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Oracle SQL.
bulk_insert_rows(table, rows, target_fields=None, commit_every=5000)
A performant bulk insert for cx_Oracle that uses prepared statements via executemany(). For best performance, pass in rows as an iterator.
get_conn()
Returns a oracle connection object Optional parameters for using a custom DSN connection (instead of using a server alias from tnsnames.ora) The dsn (data source name) is the TNS entry (from the Oracle names server or tnsnames.ora file) or is a string like the one returned from makedsn().
Parameters: |
|
---|
You can set these parameters in the extra fields of your connection
as in { "dsn":"some.host.address" , "service_name":"some.service.name" }
insert_rows(table, rows, target_fields=None, commit_every=1000)
A generic way to insert a set of tuples into a table, the whole set of inserts is treated as one transaction Changes from standard DbApiHook implementation: - Oracle SQL queries in cx_Oracle can not be terminated with a semicolon (‘;’) - Replace NaN values with NULL using numpy.nan_to_num (not using is_nan()
because of input types error for strings)
class airflow.hooks.pig_hook.PigCliHook(pig_cli_conn_id='pig_cli_default')
Bases: airflow.hooks.base_hook.BaseHook
Simple wrapper around the pig CLI.
Note that you can also set default pig CLI properties using the
pig_properties
to be used in your connection as in
{"pig_properties": "-Dpig.tmpfilecompression=true"}
run_cli(pig, verbose=True)
Run an pig script using the pig cli
>>> ph = PigCliHook()
>>> result = ph.run_cli("ls /;")
>>> ("hdfs://" in result)
True
class airflow.hooks.postgres_hook.PostgresHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Postgres.
You can specify ssl parameters in the extra field of your connection
as {"sslmode": "require", "sslcert": "/path/to/cert.pem", etc}
.
Note: For Redshift, use keepalives_idle in the extra connection parameters and set it to less than 300 seconds.
bulk_dump(table, tmp_file)
Dumps a database table into a tab-delimited file
bulk_load(table, tmp_file)
Loads a tab-delimited file into a database table
copy_expert(sql, filename, open=<built-in function open>)
Executes SQL using psycopg2 copy_expert method. Necessary to execute COPY command without access to a superuser.
Note: if this method is called with a “COPY FROM” statement and the specified input file does not exist, it creates an empty file and no data is loaded, but the operation succeeds. So if users want to be aware when the input file does not exist, they have to check its existence by themselves.
get_conn()
Returns a connection object
class airflow.hooks.presto_hook.PrestoHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Presto through PyHive!
>>> ph = PrestoHook()
>>> sql = "SELECT count(1) AS num FROM airflow.static_babynames"
>>> ph.get_records(sql)
[[340698]]
get_conn()
Returns a connection object
get_first(hql, parameters=None)
Returns only the first row, regardless of how many rows the query returns.
get_pandas_df(hql, parameters=None)
Get a pandas dataframe from a sql query.
get_records(hql, parameters=None)
Get a set of records from Presto
insert_rows(table, rows, target_fields=None)
A generic way to insert a set of tuples into a table.
Parameters: |
|
---|
run(hql, parameters=None)
Execute the statement against Presto. Can be used to create views.
class airflow.hooks.S3_hook.S3Hook(aws_conn_id='aws_default')
Bases: airflow.contrib.hooks.aws_hook.AwsHook
Interact with AWS S3, using the boto3 library.
check_for_bucket(bucket_name)
Check if bucket_name exists.
Parameters: | bucket_name (str) – the name of the bucket |
---|
check_for_key(key, bucket_name=None)
Checks if a key exists in a bucket
Parameters: |
|
---|
check_for_prefix(bucket_name, prefix, delimiter)
Checks that a prefix exists in a bucket
check_for_wildcard_key(wildcard_key, bucket_name=None, delimiter='')
Checks that a key matching a wildcard expression exists in a bucket
get_bucket(bucket_name)
Returns a boto3.S3.Bucket object
Parameters: | bucket_name (str) – the name of the bucket |
---|
get_key(key, bucket_name=None)
Returns a boto3.s3.Object
Parameters: |
|
---|
get_wildcard_key(wildcard_key, bucket_name=None, delimiter='')
Returns a boto3.s3.Object object matching the wildcard expression
Parameters: |
|
---|
list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
Lists keys in a bucket under prefix and not containing delimiter
Parameters: |
|
---|
list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
Lists prefixes in a bucket under prefix
Parameters: |
|
---|
load_bytes(bytes_data, key, bucket_name=None, replace=False, encrypt=False)
Loads bytes to S3
This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.
Parameters: |
|
---|
load_file(filename, key, bucket_name=None, replace=False, encrypt=False)
Loads a local file to S3
Parameters: |
|
---|
load_string(string_data, key, bucket_name=None, replace=False, encrypt=False, encoding='utf-8')
Loads a string to S3
This is provided as a convenience to drop a string in S3. It uses the boto infrastructure to ship a file to s3.
Parameters: |
|
---|
read_key(key, bucket_name=None)
Reads a key from S3
Parameters: |
|
---|
select_key(key, bucket_name=None, expression='SELECT * FROM S3Object', expression_type='SQL', input_serialization={'CSV': {}}, output_serialization={'CSV': {}})
Reads a key with S3 Select.
Parameters: |
|
---|---|
Returns: | retrieved subset of original data by S3 Select |
Return type: | str |
See also
For more details about S3 Select parameters: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.select_object_content
class airflow.hooks.samba_hook.SambaHook(samba_conn_id)
Bases: airflow.hooks.base_hook.BaseHook
Allows for interaction with an samba server.
class airflow.hooks.slack_hook.SlackHook(token=None, slack_conn_id=None)
Bases: airflow.hooks.base_hook.BaseHook
Interact with Slack, using slackclient library.
class airflow.hooks.sqlite_hook.SqliteHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with SQLite.
get_conn()
Returns a sqlite connection object
class airflow.hooks.webhdfs_hook.WebHDFSHook(webhdfs_conn_id='webhdfs_default', proxy_user=None)
Bases: airflow.hooks.base_hook.BaseHook
Interact with HDFS. This class is a wrapper around the hdfscli library.
check_for_path(hdfs_path)
Check for the existence of a path in HDFS by querying FileStatus.
get_conn()
Returns a hdfscli InsecureClient object.
load_file(source, destination, overwrite=True, parallelism=1, **kwargs)
Uploads a file to HDFS
Parameters: |
|
---|
class airflow.hooks.zendesk_hook.ZendeskHook(zendesk_conn_id)
Bases: airflow.hooks.base_hook.BaseHook
A hook to talk to Zendesk
call(path, query=None, get_all_pages=True, side_loading=False)
Call Zendesk API and return results
Parameters: |
|
---|
class airflow.contrib.hooks.aws_dynamodb_hook.AwsDynamoDBHook(table_keys=None, table_name=None, region_name=None, *args, **kwargs)
Bases: airflow.contrib.hooks.aws_hook.AwsHook
Interact with AWS DynamoDB.
Parameters: |
|
---|
write_batch_data(items)
Write batch items to dynamodb table with provisioned throughout capacity.
class airflow.contrib.hooks.aws_hook.AwsHook(aws_conn_id='aws_default')
Bases: airflow.hooks.base_hook.BaseHook
Interact with AWS. This class is a thin wrapper around the boto3 python library.
get_credentials(region_name=None)
Get the underlying botocore.Credentials object.
This contains the attributes: access_key, secret_key and token.
get_session(region_name=None)
Get the underlying boto3.session.
class airflow.contrib.hooks.aws_lambda_hook.AwsLambdaHook(function_name, region_name=None, log_type='None', qualifier='$LATEST', invocation_type='RequestResponse', *args, **kwargs)
Bases: airflow.contrib.hooks.aws_hook.AwsHook
Interact with AWS Lambda
Parameters: |
|
---|
invoke_lambda(payload)
Invoke Lambda Function
class airflow.contrib.hooks.azure_data_lake_hook.AzureDataLakeHook(azure_data_lake_conn_id='azure_data_lake_default')
Bases: airflow.hooks.base_hook.BaseHook
Interacts with Azure Data Lake.
Client ID and client secret should be in user and password parameters. Tenant and account name should be extra field as {“tenant”: “<TENANT>”, “account_name”: “ACCOUNT_NAME”}.
Parameters: | azure_data_lake_conn_id (str) – Reference to the Azure Data Lake connection. |
---|
check_for_file(file_path)
Check if a file exists on Azure Data Lake.
Parameters: | file_path (str) – Path and name of the file. |
---|---|
Returns: | True if the file exists, False otherwise. |
:rtype bool
download_file(local_path, remote_path, nthreads=64, overwrite=True, buffersize=4194304, blocksize=4194304)
Download a file from Azure Blob Storage.
Parameters: |
|
---|
get_conn()
Return a AzureDLFileSystem object.
upload_file(local_path, remote_path, nthreads=64, overwrite=True, buffersize=4194304, blocksize=4194304)
Upload a file to Azure Data Lake.
Parameters: |
|
---|
class airflow.contrib.hooks.azure_fileshare_hook.AzureFileShareHook(wasb_conn_id='wasb_default')
Bases: airflow.hooks.base_hook.BaseHook
Interacts with Azure FileShare Storage.
Additional options passed in the ‘extra’ field of the connection will be passed to the FileService() constructor.
Parameters: | wasb_conn_id (str) – Reference to the wasb connection. |
---|
check_for_directory(share_name, directory_name, **kwargs)
Check if a directory exists on Azure File Share.
Parameters: |
|
---|---|
Returns: | True if the file exists, False otherwise. |
:rtype bool
check_for_file(share_name, directory_name, file_name, **kwargs)
Check if a file exists on Azure File Share.
Parameters: |
|
---|---|
Returns: | True if the file exists, False otherwise. |
:rtype bool
create_directory(share_name, directory_name, **kwargs)
Create a new direcotry on a Azure File Share.
Parameters: |
|
---|---|
Returns: | A list of files and directories |
:rtype list
get_conn()
Return the FileService object.
get_file(file_path, share_name, directory_name, file_name, **kwargs)
Download a file from Azure File Share.
Parameters: |
|
---|
get_file_to_stream(stream, share_name, directory_name, file_name, **kwargs)
Download a file from Azure File Share.
Parameters: |
|
---|
list_directories_and_files(share_name, directory_name=None, **kwargs)
Return the list of directories and files stored on a Azure File Share.
Parameters: |
|
---|---|
Returns: | A list of files and directories |
:rtype list
load_file(file_path, share_name, directory_name, file_name, **kwargs)
Upload a file to Azure File Share.
Parameters: |
|
---|
load_stream(stream, share_name, directory_name, file_name, count, **kwargs)
Upload a stream to Azure File Share.
Parameters: |
|
---|
load_string(string_data, share_name, directory_name, file_name, **kwargs)
Upload a string to Azure File Share.
Parameters: |
|
---|
class airflow.contrib.hooks.bigquery_hook.BigQueryHook(bigquery_conn_id='bigquery_default', delegate_to=None, use_legacy_sql=True)
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
, airflow.hooks.dbapi_hook.DbApiHook
, airflow.utils.log.logging_mixin.LoggingMixin
Interact with BigQuery. This hook uses the Google Cloud Platform connection.
get_conn()
Returns a BigQuery PEP 249 connection object.
get_pandas_df(sql, parameters=None, dialect=None)
Returns a Pandas DataFrame for the results produced by a BigQuery query. The DbApiHook method must be overridden because Pandas doesn’t support PEP 249 connections, except for SQLite. See:
https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L447 https://github.com/pydata/pandas/issues/6900
Parameters: |
|
---|
get_service()
Returns a BigQuery service object.
insert_rows(table, rows, target_fields=None, commit_every=1000)
Insertion is currently unsupported. Theoretically, you could use BigQuery’s streaming API to insert rows into a table, but this hasn’t been implemented.
table_exists(project_id, dataset_id, table_id)
Checks for the existence of a table in Google BigQuery.
Parameters: |
|
---|
class airflow.contrib.hooks.cassandra_hook.CassandraHook(cassandra_conn_id='cassandra_default')
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Hook used to interact with Cassandra
Contact points can be specified as a comma-separated string in the ‘hosts’ field of the connection.
Port can be specified in the port field of the connection.
If SSL is enabled in Cassandra, pass in a dict in the extra field as kwargs for
ssl.wrap_socket()
. For example:
{‘ssl_options’ : {‘ca_certs’ : PATH_TO_CA_CERTS}
}
Default load balancing policy is RoundRobinPolicy. To specify a different LB policy:
DCAwareRoundRobinPolicy
{
‘load_balancing_policy’: ‘DCAwareRoundRobinPolicy’, ‘load_balancing_policy_args’: {
‘local_dc’: LOCAL_DC_NAME, // optional ‘used_hosts_per_remote_dc’: SOME_INT_VALUE, // optional
}
}
WhiteListRoundRobinPolicy
{
‘load_balancing_policy’: ‘WhiteListRoundRobinPolicy’, ‘load_balancing_policy_args’: {
‘hosts’: [‘HOST1’, ‘HOST2’, ‘HOST3’]
}
}
TokenAwarePolicy
{
‘load_balancing_policy’: ‘TokenAwarePolicy’, ‘load_balancing_policy_args’: {
‘child_load_balancing_policy’: CHILD_POLICY_NAME, // optional ‘child_load_balancing_policy_args’: { … } // optional
}
}
For details of the Cluster config, see cassandra.cluster.
get_conn()
Returns a cassandra Session object
record_exists(table, keys)
Checks if a record exists in Cassandra
Parameters: |
|
---|
shutdown_cluster()
Closes all sessions and connections associated with this Cluster.
class airflow.contrib.hooks.cloudant_hook.CloudantHook(cloudant_conn_id='cloudant_default')
Bases: airflow.hooks.base_hook.BaseHook
Interact with Cloudant.
This class is a thin wrapper around the cloudant python library. See the documentation here.
db()
Returns the Database object for this hook.
See the documentation for cloudant-python here https://github.com/cloudant-labs/cloudant-python.
class airflow.contrib.hooks.databricks_hook.DatabricksHook(databricks_conn_id='databricks_default', timeout_seconds=180, retry_limit=3)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Interact with Databricks.
submit_run(json)
Utility function to call the api/2.0/jobs/runs/submit
endpoint.
Parameters: | json (dict) – The data used in the body of the request to the submit endpoint. |
---|---|
Returns: | the run_id as a string |
Return type: | string |
class airflow.contrib.hooks.datadog_hook.DatadogHook(datadog_conn_id='datadog_default')
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Uses datadog API to send metrics of practically anything measurable, so it’s possible to track # of db records inserted/deleted, records read from file and many other useful metrics.
Depends on the datadog API, which has to be deployed on the same server where Airflow runs.
Parameters: |
|
---|
post_event(title, text, tags=None, alert_type=None, aggregation_key=None)
Posts an event to datadog (processing finished, potentially alerts, other issues) Think about this as a means to maintain persistence of alerts, rather than alerting itself.
Parameters: |
|
---|
query_metric(query, from_seconds_ago, to_seconds_ago)
Queries datadog for a specific metric, potentially with some function applied to it and returns the results.
Parameters: |
|
---|
send_metric(metric_name, datapoint, tags=None)
Sends a single datapoint metric to DataDog
Parameters: |
|
---|
class airflow.contrib.hooks.datastore_hook.DatastoreHook(datastore_conn_id='google_cloud_datastore_default', delegate_to=None)
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Interact with Google Cloud Datastore. This hook uses the Google Cloud Platform connection.
This object is not threads safe. If you want to make multiple requests simultaneously, you will need to create a hook per thread.
allocate_ids(partialKeys)
Allocate IDs for incomplete keys. see https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds
Parameters: | partialKeys – a list of partial keys |
---|---|
Returns: | a list of full keys. |
begin_transaction()
Get a new transaction handle
Returns: | a transaction handle |
---|
commit(body)
Commit a transaction, optionally creating, deleting or modifying some entities.
Parameters: | body – the body of the commit request |
---|---|
Returns: | the response body of the commit request |
delete_operation(name)
Deletes the long-running operation
Parameters: | name – the name of the operation resource |
---|
export_to_storage_bucket(bucket, namespace=None, entity_filter=None, labels=None)
Export entities from Cloud Datastore to Cloud Storage for backup
get_conn(version='v1')
Returns a Google Cloud Storage service object.
get_operation(name)
Gets the latest state of a long-running operation
Parameters: | name – the name of the operation resource |
---|
import_from_storage_bucket(bucket, file, namespace=None, entity_filter=None, labels=None)
Import a backup from Cloud Storage to Cloud Datastore
lookup(keys, read_consistency=None, transaction=None)
Lookup some entities by key
Parameters: |
|
---|---|
Returns: | the response body of the lookup request. |
poll_operation_until_done(name, polling_interval_in_seconds)
Poll backup operation state until it’s completed
rollback(transaction)
Roll back a transaction
Parameters: | transaction – the transaction to roll back |
---|
run_query(body)
Run a query for entities.
Parameters: | body – the body of the query request |
---|---|
Returns: | the batch of query results. |
class airflow.contrib.hooks.discord_webhook_hook.DiscordWebhookHook(http_conn_id=None, webhook_endpoint=None, message='', username=None, avatar_url=None, tts=False, proxy=None, *args, **kwargs)
Bases: airflow.hooks.http_hook.HttpHook
This hook allows you to post messages to Discord using incoming webhooks. Takes a Discord connection ID with a default relative webhook endpoint. The default endpoint can be overridden using the webhook_endpoint parameter (https://discordapp.com/developers/docs/resources/webhook).
Each Discord webhook can be pre-configured to use a specific username and avatar_url. You can override these defaults in this hook.
Parameters: |
|
---|
execute()
Execute the Discord webhook call
class airflow.contrib.hooks.emr_hook.EmrHook(emr_conn_id=None, *args, **kwargs)
Bases: airflow.contrib.hooks.aws_hook.AwsHook
Interact with AWS EMR. emr_conn_id is only neccessary for using the create_job_flow method.
create_job_flow(job_flow_overrides)
Creates a job flow using the config from the EMR connection. Keys of the json extra hash may have the arguments of the boto3 run_job_flow method. Overrides for this config may be passed as the job_flow_overrides.
class airflow.contrib.hooks.fs_hook.FSHook(conn_id='fs_default')
Bases: airflow.hooks.base_hook.BaseHook
Allows for interaction with an file server.
Connection should have a name and a path specified under extra:
example: Conn Id: fs_test Conn Type: File (path) Host, Shchema, Login, Password, Port: empty Extra: {“path”: “/tmp”}
class airflow.contrib.hooks.ftp_hook.FTPHook(ftp_conn_id='ftp_default')
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Interact with FTP.
Errors that may occur throughout but should be handled downstream.
close_conn()
Closes the connection. An error will occur if the connection wasn’t ever opened.
create_directory(path)
Creates a directory on the remote system.
Parameters: | path (str) – full path to the remote directory to create |
---|
delete_directory(path)
Deletes a directory on the remote system.
Parameters: | path (str) – full path to the remote directory to delete |
---|
delete_file(path)
Removes a file on the FTP Server.
Parameters: | path (str) – full path to the remote file |
---|
describe_directory(path)
Returns a dictionary of {filename: {attributes}} for all files on the remote system (where the MLSD command is supported).
Parameters: | path (str) – full path to the remote directory |
---|
get_conn()
Returns a FTP connection object
list_directory(path, nlst=False)
Returns a list of files on the remote system.
Parameters: | path (str) – full path to the remote directory to list |
---|
rename(from_name, to_name)
Rename a file.
Parameters: |
|
---|
retrieve_file(remote_full_path, local_full_path_or_buffer)
Transfers the remote file to a local location.
If local_full_path_or_buffer is a string path, the file will be put at that location; if it is a file-like buffer, the file will be written to the buffer but not closed.
Parameters: |
|
---|
store_file(remote_full_path, local_full_path_or_buffer)
Transfers a local file to the remote location.
If local_full_path_or_buffer is a string path, the file will be read from that location; if it is a file-like buffer, the file will be read from the buffer but not closed.
Parameters: |
|
---|
class airflow.contrib.hooks.ftp_hook.FTPSHook(ftp_conn_id='ftp_default')
Bases: airflow.contrib.hooks.ftp_hook.FTPHook
get_conn()
Returns a FTPS connection object.
class airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook(gcp_conn_id='google_cloud_default', delegate_to=None)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
A base hook for Google cloud-related hooks. Google cloud has a shared REST API client that is built in the same way no matter which service you use. This class helps construct and authorize the credentials needed to then call apiclient.discovery.build() to actually discover and build a client for a Google cloud service.
The class also contains some miscellaneous helper functions.
All hook derived from this base hook use the ‘Google Cloud Platform’ connection type. Two ways of authentication are supported:
Default credentials: Only the ‘Project Id’ is required. You’ll need to
have set up default credentials, such as by the
GOOGLE_APPLICATION_DEFAULT
environment variable or from the metadata
server on Google Compute Engine.
JSON key file: Specify ‘Project Id’, ‘Key Path’ and ‘Scope’.
Legacy P12 key files are not supported.
class airflow.contrib.hooks.gcp_container_hook.GKEClusterHook(project_id, location)
Bases: airflow.hooks.base_hook.BaseHook
create_cluster(cluster, retry=<object object>, timeout=<object object>)
Creates a cluster, consisting of the specified number and type of Google Compute Engine instances.
Parameters: |
|
---|---|
Returns: | The full url to the new, or existing, cluster |
:raisesParseError: On JSON parsing problems when trying to convert dict AirflowException: cluster is not dict type nor Cluster proto type
delete_cluster(name, retry=<object object>, timeout=<object object>)
Deletes the cluster, including the Kubernetes endpoint and all worker nodes. Firewalls and routes that were configured during cluster creation are also deleted. Other Google Compute Engine resources that might be in use by the cluster (e.g. load balancer resources) will not be deleted if they weren’t present at the initial create time.
Parameters: |
|
---|---|
Returns: | The full url to the delete operation if successful, else None |
get_cluster(name, retry=<object object>, timeout=<object object>)
Gets details of specified cluster :param name: The name of the cluster to retrieve :type name: str :param retry: A retry object used to retry requests. If None is specified,
requests will not be retried.
Parameters: | timeout (float) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt. |
---|---|
Returns: | A google.cloud.container_v1.types.Cluster instance |
get_operation(operation_name)
Fetches the operation from Google Cloud :param operation_name: Name of operation to fetch :type operation_name: str :return: The new, updated operation from Google Cloud
wait_for_operation(operation)
Given an operation, continuously fetches the status from Google Cloud until either completion or an error occurring :param operation: The Operation to wait for :type operation: A google.cloud.container_V1.gapic.enums.Operator :return: A new, updated operation fetched from Google Cloud
class airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook(gcp_conn_id='google_cloud_default', delegate_to=None, poll_sleep=10)
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
get_conn()
Returns a Google Cloud Storage service object.
class airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook(gcp_conn_id='google_cloud_default', delegate_to=None, api_version='v1beta2')
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Hook for Google Cloud Dataproc APIs.
await(operation)
Awaits for Google Cloud Dataproc Operation to complete.
get_conn()
Returns a Google Cloud Dataproc service object.
wait(operation)
Awaits for Google Cloud Dataproc Operation to complete.
class airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook(gcp_conn_id='google_cloud_default', delegate_to=None)
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
create_job(project_id, job, use_existing_job_fn=None)
Launches a MLEngine job and wait for it to reach a terminal state.
Parameters: |
|
---|---|
Returns: | The MLEngine job object if the job successfully reach a terminal state (which might be FAILED or CANCELLED state). |
Return type: | dict |
create_model(project_id, model)
Create a Model. Blocks until finished.
create_version(project_id, model_name, version_spec)
Creates the Version on Google Cloud ML Engine.
Returns the operation if the version was created successfully and raises an error otherwise.
delete_version(project_id, model_name, version_name)
Deletes the given version of a model. Blocks until finished.
get_conn()
Returns a Google MLEngine service object.
get_model(project_id, model_name)
Gets a Model. Blocks until finished.
list_versions(project_id, model_name)
Lists all available versions of a model. Blocks until finished.
set_default_version(project_id, model_name, version_name)
Sets a version to be the default. Blocks until finished.
class airflow.contrib.hooks.gcp_pubsub_hook.PubSubHook(gcp_conn_id='google_cloud_default', delegate_to=None)
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Hook for accessing Google Pub/Sub.
The GCP project against which actions are applied is determined by the project embedded in the Connection referenced by gcp_conn_id.
acknowledge(project, subscription, ack_ids)
Pulls up to max_messages
messages from Pub/Sub subscription.
Parameters: |
|
---|
create_subscription(topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False)
Creates a Pub/Sub subscription, if it does not already exist.
Parameters: |
|
---|---|
Returns: | subscription name which will be the system-generated value if
the |
Return type: | string |
create_topic(project, topic, fail_if_exists=False)
Creates a Pub/Sub topic, if it does not already exist.
Parameters: |
|
---|
delete_subscription(project, subscription, fail_if_not_exists=False)
Deletes a Pub/Sub subscription, if it exists.
Parameters: |
|
---|
delete_topic(project, topic, fail_if_not_exists=False)
Deletes a Pub/Sub topic if it exists.
Parameters: |
|
---|
get_conn()
Returns a Pub/Sub service object.
Return type: | apiclient.discovery.Resource |
---|
publish(project, topic, messages)
Publishes messages to a Pub/Sub topic.
Parameters: |
|
---|
pull(project, subscription, max_messages, return_immediately=False)
Pulls up to max_messages
messages from Pub/Sub subscription.
Parameters: |
|
---|
:return A list of Pub/Sub ReceivedMessage objects each containingan
ackId
property and a message
property, which includes
the base64-encoded message content. See
https://cloud.google.com/pubsub/docs/reference/rest/v1/ projects.subscriptions/pull#ReceivedMessage
class airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook(google_cloud_storage_conn_id='google_cloud_default', delegate_to=None)
Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Interact with Google Cloud Storage. This hook uses the Google Cloud Platform connection.
copy(source_bucket, source_object, destination_bucket=None, destination_object=None)
Copies an object from a bucket to another, with renaming if requested.
destination_bucket or destination_object can be omitted, in which case source bucket/object is used, but not both.
Parameters: |
|
---|
create_bucket(bucket_name, storage_class='MULTI_REGIONAL', location='US', project_id=None, labels=None)
Creates a new bucket. Google Cloud Storage uses a flat namespace, so you can’t create a bucket with a name that is already in use.
See also
For more information, see Bucket Naming Guidelines: https://cloud.google.com/storage/docs/bucketnaming.html#requirements
Parameters: |
|
---|---|
Returns: | If successful, it returns the |
delete(bucket, object, generation=None)
Delete an object if versioning is not enabled for the bucket, or if generation parameter is used.
Parameters: |
|
---|---|
Returns: | True if succeeded |
download(bucket, object, filename=None)
Get a file from Google Cloud Storage.
Parameters: |
|
---|
exists(bucket, object)
Checks for the existence of a file in Google Cloud Storage.
Parameters: |
|
---|
get_conn()
Returns a Google Cloud Storage service object.
get_crc32c(bucket, object)
Gets the CRC32c checksum of an object in Google Cloud Storage.
Parameters: |
|
---|
get_md5hash(bucket, object)
Gets the MD5 hash of an object in Google Cloud Storage.
Parameters: |
|
---|
get_size(bucket, object)
Gets the size of a file in Google Cloud Storage.
Parameters: |
|
---|
is_updated_after(bucket, object, ts)
Checks if an object is updated in Google Cloud Storage.
Parameters: |
|
---|
list(bucket, versions=None, maxResults=None, prefix=None, delimiter=None)
List all objects from the bucket with the give string prefix in name
Parameters: |
|
---|---|
Returns: | a stream of object names matching the filtering criteria |
rewrite(source_bucket, source_object, destination_bucket, destination_object=None)
Has the same functionality as copy, except that will work on files over 5 TB, as well as when copying between locations and/or storage classes.
destination_object can be omitted, in which case source_object is used.
Parameters: |
|
---|
upload(bucket, object, filename, mime_type='application/octet-stream')
Uploads a local file to Google Cloud Storage.
Parameters: |
|
---|
class airflow.contrib.hooks.jenkins_hook.JenkinsHook(conn_id='jenkins_default')
Bases: airflow.hooks.base_hook.BaseHook
Hook to manage connection to jenkins server
class airflow.contrib.hooks.jira_hook.JiraHook(jira_conn_id='jira_default', proxies=None)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Jira interaction hook, a Wrapper around JIRA Python SDK.
Parameters: | jira_conn_id (string) – reference to a pre-defined Jira Connection |
---|
class airflow.contrib.hooks.mongo_hook.MongoHook(conn_id='mongo_default', *args, **kwargs)
Bases: airflow.hooks.base_hook.BaseHook
PyMongo Wrapper to Interact With Mongo Database Mongo Connection Documentation https://docs.mongodb.com/manual/reference/connection-string/index.html You can specify connection string options in extra field of your connection https://docs.mongodb.com/manual/reference/connection-string/index.html#connection-string-options ex.
{replicaSet: test, ssl: True, connectTimeoutMS: 30000}
aggregate(mongo_collection, aggregate_query, mongo_db=None, **kwargs)
Runs an aggregation pipeline and returns the results https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.aggregate https://api.mongodb.com/python/current/examples/aggregation.html
find(mongo_collection, query, find_one=False, mongo_db=None, **kwargs)
Runs a mongo find query and returns the results https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.find
get_collection(mongo_collection, mongo_db=None)
Fetches a mongo collection object for querying.
Uses connection schema as DB unless specified.
get_conn()
Fetches PyMongo Client
insert_many(mongo_collection, docs, mongo_db=None, **kwargs)
Inserts many docs into a mongo collection. https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_many
insert_one(mongo_collection, doc, mongo_db=None, **kwargs)
Inserts a single document into a mongo collection https://api.mongodb.com/python/current/api/pymongo/collection.html#pymongo.collection.Collection.insert_one
class airflow.contrib.hooks.pinot_hook.PinotDbApiHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Connect to pinot db(https://github.com/linkedin/pinot) to issue pql
get_conn()
Establish a connection to pinot broker through pinot dbqpi.
get_first(sql)
Executes the sql and returns the first resulting row.
Parameters: | sql (str or list) – the sql statement to be executed (str) or a list of sql statements to execute |
---|
get_pandas_df(sql, parameters=None)
Executes the sql and returns a pandas dataframe
Parameters: |
|
---|
get_records(sql)
Executes the sql and returns a set of records.
Parameters: | sql (str) – the sql statement to be executed (str) or a list of sql statements to execute |
---|
get_uri()
Get the connection uri for pinot broker.
e.g: http://localhost:9000/pql
insert_rows(table, rows, target_fields=None, commit_every=1000)
A generic way to insert a set of tuples into a table, a new transaction is created every commit_every rows
Parameters: |
|
---|
set_autocommit(conn, autocommit)
Sets the autocommit flag on the connection
class airflow.contrib.hooks.qubole_hook.QuboleHook(*args, **kwargs)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
get_jobs_id(ti)
Get jobs associated with a Qubole commands :param ti: Task Instance of the dag, used to determine the Quboles command id :return: Job informations assoiciated with command
get_log(ti)
Get Logs of a command from Qubole :param ti: Task Instance of the dag, used to determine the Quboles command id :return: command log as text
get_results(ti=None, fp=None, inline=True, delim=None, fetch=True)
Get results (or just s3 locations) of a command from Qubole and save into a file :param ti: Task Instance of the dag, used to determine the Quboles command id :param fp: Optional file pointer, will create one and return if None passed :param inline: True to download actual results, False to get s3 locations only :param delim: Replaces the CTL-A chars with the given delim, defaults to ‘,’ :param fetch: when inline is True, get results directly from s3 (if large) :return: file location containing actual results or s3 locations of results
kill(ti)
Kill (cancel) a Qubole commmand :param ti: Task Instance of the dag, used to determine the Quboles command id :return: response from Qubole
class airflow.contrib.hooks.redis_hook.RedisHook(redis_conn_id='redis_default')
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Hook to interact with Redis database
get_conn()
Returns a Redis connection.
key_exists(key)
Checks if a key exists in Redis database
Parameters: | key (string) – The key to check the existence. |
---|
class airflow.contrib.hooks.redshift_hook.RedshiftHook(aws_conn_id='aws_default')
Bases: airflow.contrib.hooks.aws_hook.AwsHook
Interact with AWS Redshift, using the boto3 library
cluster_status(cluster_identifier)
Return status of a cluster
Parameters: | cluster_identifier (str) – unique identifier of a cluster |
---|
create_cluster_snapshot(snapshot_identifier, cluster_identifier)
Creates a snapshot of a cluster
Parameters: |
|
---|
delete_cluster(cluster_identifier, skip_final_cluster_snapshot=True, final_cluster_snapshot_identifier='')
Delete a cluster and optionally create a snapshot
Parameters: |
|
---|
describe_cluster_snapshots(cluster_identifier)
Gets a list of snapshots for a cluster
Parameters: | cluster_identifier (str) – unique identifier of a cluster |
---|
restore_from_cluster_snapshot(cluster_identifier, snapshot_identifier)
Restores a cluster from its snapshot
Parameters: |
|
---|
class airflow.contrib.hooks.segment_hook.SegmentHook(segment_conn_id='segment_default', segment_debug_mode=False, *args, **kwargs)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
on_error(error, items)
Handles error callbacks when using Segment with segment_debug_mode set to True
class airflow.contrib.hooks.sftp_hook.SFTPHook(ftp_conn_id='sftp_default')
Bases: airflow.hooks.base_hook.BaseHook
Interact with SFTP. Aims to be interchangeable with FTPHook.
Pitfalls: - In contrast with FTPHook describe_directory only returns size, type and
modify. It doesn’t return unix.owner, unix.mode, perm, unix.group and unique.
Errors that may occur throughout but should be handled downstream.
close_conn()
Closes the connection. An error will occur if the connection wasnt ever opened.
create_directory(path, mode=777)
Creates a directory on the remote system. :param path: full path to the remote directory to create :type path: str :param mode: int representation of octal mode for directory
delete_directory(path)
Deletes a directory on the remote system. :param path: full path to the remote directory to delete :type path: str
delete_file(path)
Removes a file on the FTP Server :param path: full path to the remote file :type path: str
describe_directory(path)
Returns a dictionary of {filename: {attributes}} for all files on the remote system (where the MLSD command is supported). :param path: full path to the remote directory :type path: str
get_conn()
Returns an SFTP connection object
list_directory(path)
Returns a list of files on the remote system. :param path: full path to the remote directory to list :type path: str
retrieve_file(remote_full_path, local_full_path)
Transfers the remote file to a local location. If local_full_path is a string path, the file will be put at that location :param remote_full_path: full path to the remote file :type remote_full_path: str :param local_full_path: full path to the local file :type local_full_path: str
store_file(remote_full_path, local_full_path)
Transfers a local file to the remote location. If local_full_path_or_buffer is a string path, the file will be read from that location :param remote_full_path: full path to the remote file :type remote_full_path: str :param local_full_path: full path to the local file :type local_full_path: str
class airflow.contrib.hooks.slack_webhook_hook.SlackWebhookHook(http_conn_id=None, webhook_token=None, message='', channel=None, username=None, icon_emoji=None, link_names=False, proxy=None, *args, **kwargs)
Bases: airflow.hooks.http_hook.HttpHook
This hook allows you to post messages to Slack using incoming webhooks. Takes both Slack webhook token directly and connection that has Slack webhook token. If both supplied, Slack webhook token will be used.
Each Slack webhook token can be pre-configured to use a specific channel, username and icon. You can override these defaults in this hook.
Parameters: |
|
---|
execute()
Remote Popen (actually execute the slack webhook call)
Parameters: |
|
---|
class airflow.contrib.hooks.snowflake_hook.SnowflakeHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Snowflake.
get_sqlalchemy_engine() depends on snowflake-sqlalchemy
get_conn()
Returns a snowflake.connection object
get_uri()
override DbApiHook get_uri method for get_sqlalchemy_engine()
set_autocommit(conn, autocommit)
Sets the autocommit flag on the connection
class airflow.contrib.hooks.spark_jdbc_hook.SparkJDBCHook(spark_app_name='airflow-spark-jdbc', spark_conn_id='spark-default', spark_conf=None, spark_py_files=None, spark_files=None, spark_jars=None, num_executors=None, executor_cores=None, executor_memory=None, driver_memory=None, verbose=False, principal=None, keytab=None, cmd_type='spark_to_jdbc', jdbc_table=None, jdbc_conn_id='jdbc-default', jdbc_driver=None, metastore_table=None, jdbc_truncate=False, save_mode=None, save_format=None, batch_size=None, fetch_size=None, num_partitions=None, partition_column=None, lower_bound=None, upper_bound=None, create_table_column_types=None, *args, **kwargs)
Bases: airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook
This hook extends the SparkSubmitHook specifically for performing data transfers to/from JDBC-based databases with Apache Spark.
Parameters: |
|
---|---|
Type: | jdbc_conn_id: str |
class airflow.contrib.hooks.spark_sql_hook.SparkSqlHook(sql, conf=None, conn_id='spark_sql_default', total_executor_cores=None, executor_cores=None, executor_memory=None, keytab=None, principal=None, master='yarn', name='default-name', num_executors=None, verbose=True, yarn_queue='default')
Bases: airflow.hooks.base_hook.BaseHook
This hook is a wrapper around the spark-sql binary. It requires that the “spark-sql” binary is in the PATH. :param sql: The SQL query to execute :type sql: str :param conf: arbitrary Spark configuration property :type conf: str (format: PROP=VALUE) :param conn_id: connection_id string :type conn_id: str :param total_executor_cores: (Standalone & Mesos only) Total cores for all executors
(Default: all the available cores on the worker)
Parameters: |
|
---|
run_query(cmd='', **kwargs)
Remote Popen (actually execute the Spark-sql query)
Parameters: |
|
---|
class airflow.contrib.hooks.spark_submit_hook.SparkSubmitHook(conf=None, conn_id='spark_default', files=None, py_files=None, driver_classpath=None, jars=None, java_class=None, packages=None, exclude_packages=None, repositories=None, total_executor_cores=None, executor_cores=None, executor_memory=None, driver_memory=None, keytab=None, principal=None, name='default-name', num_executors=None, application_args=None, env_vars=None, verbose=False)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
This hook is a wrapper around the spark-submit binary to kick off a spark-submit job. It requires that the “spark-submit” binary is in the PATH or the spark_home to be supplied. :param conf: Arbitrary Spark configuration properties :type conf: dict :param conn_id: The connection id as configured in Airflow administration. When an
invalid connection_id is supplied, it will default to yarn.
Parameters: |
|
---|
driver and executor classpaths :type packages: str :param exclude_packages: Comma-separated list of maven coordinates of jars to exclude while resolving the dependencies provided in ‘packages’ :type exclude_packages: str :param repositories: Comma-separated list of additional remote repositories to search for the maven coordinates given with ‘packages’ :type repositories: str :param total_executor_cores: (Standalone & Mesos only) Total cores for all executors (Default: all the available cores on the worker) :type total_executor_cores: int :param executor_cores: (Standalone, YARN and Kubernetes only) Number of cores per executor (Default: 2) :type executor_cores: int :param executor_memory: Memory per executor (e.g. 1000M, 2G) (Default: 1G) :type executor_memory: str :param driver_memory: Memory allocated to the driver (e.g. 1000M, 2G) (Default: 1G) :type driver_memory: str :param keytab: Full path to the file that contains the keytab :type keytab: str :param principal: The name of the kerberos principal used for keytab :type principal: str :param name: Name of the job (default airflow-spark) :type name: str :param num_executors: Number of executors to launch :type num_executors: int :param application_args: Arguments for the application being submitted :type application_args: list :param env_vars: Environment variables for spark-submit. It
supports yarn and k8s mode too.
Parameters: | verbose (bool) – Whether to pass the verbose flag to spark-submit process for debugging |
---|
submit(application='', **kwargs)
Remote Popen to execute the spark-submit job
Parameters: |
|
---|
class airflow.contrib.hooks.sqoop_hook.SqoopHook(conn_id='sqoop_default', verbose=False, num_mappers=None, hcatalog_database=None, hcatalog_table=None, properties=None)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
This hook is a wrapper around the sqoop 1 binary. To be able to use the hook it is required that “sqoop” is in the PATH.
Additional arguments that can be passed via the ‘extra’ JSON field of the sqoop connection: * job_tracker: Job tracker local|jobtracker:port. * namenode: Namenode. * lib_jars: Comma separated jar files to include in the classpath. * files: Comma separated files to be copied to the map reduce cluster. * archives: Comma separated archives to be unarchived on the compute
machines.
Parameters: |
|
---|
Popen(cmd, **kwargs)
Remote Popen
Parameters: |
|
---|---|
Returns: | handle to subprocess |
export_table(table, export_dir, input_null_string, input_null_non_string, staging_table, clear_staging_table, enclosed_by, escaped_by, input_fields_terminated_by, input_lines_terminated_by, input_optionally_enclosed_by, batch, relaxed_isolation, extra_export_options=None)
Exports Hive table to remote location. Arguments are copies of direct sqoop command line Arguments :param table: Table remote destination :param export_dir: Hive table to export :param input_null_string: The string to be interpreted as null for
string columns
Parameters: |
|
---|
import_query(query, target_dir, append=False, file_type='text', split_by=None, direct=None, driver=None, extra_import_options=None)
Imports a specific query from the rdbms to hdfs :param query: Free format query to run :param target_dir: HDFS destination dir :param append: Append data to an existing dataset in HDFS :param file_type: “avro”, “sequence”, “text” or “parquet”
Imports data to hdfs into the specified format. Defaults to text.
Parameters: |
|
---|
import_table(table, target_dir=None, append=False, file_type='text', columns=None, split_by=None, where=None, direct=False, driver=None, extra_import_options=None)
Imports table from remote location to target dir. Arguments are copies of direct sqoop command line arguments :param table: Table to read :param target_dir: HDFS destination dir :param append: Append data to an existing dataset in HDFS :param file_type: “avro”, “sequence”, “text” or “parquet”.
Imports data to into the specified format. Defaults to text.
Parameters: |
|
---|
class airflow.contrib.hooks.ssh_hook.SSHHook(ssh_conn_id=None, remote_host=None, username=None, password=None, key_file=None, port=22, timeout=10, keepalive_interval=30)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Hook for ssh remote execution using Paramiko. ref: https://github.com/paramiko/paramiko This hook also lets you create ssh tunnel and serve as basis for SFTP file transfer
Parameters: |
|
---|
create_tunnel(**kwds)
Creates a tunnel between two hosts. Like ssh -L <LOCAL_PORT>:host:<REMOTE_PORT>. Remember to close() the returned “tunnel” object in order to clean up after yourself when you are done with the tunnel.
Parameters: |
|
---|---|
Returns: |
class airflow.contrib.hooks.vertica_hook.VerticaHook(*args, **kwargs)
Bases: airflow.hooks.dbapi_hook.DbApiHook
Interact with Vertica.
get_conn()
Returns verticaql connection object
class airflow.contrib.hooks.wasb_hook.WasbHook(wasb_conn_id='wasb_default')
Bases: airflow.hooks.base_hook.BaseHook
Interacts with Azure Blob Storage through the wasb:// protocol.
Additional options passed in the ‘extra’ field of the connection will be passed to the BlockBlockService() constructor. For example, authenticate using a SAS token by adding {“sas_token”: “YOUR_TOKEN”}.
Parameters: | wasb_conn_id (str) – Reference to the wasb connection. |
---|
check_for_blob(container_name, blob_name, **kwargs)
Check if a blob exists on Azure Blob Storage.
Parameters: |
|
---|---|
Returns: | True if the blob exists, False otherwise. |
:rtype bool
check_for_prefix(container_name, prefix, **kwargs)
Check if a prefix exists on Azure Blob storage.
Parameters: |
|
---|---|
Returns: | True if blobs matching the prefix exist, False otherwise. |
:rtype bool
get_conn()
Return the BlockBlobService object.
get_file(file_path, container_name, blob_name, **kwargs)
Download a file from Azure Blob Storage.
Parameters: |
|
---|
load_file(file_path, container_name, blob_name, **kwargs)
Upload a file to Azure Blob Storage.
Parameters: |
|
---|
load_string(string_data, container_name, blob_name, **kwargs)
Upload a string to Azure Blob Storage.
Parameters: |
|
---|
read_file(container_name, blob_name, **kwargs)
Read a file from Azure Blob Storage and return as a string.
Parameters: |
|
---|
class airflow.contrib.hooks.winrm_hook.WinRMHook(ssh_conn_id=None, remote_host=None, username=None, password=None, key_file=None, timeout=10, keepalive_interval=30)
Bases: airflow.hooks.base_hook.BaseHook
, airflow.utils.log.logging_mixin.LoggingMixin
Hook for winrm remote execution using pywinrm.
Parameters: |
|
---|
Executors are the mechanism by which task instances get run.
class airflow.executors.local_executor.LocalExecutor(parallelism=32)
Bases: airflow.executors.base_executor.BaseExecutor
LocalExecutor executes tasks locally in parallel. It uses the multiprocessing Python library and queues to parallelize the execution of tasks.
end()
This method is called when the caller is done submitting job and is wants to wait synchronously for the job submitted previously to be all done.
execute_async(key, command, queue=None, executor_config=None)
This method will execute the command asynchronously.
start()
Executors may need to get things started. For example LocalExecutor starts N workers.
sync()
Sync will get called periodically by the heartbeat method. Executors should override this to perform gather statuses.
class airflow.executors.celery_executor.CeleryExecutor(parallelism=32)
Bases: airflow.executors.base_executor.BaseExecutor
CeleryExecutor is recommended for production use of Airflow. It allows distributing the execution of task instances to multiple worker nodes.
Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.
end(synchronous=False)
This method is called when the caller is done submitting job and is wants to wait synchronously for the job submitted previously to be all done.
execute_async(key, command, queue='default', executor_config=None)
This method will execute the command asynchronously.
start()
Executors may need to get things started. For example LocalExecutor starts N workers.
sync()
Sync will get called periodically by the heartbeat method. Executors should override this to perform gather statuses.
class airflow.executors.sequential_executor.SequentialExecutor
Bases: airflow.executors.base_executor.BaseExecutor
This executor will only run one task instance at a time, can be used for debugging. It is also the only executor that can be used with sqlite since sqlite doesn’t support multiple connections.
Since we want airflow to work out of the box, it defaults to this SequentialExecutor alongside sqlite as you first install it.
end()
This method is called when the caller is done submitting job and is wants to wait synchronously for the job submitted previously to be all done.
execute_async(key, command, queue=None, executor_config=None)
This method will execute the command asynchronously.
sync()
Sync will get called periodically by the heartbeat method. Executors should override this to perform gather statuses.
class airflow.contrib.executors.mesos_executor.MesosExecutor(parallelism=32)
Bases: airflow.executors.base_executor.BaseExecutor
, airflow.www.utils.LoginMixin
MesosExecutor allows distributing the execution of task instances to multiple mesos workers.
Apache Mesos is a distributed systems kernel which abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. See http://mesos.apache.org/
end()
This method is called when the caller is done submitting job and is wants to wait synchronously for the job submitted previously to be all done.
execute_async(key, command, queue=None, executor_config=None)
This method will execute the command asynchronously.
start()
Executors may need to get things started. For example LocalExecutor starts N workers.
sync()
Sync will get called periodically by the heartbeat method. Executors should override this to perform gather statuses.