Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It is not possible as of now but we are about to start integrating with DAG managers (Airflow, Dagster, Prefect...). That will give the possibility to schedule the connectors with the proper dependencies.

See: https://github.com/airbytehq/airbyte/issues/836



Correct me if I'm wrong, but Airbyte is mainly an EL to be used with other schedulers? The scheduler will have to provide, internally or externally, the logic and parameters to the nodes that drive Airbyte connectors?


we are still figuring out how we want to integrate with other schedulers.

One option would be that you configure your source/destination with the Airbyte UI or API and with the external scheduler you just reference to a connection object with an id.

We need to run some experimentations and talk to the community to see what makes the most sense. If you have some opinion / scenarios, do you want to write them in the ticket?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: