Extra Pip Libraries Field Not Respected in Ingestion

When configuring ingestion recipes with custom Python libraries specified in the "Extra Pip Libraries" field, the additional dependencies were not being installed or used during ingestion execution. This prevented users from extending ingestion capabilities with custom transformers or source connectors that required additional Python packages.

This issue has been resolved by fixing the ingestion executor's handling of extra pip dependencies. The extra libraries specified in the ingestion configuration are now properly installed before recipe execution.

When configuring an ingestion source: 1. Navigate to the ingestion source configuration 2. In the "Advanced" section, locate "Extra Pip Libraries" 3. Specify your dependencies in the format: package-name==version 4. Save and execute the recipe

Example configuration:

source:
  type: custom
  config:
    # Your source config

pipeline_name: custom-ingestion
extra_pip_packages:
  - custom-transformer==1.0.0
  - additional-library>=2.0.0

Fixed in version v0.3.12. This fix applies to both UI-configured ingestion sources and CLI-based recipes. Ensure your remote executor is also updated to v0.3.12 or later.

• Ingestion Sources
• Custom Transformers