Obtaining Remote Executor Docker Image Package Inventory for Security Review

Organizations conducting internal security reviews need to obtain a comprehensive list of all Python packages, libraries, and frameworks included in the DataHub Remote Executor (RE) docker image. The package inventory varies depending on the ingestion source configuration, as additional dependencies may be installed dynamically at runtime for specific connectors. The base Remote Executor image contains a fixed set of core dependencies, while connector-specific packages are installed on-demand during ingestion runs.

You Might Be Asking:

• How can I get a complete list of packages in the Remote Executor image for security scanning?
• Does the package list vary by ingestion source?
• Are packages installed dynamically during runtime or are they all in the base image?
• What Python version and base OS does the Remote Executor use?
• Can I access the Remote Executor image before security approval?

The DataHub Remote Executor base image (v0.3.14.1+) contains 307 pre-installed Python packages on Ubuntu 24.04.3 LTS with Python 3.10.18.

Step 1: Understand the Package Structure

The Remote Executor image has two types of dependencies: - Base image packages: Pre-installed in every Remote Executor container (307+ packages - see complete list at the bottom of this article) - Runtime packages: Installed dynamically based on the specific ingestion connector being used

Step 2: Review the Complete Base Image Package List

Scroll to the bottom of this article to review the complete list of all 307 packages included in the DataHub Remote Executor base image. This list is current as of version 0.3.14.

If you need more information, contact DataHub Support and request the full package inventory file for your specific Remote Executor version. Provide: - Your Remote Executor version (e.g., v0.3.14.1) - The purpose of the request (security review, vulnerability scanning, compliance audit) - Whether you need just the base image packages or information about runtime dependencies for specific connectors

Step 3: Review Key Package Categories

The package manifest includes several important categories:

Core DataHub Packages:
- acryl-datahub==1!0.0.0+docker.1.0.0rc3.snapshot
- acryl-datahub-actions==0.2.1
- acryl-datahub-classify==0.0.11
- acryl-datahub-executor==1!0.0.0+docker.1.0.0rc3.snapshot
- acryl-executor==0.2.10

Cloud Platform SDKs:
- boto3==1.35.76 (AWS)
- google-cloud-bigquery==3.27.0 (GCP)
- azure-identity==1.21.0 (Azure)

Database Connectors:
- snowflake-connector-python==3.13.2
- databricks-sql-connector==2.9.6
- redshift-connector==2.1.5

Data Processing:
- pandas==2.1.4
- numpy==1.26.4
- pyarrow==18.1.0
- pyspark==3.5.6

Security & Authentication:
- cryptography==44.0.0
- PyJWT==2.10.1
- PyOpenSSL==24.3.0

Web Frameworks:
- fastapi==0.95.2
- uvicorn==0.32.1
- starlette==0.27.0

Orchestration:
- celery==5.4.0
- APScheduler==3.10.1.post1

ML/Analytics:
- scikit-learn==1.6.1
- scipy==1.14.1
- prophet==1.1.5
- spacy==3.7.5

(Total: 307 packages - see complete list below)

Step 4: Perform Security Scanning

Use the package manifest with your security scanning tools:

# Example: Using Trivy to scan for vulnerabilities
trivy image acryldata/datahub-ingestion-executor:v0.3.14.1

# Example: Using Snyk
snyk test --docker acryldata/datahub-ingestion-executor:v0.3.14.1

# Example: Using Grype
grype acryldata/datahub-ingestion-executor:v0.3.14.1

Step 5: Document Runtime Dependencies

For complete security coverage, also document which ingestion connectors you plan to use, as these may install additional packages at runtime:

• Dremio: Additional JDBC drivers and connector-specific packages
• Salesforce: Salesforce API client libraries
• SAP: SAP connector dependencies
• Custom sources: Any custom plugin dependencies

• The package list is specific to each Remote Executor version and is updated with each release
• Runtime-installed packages are NOT included in the base image inventory and vary by connector configuration
• If you don't have access to the Remote Executor image yet due to pending security approval, you can use the package list below to facilitate your security review
• For air-gapped or highly restricted environments, consider building a custom image with all required connector dependencies pre-installed
• The base OS is Ubuntu 24.04.3 LTS with Python 3.10.18
• Some packages use development/snapshot versions (indicated by +docker in version string) which are DataHub-maintained forks

• DataHub Deployment Architecture
• Remote Executor Configuration
• Security Best Practices

DataHub Executor Package Inventory
==================================
Base OS: Ubuntu 24.04.3 LTS
Python Version: Python 3.10.18
Total Python Packages: 307

All Python Packages:
====================
acryl-datahub==1!0.0.0+docker.1.0.0rc3.snapshot
acryl-datahub-actions==0.2.1
acryl-datahub-classify==0.0.11
acryl-datahub-executor==1!0.0.0+docker.1.0.0rc3.snapshot
acryl-executor==0.2.10
acryl_great_expectations==0.15.50.1
aiohappyeyeballs==2.4.4
aiohttp==3.11.10
aiosignal==1.3.1
alembic==1.14.0
altair==4.2.0
amqp==5.3.1
anyio==4.7.0
APScheduler==3.10.1.post1
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asn1crypto==1.5.1
asttokens==3.0.0
async-timeout==5.0.1
attrs==24.2.0
avro==1.11.3
avro-gen3==0.7.16
aws-msk-iam-sasl-signer-python==1.0.2
azure-core==1.33.0
azure-identity==1.21.0
beautifulsoup4==4.12.3
billiard==4.2.1
bleach==6.2.0
blis==0.7.11
boto3==1.35.76
botocore==1.35.76
bracex==2.5.post1
cached-property==2.0.1
cachetools==5.5.0
catalogue==2.0.10
celery==5.4.0
certifi==2024.8.30
cffi==1.17.1
chardet==5.2.0
charset-normalizer==3.4.0
click==8.1.7
click-default-group==1.2.4
click-didyoumean==0.3.1
click-plugins==1.1.1
click-repl==0.3.0
click-spinner==0.1.10
cloudpathlib==0.20.0
cmdstanpy==1.2.5
colorama==0.4.6
comm==0.2.2
confection==0.1.5
confluent-kafka==2.6.1
contourpy==1.3.1
croniter==1.3.15
cryptography==44.0.0
cycler==0.12.1
cymem==2.0.10
databricks-sdk==0.38.0
databricks-sql-connector==2.9.6
dataflows-tabulator==1.54.3
db-dtypes==1.3.1
debugpy==1.8.9
decorator==5.1.1
defusedxml==0.7.1
Deprecated==1.2.15
dictdiffer==0.9.0
docker==7.1.0
entrypoints==0.4
et_xmlfile==2.0.0
exceptiongroup==1.2.2
executing==2.1.0
expandvars==0.12.0
fastapi==0.95.2
fastavro==1.9.7
fastjsonschema==2.21.1
filelock==3.16.1
fonttools==4.56.0
fqdn==1.5.1
frozenlist==1.5.0
GeoAlchemy2==0.16.0
google-api-core==2.23.0
google-auth==2.36.0
google-cloud-appengine-logging==1.5.0
google-cloud-audit-log==0.3.0
google-cloud-bigquery==3.27.0
google-cloud-core==2.4.1
google-cloud-datacatalog==3.23.0
google-cloud-datacatalog-lineage==0.2.2
google-cloud-logging==3.5.0
google-cloud-resource-manager==1.13.1
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.66.0
greenlet==3.1.1
grpc-google-iam-v1==0.13.1
grpcio==1.68.1
grpcio-status==1.62.3
h11==0.16.0
holidays==0.68
httptools==0.6.4
humanfriendly==10.0
idna==3.10
ijson==3.3.0
importlib_metadata==8.5.0
importlib_resources==6.4.5
ipaddress==1.0.23
ipykernel==6.17.1
ipython==8.21.0
ipython-genutils==0.2.0
ipywidgets==8.1.5
isodate==0.7.2
isoduration==20.11.0
jedi==0.19.2
Jinja2==3.1.4
jmespath==1.0.1
joblib==1.4.2
jsonlines==4.0.0
jsonpatch==1.33
jsonpointer==3.0.0
jsonref==1.1.0
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter_client==7.4.9
jupyter_core==4.12.0
jupyter-events==0.12.0
jupyter_server==2.17.0
jupyter_server_terminals==0.5.3
jupyterlab_pygments==0.3.0
jupyterlab_widgets==3.0.13
kiwisolver==1.4.8
kombu==5.4.0
langcodes==3.5.0
language_data==1.3.0
linear-tsv==1.1.0
loguru==0.7.2
lxml==5.3.0
lz4==4.3.3
makefun==1.15.6
Mako==1.3.7
marisa-trie==1.2.1
markdown-it-py==3.0.0
MarkupSafe==3.0.2
marshmallow==3.23.1
matplotlib==3.10.1
matplotlib-inline==0.1.7
mdurl==0.1.2
mistune==3.0.2
mixpanel==4.10.1
more-itertools==10.5.0
moto==4.2.14
msal==1.31.1
msal-extensions==1.3.1
multidict==6.1.0
murmurhash==1.0.11
mypy==1.13.0
mypy-extensions==1.0.0
nbclassic==1.1.0
nbclient==0.6.3
nbconvert==7.16.4
nbformat==5.10.4
nest-asyncio==1.6.0
notebook==6.5.7
notebook_shim==0.2.4
numpy==1.26.4
oauthlib==3.2.2
openpyxl==3.1.5
oscrypto==1.3.0
overrides==7.7.0
packaging==24.2
pandas==2.1.4
pandocfilters==1.5.1
parse==1.20.2
parso==0.8.4
patchy==2.8.0
pexpect==4.9.0
phonenumbers==8.13.0
pillow==11.1.0
pip==24.3.1
platformdirs==4.3.6
preshed==3.0.9
progressbar2==4.4.2
prometheus_client==0.21.1
prompt_toolkit==3.0.48
propcache==0.2.1
prophet==1.1.5
proto-plus==1.25.0
protobuf==4.25.5
psutil==7.0.0
ptyprocess==0.7.0
pure_eval==0.2.3
py-partiql-parser==0.5.0
py4j==0.10.9.7
pyarrow==18.1.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pycountry==24.6.1
pycparser==2.22
pycurl==7.45.3
pydantic==1.10.19
pydeequ==1.1.1
Pygments==2.18.0
PyJWT==2.10.1
pyOpenSSL==24.3.0
pyparsing==3.2.0
pyspark==3.5.6
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-json-logger==3.3.0
python-stdnum==1.20
python-utils==3.9.1
pytz==2024.1
PyYAML==6.0.2
pyzmq==26.2.0
ratelimit==2.2.1
redis==5.2.0
redshift-connector==2.1.5
referencing==0.35.1
requests==2.32.3
requests-file==2.1.0
responses==0.25.3
rfc3339-validator==0.1.4
rfc3986==2.0.0
rfc3986-validator==0.1.1
rich==13.9.4
rpds-py==0.22.3
rsa==4.9
rstr==3.2.2
ruamel.yaml==0.17.17
s3transfer==0.10.4
schwifty==2024.6.1
scikit-learn==1.6.1
scipy==1.14.1
scramp==1.4.5
Send2Trash==1.8.3
sentry-sdk==2.33.2
setuptools==75.6.0
shellingham==1.5.4
six==1.17.0
smart-open==7.0.5
sniffio==1.3.1
snowflake-connector-python==3.13.2
snowflake-sqlalchemy==1.7.1
sortedcontainers==2.4.0
soupsieve==2.6
spacy==3.7.5
spacy-legacy==3.0.12
spacy-loggers==1.0.5
SQLAlchemy==1.4.54
sqlalchemy-bigquery==1.12.0
sqlalchemy-redshift==0.8.14
sqlalchemy-stubs==0.4
sqlglot==27.12.0
sqlglotrs==0.6.2
sqlparse==0.4.4
srsly==2.4.8
stack-data==0.6.3
stackprinter==0.2.12
stanio==0.5.1
starlette==0.27.0
tableschema==1.21.0
tabulate==0.9.0
tenacity==8.5.0
terminado==0.18.1
thinc==8.2.5
threadpoolctl==3.5.0
thrift==0.16.0
tinycss2==1.4.0
toml==0.10.2
tomli==2.2.1
tomlkit==0.13.2
toolz==1.0.0
tornado==6.4.2
tqdm==4.67.1
traitlets==5.14.3
typer==0.15.1
types-croniter==1.4.0.1
types-python-dateutil==2.9.0.20250822
typing_extensions==4.12.2
typing-inspect==0.9.0
tzdata==2024.2
tzlocal==5.2
ujson==5.10.0
unicodecsv==0.14.1
uri-template==1.3.0
urllib3==1.26.20
uvicorn==0.32.1
uvloop==0.21.0
vine==5.1.0
vininfo==1.8.0
wasabi==1.1.3
watchdog==3.0.0
watchfiles==1.0.0
wcmatch==10.0
wcwidth==0.2.13
weasel==0.4.1
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0
websockets==14.1
Werkzeug==3.1.3
widgetsnbextension==4.0.13
wrapt==1.17.0
xlrd==2.0.1
xmltodict==0.14.2
yarl==1.18.3
zipp==3.21.0

• 5553