Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@ Use Observability Pipelines' Amazon OpenSearch destination to send logs to Amazo

## Setup

Configure the Amazon OpenSearch destination when you [set up a pipeline][6]. You can set up a pipeline in the [UI][1], using the [API][7], or with [Terraform][8]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the Amazon OpenSearch endpoint URL, and if applicable, username and password. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the Amazon OpenSearch destination when you [set up a pipeline][6]. You can set up a pipeline in the [UI][1], using the [API][7], or with [Terraform][8]. The steps in this section are configured in the UI.

After you select the Amazon OpenSearch destination in the pipeline UI:

Expand Down Expand Up @@ -49,6 +47,8 @@ After you select the Amazon OpenSearch destination in the pipeline UI:
1. Enter the ARN of the IAM role you want to assume.
1. Optionally, enter the assumed role session name and external ID.

{{% observability_pipelines/secrets_env_var_note %}}

#### Optional buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ After you select the Amazon Security Lake destination in the pipeline UI:

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the TLS key pass. Do <b>not</b> enter the actual value.</div>

{{% observability_pipelines/secrets_env_var_note %}}

{{% observability_pipelines/tls_settings %}}

{{% observability_pipelines/secrets_env_var_note %}}

##### Buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,17 @@ You need to have Datadog's [Azure integration][3] installed to set up Datadog Lo

## Set up the destination for your pipeline

Configure the Azure Storage destination when you [set up a pipeline][4]. You can set up a pipeline in the [UI][7], using the [API][8], or with [Terraform][9]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the Azure connection string. Do <b>not</b> enter the actual value.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the Azure Storage destination when you [set up a pipeline][4]. You can set up a pipeline in the [UI][7], using the [API][8], or with [Terraform][9]. The steps in this section are configured in the UI.

After you select the Azure Storage destination in the pipeline UI:

1. Enter the identifier for your Azure connection string. If you leave it blank, the [default](#secret-defaults) is used.
1. Enter the name of the Azure container you created earlier.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional settings

#### Prefix to apply to all key objects
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ Before you configure the ClickHouse destination, you must have:

## Setup

Configure the ClickHouse destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][4], using the [API][5], or with [Terraform][6]. The steps in this section are configured in the UI.

<div class="alert alert-danger">Only enter the identifiers for the ClickHouse endpoint URL and, if applicable, the username, password, and TLS key pass. Do <b>not</b> enter the actual values.</div>

Configure the ClickHouse destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][4], using the [API][5], or with [Terraform][6]. The steps in this section are configured in the UI.

After you select the ClickHouse destination in the pipeline UI:

1. Enter the identifier for your ClickHouse HTTP endpoint URL. If you leave it blank, the [default](#secret-defaults) is used.
Expand All @@ -48,6 +48,8 @@ After you select the ClickHouse destination in the pipeline UI:

**Note**: When you use `arrow_stream`, your ClickHouse server must be reachable when your worker loads the pipeline configuration from the remote configuration backend, or upon worker restart. The Clickhouse table schema is not persisted between restarts.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional settings

#### Skip unknown fields
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,18 @@ To use the CrowdStrike NG-SIEM destination, you need to set up a CrowdStrike dat

## Setup

Configure the CrowdStrike NG-SIEM destination when you [set up a pipeline][4]. You can set up a pipeline in the [UI][1], using the [API][5], or with [Terraform][6]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the CrowdStrike NG-SIEM endpoint URL, token, and if applicable, the TLS pass key. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the CrowdStrike NG-SIEM destination when you [set up a pipeline][4]. You can set up a pipeline in the [UI][1], using the [API][5], or with [Terraform][6]. The steps in this section are configured in the UI.

After you select the CrowdStrike NG-SIEM destination in the pipeline UI:

1. Enter the identifier for your CrowdStrike NG-SIEM endpoint URL. If you leave it blank, the [default](#secret-defaults) is used.
1. Enter the identifier for your CrowdStrike NG-SIEM token. If you leave it blank, the [default](#secret-defaults) is used.
1. Select {{< ui >}}JSON{{< /ui >}} or {{< ui >}}Raw{{< /ui >}} encoding in the dropdown menu.

{{% observability_pipelines/secrets_env_var_note %}}

#### Optional settings

##### Enable compressions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ See Databricks' [Add service principals to your account][4] and [Grant permissio

## Setup

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the OAuth client secret. Do <b>not</b> enter the actual value.</div>

Configure the Databricks (Zerobus) destination when you [set up a pipeline][6]. You can set up a pipeline in the [UI][7], using the [API][8], or with [Terraform][9]. The steps in this section are configured in the UI.

**Note**: Log fields that are not present in the table schema are dropped. For example, if a log has the fields `id`, `name`, and `host`, and the table schema only contains the columns `name` and `host`, then the `id` field is dropped and not written to the table.
Expand All @@ -101,16 +103,14 @@ After you select the Databricks (Zerobus) destination in the pipeline UI:

<div class="alert alert-warning">Databricks (Zerobus) doesn't convert timestamps in string format to Databricks' <a href="https://docs.databricks.com/aws/en/sql/language-manual/data-types/timestamp-type"><code>TIMESTAMP</code> type</a>. If your table uses a timestamp column, see <a href="#convert-string-timestamps-to-timestamp-format">Convert string timestamps to timestamp format</a> for more information.</div>

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the OAuth client secret. Do <b>not</b> enter the actual value.</div>

{{% observability_pipelines/secrets_env_var_note %}}

1. Enter the {{< ui >}}Ingestion Endpoint{{< /ui >}} for your Databricks workspace, such as `https://<workspace_id>.zerobus.<region>.cloud.databricks.com`. The Worker sends logs to this endpoint.
1. Enter the {{< ui >}}Table Name{{< /ui >}} in the format `catalog.schema.table`, such as `main.obs_pipelines.apache_common_logs`.
1. Enter the {{< ui >}}Unity Catalog Endpoint{{< /ui >}} for your Databricks workspace, such as `https://<workspace>.cloud.databricks.com`. The Worker uses this endpoint to read the table's schema.
1. In the {{< ui >}}Auth - Client ID{{< /ui >}} field, enter the application ID of the service principal, such as `abcdefgh-1234-5678-abcd-ef0123456789`.
1. In the {{< ui >}}Auth - Client Secret{{< /ui >}} field, enter the identifier for your OAuth client secret. If you leave it blank, the [default](#secret-defaults) is used.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional settings

#### Buffering
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,9 @@ Use Observability Pipelines' Elasticsearch destination to send logs or metrics t

## Setup

Configure the Elasticsearch destination when you [set up a pipeline][7]. You can set up a pipeline in the [UI][1], using the [API][8], or with [Terraform][9]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the Elasticsearch endpoint URL, username, and password. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the Elasticsearch destination when you [set up a pipeline][7]. You can set up a pipeline in the [UI][1], using the [API][8], or with [Terraform][9]. The steps in this section are configured in the UI.

After you select the Elasticsearch destination in the pipeline UI:

Expand All @@ -48,6 +46,8 @@ After you select the Elasticsearch destination in the pipeline UI:
- `logs-apache-production` for logs
- `metrics-apache-production` for metrics

{{% observability_pipelines/secrets_env_var_note %}}

#### Optional settings

##### Enable TLS
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,10 @@ After you select the Google Pub/Sub destination in the pipeline UI:

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the TLS key pass. Do <b>not</b> enter the actual value.</div>

{{% observability_pipelines/secrets_env_var_note %}}

{{% observability_pipelines/tls_settings %}}

{{% observability_pipelines/secrets_env_var_note %}}

#### Buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,9 @@ The Observability Pipelines Worker uses standard Google authentication methods.

## Setup

Configure the Google SecOps destination when you [set up a pipeline][8]. You can set up a pipeline in the [UI][1], using the [API][9], or with [Terraform][10]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the Google SecOps endpoint URL. Do <b>not</b> enter the actual value.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the Google SecOps destination when you [set up a pipeline][8]. You can set up a pipeline in the [UI][1], using the [API][9], or with [Terraform][10]. The steps in this section are configured in the UI.

After you select the Google SecOps destination in the pipeline UI:

Expand All @@ -33,6 +31,8 @@ After you select the Google SecOps destination in the pipeline UI:
1. Select {{< ui >}}JSON{{< /ui >}} or {{< ui >}}Raw{{< /ui >}} encoding in the dropdown menu.
1. Enter the log type. See [template syntax][4] if you want to route logs to different log types based on specific fields in your logs.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,9 @@ Use Observability Pipelines' HTTP Client destination to send logs to an HTTP cli

## Set up destination

Configure the HTTP Client destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][1], using the [API][4], or with [Terraform][5]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the HTTP Client URI and, if applicable, username and password for basic authorization and the TLS key pass. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the HTTP Client destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][1], using the [API][4], or with [Terraform][5]. The steps in this section are configured in the UI.

After you select the HTTP Client destination in the pipeline UI:

Expand All @@ -35,6 +33,8 @@ After you select the HTTP Client destination in the pipeline UI:
- Enter the identifier for your HTTP Client token. If you leave it blank, the [default](#secret-defaults) is used.
1. JSON is the only available encoder.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional settings

#### Enable compression
Expand Down
6 changes: 3 additions & 3 deletions content/en/observability_pipelines/destinations/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,9 @@ Common scenarios when you might use this destination:

## Setup

Configure the Kafka destination when you [set up a pipeline][10]. You can set up a pipeline in the [UI][5], using the [API][11], or with [Terraform][12]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the Kafka bootstrap servers and, if applicable, the SASL username and password and the TLS key pass. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the Kafka destination when you [set up a pipeline][10]. You can set up a pipeline in the [UI][5], using the [API][11], or with [Terraform][12]. The steps in this section are configured in the UI.

After you select the Kafka destination in the pipeline UI:

Expand All @@ -41,6 +39,8 @@ After you select the Kafka destination in the pipeline UI:

{{< img src="observability_pipelines/destinations/kafka_settings.png" alt="The Kafka destination with sample values" style="width:30%;" >}}

{{% observability_pipelines/secrets_env_var_note %}}

#### Optional settings

##### Enable TLS
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -156,11 +156,9 @@ The table below summarizes the Azure and Microsoft Sentinel information you need

## Setup

Set up the Microsoft Sentinel destination when you [set up a pipeline][10]. You can set up a pipeline in the [UI][1], using the [API][11], or with [Terraform][12]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the Microsoft Sentinel client secret and Data Collection Endpoint. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Set up the Microsoft Sentinel destination when you [set up a pipeline][10]. You can set up a pipeline in the [UI][1], using the [API][11], or with [Terraform][12]. The steps in this section are configured in the UI.

After you select the Microsoft Sentinel destination in the pipeline UI:

Expand All @@ -171,6 +169,8 @@ After you select the Microsoft Sentinel destination in the pipeline UI:
1. Enter the full table name to which you are sending logs. An example table name: `Custom-MyOPWLogs_CL`.
1. Enter the Data Collection Rule (DCR) immutable ID, such as `dcr-000a00a000a00000a000000aa000a0aa`.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,18 @@ Use Observability Pipelines' New Relic destination to send logs to New Relic.

## Setup

Configure the New Relic destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][1], using the [API][4], or with [Terraform][5]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the account ID and license. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the New Relic destination when you [set up a pipeline][3]. You can set up a pipeline in the [UI][1], using the [API][4], or with [Terraform][5]. The steps in this section are configured in the UI.

After you select the New Relic destination in the pipeline UI:

1. Enter the identifier for your account ID. If you leave it blank, the [default](#secret-defaults) is used.
1. Enter the identifier for your license. If you leave it blank, the [default](#secret-defaults) is used.
1. Select the data center region ({{< ui >}}US{{< /ui >}} or {{< ui >}}EU{{< /ui >}}) of your New Relic account.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,9 @@ Use Observability Pipelines' OpenSearch destination to send logs to OpenSearch.

## Setup

Configure the OpenSearch destination when you [set up a pipeline][6]. You can set up a pipeline in the [UI][1], using the [API][7], or with [Terraform][8]. The steps in this section are configured in the UI.

<div class="alert alert-danger">For Secrets Management: Only enter the identifiers for the OpenSearch endpoint URL, username, and password. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}
Configure the OpenSearch destination when you [set up a pipeline][6]. You can set up a pipeline in the [UI][1], using the [API][7], or with [Terraform][8]. The steps in this section are configured in the UI.

After you select the OpenSearch destination in the pipeline UI:

Expand All @@ -41,6 +39,8 @@ After you select the OpenSearch destination in the pipeline UI:
- In the {{< ui >}}Namespace{{< /ui >}} field, enter the grouping for organizing your data streams, for example `production`.
- In the UI, there is a preview of the data stream name you configured. With the above example inputs, the data stream name that the Worker writes to is `logs-apache-production`.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional settings

#### OpenSearch index
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,17 @@ Use Observability Pipelines' SentinelOne destination to send logs to SentinelOne

## Setup

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the token. Do <b>not</b> enter the actual value.</div>

Configure the SentinelOne destination when you [set up a pipeline][4]. You can set up a pipeline in the [UI][1], using the [API][5], or with [Terraform][6]. The steps in this section are configured in the UI.

After you select the SentinelOne destination in the pipeline UI:

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the token. Do <b>not</b> enter the actual value.</div>

{{% observability_pipelines/secrets_env_var_note %}}

1. Enter the identifier for your token. If you leave it blank, the [default](#secret-defaults) is used.
1. Select your SentinelOne logs environment in the dropdown menu.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional buffering

{{% observability_pipelines/destination_buffer %}}
Expand Down
8 changes: 4 additions & 4 deletions content/en/observability_pipelines/destinations/socket.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,18 @@ Use Observability Pipelines' Socket destination to send logs to a socket endpoin

## Setup

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the socket address and, if applicable, the key pass. Do <b>not</b> enter the actual values.</div>

Configure the Socket destination when you [set up a pipeline][2]. You can set up a pipeline in the [UI][1], using the [API][3], or with [Terraform][4]. The steps in this section are configured in the UI.

After you select the Socket destination in the pipeline UI:

<div class="alert alert-danger">For Secrets Management: Only enter the identifier for the socket address and, if applicable, the key pass. Do <b>not</b> enter the actual values.</div>

{{% observability_pipelines/secrets_env_var_note %}}

1. Enter the identifier for your address. If you leave it blank, the [default](#secret-defaults) is used.
1. In the {{< ui >}}Mode{{< /ui >}} dropdown menu, select the socket type to use.
1. In the {{< ui >}}Encoding{{< /ui >}} dropdown menu, select either {{< ui >}}JSON{{< /ui >}} or {{< ui >}}Raw message{{< /ui >}} as the output format.

{{% observability_pipelines/secrets_env_var_note %}}

### Optional settings

#### Enable TLS
Expand Down
Loading
Loading