TBC
Yu Watanabe yuwtennis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ywatanabe@lemp13 beam]$ ./start-build-env.sh ./gradlew checkSetup | |
| ++ dirname ./start-build-env.sh | |
| + cd . | |
| + DOCKER_DIR=dev-support/docker | |
| + DOCKER_FILE=dev-support/docker/Dockerfile | |
| + CONTAINER_NAME=beam-dev-ywatanabe-202434 | |
| ++ command -v docker | |
| + '[' '!' -x /usr/bin/docker ']' | |
| + docker build -t beam-build -f dev-support/docker/Dockerfile dev-support/docker | |
| [+] Building 211.8s (24/24) FINISHED docker:default |
yuwtennis
/ gist:1abf229918161ec2a54f54092af78c58
Last active
March 9, 2025 07:41
Using A record as DNS for Cloud SQL Auth Proxy connection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| resource "google_compute_network" "vpc" { | |
| name = "dsongcp" | |
| auto_create_subnetworks = false | |
| } | |
| resource "google_compute_subnetwork" "private_subnet" { | |
| name = "prv-subnet" | |
| ip_cidr_range = "192.168.0.0/28" | |
| network = google_compute_network.vpc.id | |
| region = local.region |
yuwtennis
/ DagOptions.java
Created
July 14, 2024 09:54
Practical example for setting custom options as beamTestPipelineOptions for TestPipeline in apache beam
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| public interface DagOptions extends PipelineOptions { | |
| @Description("Dag options") | |
| @Default.String("HELLOWORLD") | |
| String getDagType(); | |
| void setDagType(String dagType); | |
| } |
yuwtennis
/ az-consumption
Last active
December 29, 2022 03:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| SUBSCRIPTION_ID: str = os.environ.get("SUBSCRIPTION_ID", None) | |
| SCOPE: str =f'/subscriptions/{SUBSCRIPTION_ID}' | |
| logging.basicConfig(filename='./az-sdk-consumption-api-8-0-0-b1.log', level=logging.DEBUG, force=True) | |
| current_timestamp = datetime.datetime.now(datetime.timezone.utc).strftime("%c %") | |
| print(f"Querying Microsoft REST API. Started on {current_timestamp}") | |
| consumption_client = ConsumptionManagementClient( | |
| credential=DefaultAzureCredential(), | |
| subscription_id=SUBSCRIPTION_ID |
yuwtennis
/ elasticsearchio
Last active
August 28, 2022 11:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package elasticsearchio | |
| import ( | |
| "fmt" | |
| "github.com/apache/beam/sdks/v2/go/pkg/beam" | |
| "github.com/apache/beam/sdks/v2/go/pkg/beam/core/typex" | |
| "github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/reflectx" | |
| ) | |
| const ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ywatanabe@laptop-archlinux beam]$ ./gradlew :sdks:java:io:expansion-service:runExpansionService -PconstructionService.port=18089 | |
| Configuration on demand is an incubating feature. | |
| > Task :sdks:java:io:expansion-service:runExpansionService | |
| Starting expansion service at localhost:18089 | |
| Aug 17, 2022 6:49:19 PM org.apache.beam.sdk.expansion.service.ExpansionService loadRegisteredTransforms | |
| beam:transform:org.apache.beam:kafka_read_with_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@663c9e7a | |
| INFO: Registering external transforms: [beam:transform:org.apache.beam:kafka_read_with_metadata:v1, beam:transform:org.apache.beam:kafka_read_without_metadata:v1, beam:transform:org.apache.beam:kafka_write:v1, beam:external:java:generate_sequence:v1] | |
| beam:transform:org.apache.beam:kafka_read_without_metadata:v1: org.apache.beam.sdk.expansion.service.ExpansionService$ExternalTransformRegistrarLoader$1@19e4653c | |
| beam:transform:org.apache.beam:kafka_w |
yuwtennis
/ TUTORIAL_ES_DATASTREAM.md
Last active
June 9, 2022 05:58
Indexing document using data stream in elassticsearch
yuwtennis
/ TUTORIAL_ES_ALIAS.txt
Last active
June 9, 2022 05:59
Indexing document using alias endpoint in elasticsearch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # ILM | |
| # https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html#ilm-rollover-primar-shardsize-ex | |
| # rollover in 5 min | |
| PUT _ilm/policy/my_policy | |
| { | |
| "policy": { | |
| "phases": { | |
| "hot": { | |
| "actions": { | |
| "rollover" : { |
See go.result for full log.
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ go run examples/quickstarts/admin/report/main.go &> go.result
(venv) y-watanabe@LAPTOP-IG41EBJ5:~/repos/github/google-api-tutorials/sdks/go$ grep Page go.result
2022/04/08 18:35:22 Page: 1, Received 1000 items
yuwtennis
/ filebeat.debug.yml
Last active
April 1, 2022 02:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ###################### Filebeat Configuration Example ######################### | |
| # This file is an example configuration file highlighting only the most common | |
| # options. The filebeat.reference.yml file from the same directory contains all the | |
| # supported options with more comments. You can use it as a reference. | |
| # | |
| # You can find the full configuration reference here: | |
| # https://www.elastic.co/guide/en/beats/filebeat/index.html | |
| # For more available modules and options, please see the filebeat.reference.yml sample |
NewerOlder