Task : Parameterize downloading of file from a url(https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page), Parquetize it (convert the csv file to parquet ), Upload to GCS, Orchestrate it with Airflow.
gabidoye
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| students = { | |
| "name":"y", | |
| "studentid":"z", | |
| "courses":[ | |
| { | |
| "Coursetitle":"English", | |
| "Unit":2, | |
| "CourseCode":"ENG", |
gabidoye
/ DE_ZoomCamp Week2 Assignment.md
Last active
February 2, 2022 23:46
DE_ZoomCamp Week2 Assignment (
gabidoye
/ DE_ZoomCamp Assignment Week1
Last active
January 23, 2022 08:34
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| --Question 3: Count records | |
| select date(tpep_pickup_datetime), count(tpep_pickup_datetime) from public.yellow_taxi_trips | |
| where date(tpep_pickup_datetime)='2021-01-15' | |
| group by date(tpep_pickup_datetime) | |
| --Question 4: Largest tip for each day | |
| select date(tpep_pickup_datetime), max(tip_amount) from public.yellow_taxi_trips | |
| where (select extract(month from(date(tpep_pickup_datetime))))='01' | |
| group by date(tpep_pickup_datetime) |
gabidoye
/ sql_window_functions_01.sql
Created
August 1, 2021 18:32
— forked from aniruddha27/sql_window_functions_01.sql
SQL Window functions Code for article - https://www.analyticsvidhya.com/blog/2020/12/window-function-a-must-know-sql-concept/
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /* Sample data */ | |
| insert into emp (EMPID, NAME, JOB, SALARY) | |
| values | |
| (201, 'ANIRUDDHA', 'ANALYST', 2100), | |
| (212, 'LAKSHAY', 'DATA ENGINEER', 2700), | |
| (209, 'SIDDHARTH', 'DATA ENGINEER', 3000), | |
| (232, 'ABHIRAJ', 'DATA SCIENTIST', 2500), | |
| (205, 'RAM', 'ANALYST', 2500), | |
| (222, 'PRANAV', 'MANAGER', 4500), | |
| (202, 'SUNIL', 'MANAGER', 4800), |
gabidoye
/ Fire Incident Data Preparation.ipynb
Last active
July 5, 2021 23:28
spark/salesdata/FireIncident.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.