Last active
November 6, 2022 00:51
-
-
Save akoumjian/5cb22aa83602e7f152e75e142154e9dc to your computer and use it in GitHub Desktop.
Generate CRC32 base64 encoded hash the same way google does
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Google makes it difficult to verify if local files match the ones on their cloud storage. | |
This snippet takes a file path to run the CRC32C hashing and then base64 encode it so it matches the results | |
from the google-cloud-storage python blob api | |
You will need pip install: | |
- google-cloud-storage | |
""" | |
import google_crc32c | |
import collections | |
def generate_file_crc32c(path, blocksize=2**20): | |
""" | |
Generate a base64 encoded crc32c checksum for a file to compare with google cloud storage. | |
Returns a string like "4jvPnQ==" | |
Compare with a google storage blob instance: | |
blob.crc32c == generate_file_crc32c("path/to/local/file.txt") | |
""" | |
crc = google_crc32c.Checksum() | |
read_stream = open(path, "rb") | |
collections.deque(crc.consume(read_stream, blocksize), maxlen=0) | |
read_stream.close() | |
return base64.b64encode(crc.digest()).decode("utf-8") |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment